WO2018211312A1

WO2018211312A1 - Augmented reality system for providing support to user concerning technical device

Info

Publication number: WO2018211312A1
Application number: PCT/IB2017/052920
Authority: WO
Inventors: Dzmitry MALYSHKO; Roman Veksler; Boris BURAKOV; Yury BANDARENKA
Original assignee: Uab Atomichronica
Current assignee: Uab Atomichronica
Priority date: 2017-05-18
Filing date: 2017-05-18
Publication date: 2018-11-22
Anticipated expiration: 2019-11-18

Abstract

The invention presented in this disclosure is related to augmented reality systems. The main purpose of the augmented reality system of this invention is to provide information to user which helps to control, maintain, repair and/or accomplish other task with the different technical device. The remote information providing means using augmented reality is far more effective to compare with ordinary voice- only information or studying instructions, manuals in order to know technical device. The main difference from available augmented reality systems dedicated for similar purpose is that the system of this invention is capable to generate new data, new scripts, new instructions based on the available data and data provided by users.

Description

AUGMENTED REALITY SYSTEM FOR PROVIDING SUPPORT TO USER

CONCERNING TECHNICAL DEVICE

FIELD OF THE INVENTION

The invention relates to an augmented reality system for providing support information to control, maintain and repair of technical devices.

DESCRIPTION OF RELATED ART

Augmented reality systems, where computer generated view is overlaid with the captured real view, rapidly coming to everyday life. One of the spheres to use such systems is to provide help and support to personnel in a remote location to control, maintain and repair different technical devices. The augmented reality system of this invention is comprised of mobile unit and server, which are able to exchange the data in both directions. The augmented reality system is able to generate new ways, new scripts, new instructions of control, maintenance and reparation of the technical devices based on the internally available data and data received from users.

There are known many patent documents (e.g. US2016/0358383 published on 8 December 2016) where augmented reality system helps to communicate two persons, one of which performing task and needs additional information, and another one is presenting needed additional information using augmented reality system. In such kind of systems the information provider is a human whereas in the present invention the information provider is a computer, therefore no need for human assistance, information can be reached at any time not depending whether anyone is ready to present needed information.

Another document US2002/0044104, published on 18 April 2002, discloses invention where the information providing is accomplished by computer, i.e. human, as an information provider is eliminated. However there is nothing mentioned about ability to generate new scripts, instructions by the server.

The object of this invention differs from the related Prior art in that it is able to generate new scripts and instructions to user, based on the available data and data, received from other users. SUMMARY OF THE INVENTION

The invention, presented in this disclosure, is related to augmented reality systems. The main purpose of the augmented reality system of this invention is to provide information to user, which helps to control, maintain, repair and/or accomplish other task with the different technical device. The remote information providing means using augmented reality is far more effective to compare with ordinary voice-only information or studying instructions, manuals in order to know technical device. The main difference from available augmented reality systems, dedicated for similar purpose, is that the system of this invention is capable to generate new data, new scripts and new instructions based on the available data and data provided by users.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig.1 presents general scheme of mobile unit and server and data exchange between mobile unit and server.

Fig.2 presents data flows in the server.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

This invention is related to augmented reality processes where real view is superimposed with the graphics generated by computer. The purpose of the system described in this disclosure, is to provide for user instructions, commands, guidance and advices how to act depending on the present situation, i.e.: how to reach the desired condition(s), status(es) in technical device according to the initial state and data using particular functional steps. Mostly, this invention may be used for controlling different technical devices, industrial equipment or other similar items (apparatuses). The user can be unfamiliar with certain technical device in order to properly control, perform maintenance or repair it. In particular case user orientates his mobile unit to the unknown technical device to capture view of it. The mobile unit captures view of the technical device and presents information in a form of augmented reality to the user, also it allows the user to enter data related with a status of the device to which device must be transferred depending on what must be performed with technical device in a certain situation. According to the presented information, user changes the status (parameters, settings, adjustments) of the technical device. The amended status of the technical device is again captured by mobile unit and analyzed if reached status of the technical device satisfies the goal.. This cycle is being repeated until task is accomplished, i.e.: until the control, maintenance or reparation of the technical device is completed.

The information, presented (shown) to user, can be static (e.g.: handbooks, drawings, maintenance instructions, etc.) or can be dynamic (e.g.: movable graphical views superimposed on the real view, showing the flow of particular operations to be accomplished).

Augmented reality system (hereinafter AR system) comprises set of technical means which are dedicated to achieve the technical effect for solving the particular technical problem. In this disclosure we describe how to achieve the technical effect, i.e.: to find out solutions for transferring the system from one status to another.

A person (hereinafter user) who uses AR system in order to control, maintain, repair, improve industrial equipment, control panels, communication systems or any other technical device (hereinafter technical device). The said process of controlling, maintaining, repairing, improving of technical device using AR system hereinafter will be called "task" (e.g. to accomplish task, to begin task, etc.).

The mentioned AR system comprises at least

- a mobile unit,

- a server.

The mobile unit - is an electronic device, mobile data processing apparatus, having technical means to capture visual information and present visual information to user; to save, to analyze, to transmit and to receive data. The mobile unit captures real view, analyses this information and adds graphical and/or audio information to captured view and presents such superimposed information to user. The mentioned process of capturing, analyzing and presenting of information is permanent and repeated until task is accomplished. The mentioned mobile has a microphone to record voice of user which is analyzed and can be used to add information, control commands to AR system. The mobile unit can be a smart phone, a tablet computer, a portable computer, goggles (glasses) or set of any of the mentioned devices working together and communicating with server; or other similar device. In most cases of the present invention it is possible to transfer data analysis function to the server which can be allocated in remote location.

The mobile unit and the server have technical means for bidirectional data exchange and possibility to exchange data.

The mobile unit additionally can be equipped with sensors to input different kind of data: e.g.: can be equipped with infrared camera to measure temperature of the surface. In other embodiments mobile unit can be equipped with input sockets, to which can be connected different kind of sensors, probes, e.g.: socket to measure voltage of electric equipment.

The server is an electronic device comprising at least such functional blocks, modules:

- user database to authorize user;

- instructions and scripts database and connected to this base

- augmented reality objects database;

- expert system and connected to this base

- knowledge base.

All the above mentioned functional modules communicate with mobile unit using dedicated interface which is a part of the server.

User DB collects, contains, processes information concerning users who uses system. User DB can contain user identity data, settings of the AR system of each user. User DB provides information concerning authentication of users. In others embodiments user DB can contain information about particular user proficiency and knowledge level and adapt work of all AR system accordingly. There is bidirectional data flow between user DB and mobile unit.

Instructions and scripts DB (hereinafter scripts DB) contain instructions, scripts and other material which is provided to user in order to help him perform particular task with technical device. The mentioned instructions, scripts are tested, approved and ready use. There is bidirectional data flow between scripts DB and mobile unit. Scripts DB have two additional links for data exchange with AR-objects DB and with expert systems. AR-objects DB have a visualization means in order to present data from scripts DB in superimposed manner, i.e.: scripts DB have a data, content to present for user, AR-object DB have a visualization means of how to present this data to user while overlaying these visualizations on the real view captioned by user camera.

Another data exchange link of scripts DB is link with expert system. Expert system relates on available instructions and scripts from scripts DB, analytics of performance parameters and status of technical device and generates new ways, methods to perform task and/or to improve the performance of technical device. The generation of the new ways and methods to perform the task is based on artificial intelligence means or other ways, capable to provide generation of new data relating on the available data. Expert system can provide solutions (in some cases: sequence of functional steps), advices to user which are not tested yet; if the user accepts this advice and performs task according to this advice, then user is being asked by the AR system to describe the result of the performed task, to add possible solutions, notes to improve the earlier existing advice. As it is mentioned above, expert system relates on scripts DB and additionally on its own knowledge base. Knowledge base is dedicated to collect and store the above mentioned comments of users to provide advice. The comments can be in different format: visual, audio information, text notes, etc. To the contrary to scripts DB, data in the knowledge base can be not approved, not tested, collected from users.

Additionally to the above mentioned, expert system provides to user additional instructions and scripts if it is not enough to accomplish the task, based on the scripts from DB. The said information from knowledge base must be tested and approved by the user. Different control means are applied in order to prevent existing means which can potentially be not effective or even harmful to the technical device.

Server is dedicated to provide dedicated support for mobile unit, namely:

- authorization of connected mobile unit/user;

- providing instructions/scripts to perform task;

- providing additional instructions or notes from knowledge base upon user request;

- processing of approved data from user in order to update knowledge base.

The proceeding of the task begins when user orientates mobile device to the technical device and captures view of technical device. The view of technical device is being analyzed in order to identify technical device. In other embodiments of this invention, the user can enter information in order to identify technical device. The said process of identification, as described above, can be performed in mobile unit or server. When technical device is identified, the objects of this device (control knobs, handles, indicators, etc.) are selected which will be needed in order to use AR system. While selected the said objects, some of them can be used in order to define the status of the technical device and select data in order to amend defined status to target status. If there are no any objects which could help to define status - other methods are used including definition by user, selecting from AR system menu, etc. Depending on the selected target type of the task, AR system presents to user scenario and particular steps to reach target task. When task is being processed, AR system monitors user and technical device interaction, records the intermediate and final results and compares the performance with internal records. If there are mismatches between newly recorded interaction and interactions, recorder previously, analysis is performed and improvements added to the interaction in databases if needed. User can additionally add his own comments, notes concerning accomplished task in order to improve AR system. When task is performed, AR system monitors the current status of system in order to select ant present to user next steps to reach the target status. In other embodiments of the invention, AR system measures parameters (it can be with help of the user), defines the final status of technical device, compares defined parameters with standard, nominal parameters (which defines that system is working properly) and if there are some mismatches which cannot be eliminated during this particular task, another task is proposed in order to reach standard parameters. While performing task, the user can request AR system to provide additional notes (which were added by other users), additional instructions (which are generated by expert system based on the knowledge base). The AR system collects and processes notes not only from user but records feedback from technical device to approve or improve instructions in the data bases of the server. In other embodiments of the invention, AR system can record the movements of body parts of user while he is performing task in order to present this information to other user - this is the way to present performance of the task in more exact and comprehensible way. Other possible way to constantly monitor actions of the user - is to control status change of the technical device. While constantly monitoring the status of technical device, it is possible to determine whether the task accomplishment progress is appropriate, proceeded in the suitable manner. On the other hand, the monitoring of the user actions, technical device status change gives additional information for expert system and knowledge base for improvements of instruction or for generating new data.

The above mentioned mobile unit and the server and their functions can be accomplished not only in the unit as described above. Distribution of functions can be flexible in order to provide as quicker response to the user request as it is possible: e.g.: if mobile unit has enough internal memory and enough processor ability and other technical resources to receive data from capturing device, analyze this data and present needed data to the user, the server in this case can be eliminated from data processing. In other embodiments all needed information from server can be downloaded in advance, prior to the real need if all conditions to choose needed data are satisfied (i.e.: the technical device is known, present status of device is known and target status to be reached is known). This case can be used if no mobile data network is available at the particular place.

To illustrate and describe this invention, description of preferred embodiments is given above. This is not a complete or limiting description aimed at prescribing a precise embodiment or implementation option. The description given above should be regarded more as an illustration than a constraint. Obviously, the specialists in this field may clearly see a multitude of modifications and variations. The embodiment has been selected and described so as to enable the specialists in this field to best understand the principles behind this invention and their best practical application for different embodiments with different modifications fit for a specific application or embodiment customisation. The invention scope is defined by the attached claims and their equivalents wherein all the terms used have the broadest possible meanings unless stated otherwise.

Embodiments described by specialists in the respective field may contain changes that do not depart from the scope of this invention, as described in the claims given next.

Claims

1 . An augmented reality system suitable for providing support for a user to control, maintain, repair a technical device, comprising

a mobile unit, which is able:

to capture visual data,

to save data, to analyze data, send to server and receive from server data, present content to the user;

server, which communicates with the mobile unit

in order to transmit requested data to mobile unit and to receive data from mobile unit, and which saves, analyzes and generates support in order to complete the control, maintenance, reparation of the technical device;

characterized in that the server comprises:

user database module which can maintain bidirectional data flow with the mobile unit wherein the user database module is dedicated to collect and analyze data related to the user, his authentication data, his preferred settings of the augmented reality system and other aspects of the interaction between the user and the augmented reality system;

instructions and scripts database module and connected to this module augmented reality objects database module, wherein both modules are configured to send data to the mobile unit which is presently available in the instructions and scripts database module;

expert system module and connected to this module knowledge base module wherein both modules are configured to collect data from the user, analyze this data and generate new instructions and scripts or other support data for control, maintenance and reparation of the technical device,

wherein

all above mentioned features leads to technical effect wherein augmented reality system is able to generate new support data relating on the available data and data provided from users using artificial intelligence to transfer the technical device from initial status to requested status.

2. Augmented reality system according to claim 1 , characterized in that the mobile unit is equipped with input sockets to which can be connected different kind of sensors, probes to receive different type of signals/data.

3. Method to use augmented reality system according claims 1 -2 for providing support to the user via the mobile unit comprising the steps of:

capturing visual data, saving data related with captured visual data, analyzing data, sending to server and receiving from server data, presenting content to the user, the step is being performed by the mobile unit,

transmitting requested data to mobile unit and receiving data from mobile unit and saving, analyzing, generating support in order to complete the control, maintenance, reparation of the technical device, the step is being performed by the server while communicating with the mobile unit,

characterized in that the server comprises the following steps:

collects and analyzes data related to the user, his authentication data, his preferred settings of the augmented reality system and other aspects of the interaction between the user and the augmented reality system, the step is being performed by user database module in the server;

sending presently available data to the mobile unit the step is being performed by the instructions and scripts database module and connected to this module augmented reality objects database module;

collecting data from the user, analyzing this data and generating new instructions and scripts or other support data for control, maintenance and reparation of the technical device, the step is being performed by expert system module and connected to this module knowledge base module;

wherein

all above mentioned steps leads to technical effect wherein augmented reality system is able to generate new support data relating on the available data and data provided from users using artificial intelligence to transfer the technical device from initial status to requested status.