WO2024261266A1

WO2024261266A1 - Mobile device for controlling operation of an automation component

Info

Publication number: WO2024261266A1
Application number: PCT/EP2024/067482
Authority: WO
Inventors: Tim SCHMIEDL; Jens RIEGGER; Manuel Frey
Original assignee: Fruitcore Robotics GmbH
Current assignee: Fruitcore Robotics GmbH
Priority date: 2023-06-22
Filing date: 2024-06-21
Publication date: 2024-12-26
Anticipated expiration: 2025-12-22

Abstract

The present invention relates to a mobile device for controlling operation of at least one automation component. The mobile device comprises a processing unit configured for running an operation system and a user interface comprising a display device and configured for receiving a user's input for controlling operation of the at least one automation component based on the user's input. Further, the mobile device comprises an AI copilot module configured for providing the user with information.

Description

Mobile Device for Controlling Operation of an Automation Component

The present invention relates to the control of the operation of automation components of a factory and automatic line operation of a production line and, in particular, the operation of automation components by means of mobile devices.

Industrial facilities comprise automation components, for example, robots, the operation of which has to be carefully controlled in order to guarantee high yield at desired qualities and to avoid accidents and production downtimes. An industrial robot includes a robot controller and a manipulator and is programmed to carry out some well-defined work by moving the manipulator along a predetermined operating path.

Conventionally, on-line or teach-in programming of a robot can be performed by means of a portable user operated client device commonly called Teach Pendant Unit (TPU). The TPU may include operator control means, for example, a joystick, a ball, or a set of buttons, for instructing the robot movement. The TPU may further comprise a visual display unit and an emergency stop button. The TPU may be used for monitoring the robot program, changing certain variables in the program, starting, stopping and editing the program or switching between different controller modes such as a manual mode in that movements of the robot are controlled by the TPU and an automatic mode in that movements of the robot are controlled in accordance with a control program running on a robot controller.

TPUs are relatively complex, expensive, and proprietary devices. Therefore, robot manufacturers and end-users more and more provide and use commercial-off-the-shelf mobile devices as, for example, smart phones and tablets communicating with robot controllers, see, for example, EP2958711A1 , EP3558602A1.

However, the operation of automation components even when facilitated by a computational mobile device is a relatively complex and error-prone task for a user, in particular, for a relatively non-experienced user, and mal-operation of an automation component bears high safety risks.

In view of the above, it is a problem underlying the present invention to provide a user with a convenient, adaptive and secure control of the operation of an automation component at relatively low costs.

The above-mentioned problem is solved by providing a mobile device (for example, a tablet, notebook, PDA, or smartphone) for controlling operation of at least one automation component according to a first aspect. The mobile device comprises a processing unit configured for running an operation system and a user interface comprising a display device and configured for receiving a user’s input for controlling operation of the at least one automation component based on the user’s input. Further, the mobile device comprises an Artificial Intelligence, Al, copilot module configured for providing the user with information on at least one of connectivity and operation of the at least one automation component. Controlling the operation of the at least one automation component may comprise config- uring/programming the operation of the at least one automation component. The at least one automation component may be or comprise a robot or any automatized tool and it may be comprised by the system. The at least one automation component may be some or all of the automation components of a facility or production line.

A user can control and program operation of the at least one automation component by means of the mobile device that may also be configured for monitoring/displaying operation of the at least one automation component.

By the provision of the Al copilot module a user can be reliably and conveniently assisted in operating the at least one automation component and any other operating components that might be present, for example, a camera device, a grip component, a conveyor (belt) or a light barrier. In particular, the Al copilot module can assist the user in installing, operating, programming, maintaining and repairing automation/operating components of a factory system. Thus, management and operation of the factory by a user is significantly improved by the Al copilot module that can be conveniently used and is comprised in a mobile device that, in particular, may be configured for single-hand operation.

The processing unit may be configured for at least partly running the Al copilot module. The Al copilot module may be partly run on a remote computer. Alternatively, depending on the computational resources provided by the mobile device and particular applications the Al copilot module may be run on an external computer and only an App accessing the Al copilot module running on the remote computer will be installed in the mobile device. Flexibility of the implementation of the Al copilot module may be enhanced by a distributed approach.

The processing unit may be configured for displaying on the display device, in particular, concurrently, data representing information on the at least one automation component (and optionally any other operating component) and text output by the Al copilot module. Thus, the user may, advantageously, be assisted by the Al copilot module regarding an operation task related to the (information on the) at least one automation component (and optionally any other operating component). In fact, the Al copilot module may be configured for chatting with the user by means of the user interface by text and/or speech output based on learned context related to the at least one automation component (and optionally any other operating component) and while chatting with the Al copilot module the user may read information on the at least one automation component (and optionally any other operating component) displayed on the display device. Particularly, the Al copilot module may be configured for monitoring operation of the at least one automation component (and optionally any other operating component) and providing the user with information on the monitored operation and chatting with the user about the operation. According to an embodiment, the mobile device is configured for programming at least one of a) connection of the at least one automation component (20) to another component, b) operation of the at least one automation component (20), and c) operation of the other component. In this case, the Al copilot module is configured for assisting the user in programming of the at least one of a) the connection of the at least one automation component to the other component, b) the operation of the at least one automation component, and c) the operation of the other component. The Al copilot module can present programming solutions or correct programming attempts by the user or suggest what kind of programming is needed and, thereby, may significantly accelerate the programming procedure. In particular, the Al copilot module may be configured for at least partly programming itself the at least one of a) the connection of the at least one automation component to the other component, b) the operation of the at least one automation component, and c) the operation of the other component.

For programming the at least one automation component (or other operating component), the operation system running on the processing unit may be configured for allowing the user access to an operation system of the at least one automation component (or other operating component) for programming the operation system of the at least one automation component (or other operating component). In particular, parts of the operation system of the at least one automation component (or other operating component) may be integrated into the operation system running on the processing unit for direct access by the user and/or the Al copilot module.

The above-mentioned other operating component may comprise or may be connected with a display interface and, in this case, the mobile device may be configured for streaming the display interface on the display device. Thus, the other operating component can be directly operated by the user by means of the mobile device. Further, the Al copilot module may assist the user in operation of the other component while the display interface of the same is streamed on the display device of the mobile device. Particularly, the Al copilot module may be configured for providing the user with information of at least one of connectivity and operation of the other operating component.

According to an embodiment, the processing unit is configured for displaying a virtual keyboard on the display device, wherein the (intelligent) virtual keyboard is configured for receiving a user’s text input and assisting the user in inputting text by suggesting word completions of letters input by the user and the Al copilot module is configured for generating the suggested word completions. Different from keyboards of the art that offer word completion options, the virtual keyboard of the mobile device is instructed by the Al copilot module to offer suitable completions.

For example, the Al copilot module instructs the virtual keyboard to present possible completions based on its learned knowledge about installment, operation, maintenance, etc. regarding a (automation, operating) component under consideration. The Al copilot module, on the other hand, may be configured for being at least partly operated based on the text inputs made by the user using the suggested completions. Thereby, a fast and effective human-machine interaction can be provided.

According to another embodiment, the mobile device is configured for determining its actual position or receiving information on its actual position and the Al copilot module is configured for providing the user with the information on at least one of connectivity and operation of the at least one automation component (or any other component involved) based on the determined actual position of the mobile device or received information on the actual position of the mobile device. For example, the Al copilot module may chat differently when the user is within an operation area of an operating component under consideration as it does when the user is not within an operation area of the operating component.

The above-mentioned problem is also solved by providing use of a mobile device for operating at least one automation component, comprising receiving a first user’s input by an interface of the mobile device; outputting an output by an Artificial Intelligence, Al, copilot module of the mobile device in response to the first user’s input, receiving a second user’s input different from the first user’s input by the interface in response to the output by the Al copilot module; and controlling operation of the at least one automation component based on the second user’s input. The mobile device may be a mobile device according to any of the above-described examples and the use provides the similar advantages as described above.

Furthermore, in order to solve the above-mentioned problem according to a second aspect it is provided a mobile device comprising a user interface comprising a display device and configured for receiving a user’s input for controlling operation of at least one automation component based on the user’s input and an Artificial Intelligence, Al, copilot module integrated with the user interface and configured for chatting with the user about information on a plurality of operating components comprising at least one automation component. The mobile device may comprise combinations of the above-described embodiments of the mobile device according to the first aspect.

The mobile device according to the first aspect and any embodiment thereof as well as the mobile device according to the second aspect may have one or any combination of the following specifications.

The user interface may be configured for displaying an overview of the operating components on a first part of the display device. The user interface may be configured for displaying a representation of the Al copilot module on a second part of the display device that is different from the first part of the display device allowing for concurrent display of the overview of the operating components and the Al copilot module (representation on the display device). Particularly, the Al copilot module may be configured for asking and answering questions with respect to installment, connection and operation of a) the operating components or b) the operating components and additionally to be integrated operating components in a dialogue with the user.

The Al copilot module may be configured for determining presence and connectivity properties of a newly to be integrated operating component. Upon detection of the newly to be integrated operating component the Al copilot module may assist the user in installment and operation of the same, for example, it may automatically establish connection of the newly to be integrated operating component with the Al copilot module and other already operating components.

The mobile device may further comprise a camera device, a loudspeaker device and a microphone for facilitating communication with the user.

The Al copilot module may be configured for at least one of i) learning about operation of the at least one automation component based on an operating manual provided for the at least one automation component, ii) instructing the user based an operating manual provided for the at least one automation component, and displaying an operating manual provided for the at least one automation component on the display device. Knowledge of the content of the operating manual can, thus, directly or indirectly be used by the user.

The mobile device may be configured for displaying an operation screen on the display device allowing the user to directly operate the at least one automation component and other components if available.

Particularly, the mobile device may be configured for recognizing the user, for example, based on a speech input by the user and displaying the operation screen based on recognizing result. Further, the mobile device may be configured for recognizing the user and the Al copilot module may, in this case, be configured for adapting chatting with the user based on the recognizing result. For example, the chat may depend on the experience and/or authority level of the recognized user.

The mobile device may be configured for displaying a pop-up window including a warning or error message related to the at least one automation component on the display device in response to the detection of an event (for example, some failure or danger).

Furthermore, it is provided a factory comprising the at least one automation component, another operating component and the mobile device according to one of the above-described embodiments.

Additional features and advantages of the present invention will be described with reference to the accompanying figures that illustrate embodiments of the claimed invention. Figure 1 illustrates a mobile device connected with an automation component and another operating component according to an embodiment of the present invention.

Figure 2 illustrates screens of a mobile device for controlling operation of an automation component according to an embodiment of the present invention.

Figure 3 shows screen shots of the screens illustrated in Figure 2.

Herein, it is provided a mobile device for controlling operation of one or more automation components (for example, one or more robots) of a factory, wherein the a mobile device is configured for receiving a user’s input via a user interface for controlling the operation, for example, by means of transmitting a control signal to a (robot) controller of the one or more automation components (and possibly to any other operating component of the factory). The mobile device comprises an Al copilot module 3 for assisting the user based on learned content, for example, by chatting with the user. During operation the Al copilot module 3 can be continuously trained for further applications and/or refined assistance of the user. The Al copilot module 3 assists the user in installing, operating, programming, maintaining and repairing automation/operating components of a factory system.

An embodiment of a mobile device 10 for controlling operation of at least one automation component 20 and, optionally, another operating component 30 is shown in Figure 1. The mobile device 10 comprises a user interface 2 comprising a display device 2, for example, a touch screen, allowing reception of a user’s input and display of information related to the operation of the automation component 20 and the other operating component 30. The mobile device 10 allows the management/operation of a plurality of components via the interface 2 and it may be suitable for single-hand operation by the user.

The mobile device 10 comprises a memory and a processing unit 1 configured for program- ming/teaching/running software and an operation system. The mobile device 10 may be a tablet computer, notebook, PDA, smartphone, etc., and may comprise a camera device, a loudspeaker and a microphone. The automation component 20 may be a robot. The other operating component 30 may be a camera device, a grip component, a conveyor (belt) or a light barrier, for example.

The mobile device 10 may wirelessly communicate with the automation component 10 and the other operating component 30 (for example, by means of one of NFC, WLAN, WPAN, and UWB technologies). When the component the mobile device 10 communicated with has a display interface, (content of) this display interface may be streamed on the display device 2b of the mobile device 10. The Al copilot module 3 can refer to this streamed display interface. When the component is a camera device a camera view may be streamed on the display device 2b of the mobile device 10. Further, a controller of the at least one automation component may be configured for streaming data to the display device 2b of the mobile device. The mobile device 10 comprises an Al copilot module 3 configured for providing the user with information on at least one of connectivity and operation of the at least one automation component as well as the other operating component. Connectivity may refer to analog or digital interfaces and wireless or wired communication technologies. The operation may refer to any activity and configuration of the at least one automation component and the other operating component. Particularly, the Al copilot module 3 can be configured for assisting the user in integrating new automation/operating components in an already working system and for at least partly controlling, programming and operating any automation/operating components of the system. Installment of a new component, for example, a camera device, may be achieved step by step based on a chat of the user with the Al copilot module 3 or the Al copilot module 3 may initiate installment of the new component (semi)automatically upon detection of the same.

The Al copilot module 3, in the configuration shown in Figure 1 , can chat with the user by text output displayed in a right-hand side section of the display device 2b (representation of the Al copilot module 3 on the display device). The right-hand side section of the display device 2b may be permanently displayed or it will be temporarily shown based on the control by the processing unit 1.

Further, the Al copilot module 3 may chat with the user by means of a speech input and output. The chat may be based on questions posed by the user and answers given by the Al copilot module 3 and vice versa. For example, based on the actual context the Al copilot module 3 may offer the user a plurality of appropriate questions related to a component under consideration and the user can select one of the questions in order to get an answer to the question from the Al copilot module 3. Software for recognizing a user based on speech input, for example, may run on the processing unit 1 of the mobile device 10 and the chat may be adapted to a particular user recognized by the software for recognizing a user running on the processing unit 1. Thus, the chat with a recognized more experienced and/or more authorized user may differ from the one with a less experienced and/or less authorized user.

It is noted that the Al copilot module 3 may at least partly run on the processing unit 1 of the mobile device 10 and/or may at least partly run on a remote computer device. The Al copilot module 3 may be implemented by a trained neural network and may be based on a neural network transformer architecture. The Al copilot module 3 may, for example, be based on or comprise an adapted version of DALL-E or ChatGPT.

Particularly, the Al copilot module 3 may access a database external to the mobile device 10. For example, the Al copilot module 3 may have access to operation manuals for automation/operating components and may assist the user based on learned content provided by the operation manuals. According to an embodiment the processing unit 1 may display an intelligent keyboard on the display device 2b. The intelligent keyboard provides suggestions for word and/or sentence completions based on letters input by the user via the intelligent keyboard. The suggestions are given by the Al copilot module 3 based on learned context. For example, the choice of suggestions made by the Al copilot module 3 depends on an actual component under consideration and/or the actually recognized user.

By means of the input by the user via the intelligent keyboard or any other text or speech input and as assisted by the Al copilot module 3 an automation/operating component may be installed and/or operated and/or programmed and/or maintained and/or repaired.

The interaction between the user and the Al copilot module 3 may depend on the actual position of the user, i.e., the actual position of the mobile device 10. The actual position of the user/mobile device 10 may at least partly be determined by the mobile device itself, for example, by means of an acceleration sensor comprised by the mobile device 10.

Figure 2 shows exemplary screens displayed on the display device 2b of the mobile device 10. Figure 3 shows screen shots of the respective screens illustrated in Figure 2. In the left-hand side section a of the display device 2b a menu can be displayed that comprises three menu items. The uppermost menu item may function as a home button and button for management of components. Particularly, it may be used for the installment and initiation of operation of a new component (hardware and software) guided by the Al copilot module 3 displayed at the right-hand side of the display device 2b (screen).

The middle menu item (button) gives access to a programming tool. Particularly, the programming tool accesses an operation system of a component under consideration. The bottom item (button) of the menu displayed in the left-hand side section a of the display device 2b allows for starting operation (process control) and upon being pressed may result in opening of an operation screen. The operation screen may be shown in a full screen mode and comprises buttons allowing a user’s input, for example, and may include, for example, some emergency switch, an enabling switch, start switch, and stop switch. The operation screen may include buttons enabling the user to perform all security relevant functions and functions necessary for operation of an automation/operating component.

In principle, the operation screen can be individually configured for particular users (in form of an individualized dashboard, for example). Further, the actual operation screen may be adapted to an actual position of the user/mobile device 10 with respect to a component that is to be operated by inputs received by the operation screen. In a security sensitive environment the operation screen may be different from a non-security sensitive environment. For example, at least some of the operations/functionalities may only be unlocked, if presence of the mobile device 10 is detected at some location within an operation area around the component under consideration.

When the uppermost button the menu is used for the installment and initiation of operation of a new component a list of components may be displayed in a section b adjacent to the menu section a. For example, a new available component is included in the list and can be selected by the user. In response, information related to connectivity properties and other specifications of the component considered relevant may be displayed in a relatively large focus section of the screen/display device 2b. The Al copilot module 3 can guide the user through the procedure of connecting, installing and initiating operation of the new component. In principle, the Al copilot module 3 may be adapted for providing information to the user depending on the content displayed in the focus section.

On the right hand side of Figure 2 screens for direct access to a camera device via the user interface 2 are illustrated. For example, the screen is basically split into a large region c and a region for representation of the Al copilot module 3. In section c a display interface of the camera device can be streamed allowing direct operation of the camera device by the user. Additionally, a camera view can be streamed in section c.

A huge variety of different screen designs may be implemented and adapted to particular applications and user preferences. Permanent or temporary access to the Al copilot module 3 may be provided on any screen available.

All previously discussed embodiments are not intended as limitations but serve as examples illustrating features and advantages of the invention. It is to be understood that some or all of the above-described features can also be combined in different ways.

Claims

1. Mobile device (10) for controlling operation of at least one automation component (20), comprising a processing unit (1) configured for running an operation system; a user interface (2) comprising a display device (2b) and configured for receiving a user’s input for controlling operation of the at least one automation component (20) based on the user’s input; and an Artificial Intelligence, Al, copilot module (3) configured for providing the user with information on at least one of connectivity and operation of the at least one automation component; and wherein the mobile device (10) is configured for programming of a connection of the at least one automation component (20) to another component (30), and the Al copilot module (3) is configured for assisting the user in the programming of the connection of the at least one automation component (20) to the other component.

2. The mobile device (10) according to claim 1, wherein the processing unit (1) is configured for at least partly running the Al copilot module (3).

3. The mobile device (10) according to claim 1 or 2, wherein the processing unit (1) is configured for displaying on the display device (2b), in particular, concurrently, data representing information on the at least one automation component (20) and text output by the Al copilot module (3).

4. The mobile device (10) according to claim 1 or 2, wherein the Al copilot module (3) is configured for chatting with the user by means of the user interface (2) by text and/or speech output based on learned context related to the at least one automation component (20).

5. The mobile device (10) according to one of the preceding claims, wherein the Al copilot module (3) is configured for monitoring operation of the at least one automation component (20) and providing the user with information on the monitored operation.

6. The mobile device (10) according to one of the preceding claims, wherein the mobile device (10) is configured for programming at least one of operation of the at least one automation component (20), and operation of the other component; and the Al copilot module (3) is configured for assisting the user in programming of the at least one of the operation of the at least one automation component (20), and the operation of the other component (30).

7. The mobile device (10) according to claim 6, wherein the Al copilot module (3) is configured for at least partly programming the at least one of a) the connection of the at least one automation component (20) to the other component (30), b) the operation of the at least one automation component (20), and c) the operation of the other component (30).

8. The mobile device (10) according to claim 7, wherein the operation system running on the processing unit (1) is configured for allowing the user access to an operation system of the at least one automation component (20) for programming the operation system of the at least one automation component (20).

9. The mobile device (10) according to claim 8, wherein parts of the operation system of the at least one automation component (20) are integrated into the operation system running on the processing unit (1).

10. The mobile device (10) according to one of the claims 6 to 9, wherein the other component (30) comprises or is connected with a display interface and wherein the mobile device (10) is configured for streaming the display interface on the display device (2b).

11. The mobile device (10) according to one of the claims 6 to 10, wherein the Al copilot module (3) is configured for providing the user with information of at least one of connectivity and operation of the other component.

12. The mobile device (10) according to one of the preceding claims, wherein the processing unit (1) is configured for displaying a virtual keyboard on the display device (2b), wherein the virtual keyboard is configured for receiving a user’s text input and assisting the user in inputting text inputs by suggesting word completions of letters input by the user and wherein the Al copilot module (3) is configured for generating the suggested word completions.

13. The mobile device (10) according to claim 12, wherein the Al copilot module (3) is configured for being at least partly operated based on the text inputs.

14. The mobile device (10) according to one of the preceding claims, wherein the mobile device is configured for determining its actual position or receiving information on its actual position and wherein the Al copilot module (3) is configured for providing the user with the information on at least one of connectivity and operation of the at least one automation component based on the determined actual position of the mobile device (10) or received information on the actual position of the mobile device (10).

15. Use of a mobile device (10) for operating at least one automation component (20), comprising receiving a first user’s input by an interface (2) of the mobile device (10); outputting an output by an Artificial Intelligence, Al, copilot module (3) of the mobile device (10) in response to the first user’s input; receiving a second user’s input different from the first user’s input by the interface (2) in response to the output by the Al copilot module (3); and controlling operation of the at least one automation component (20) based on the second user’s input.