JP2025058941A - Operating device and operating system - Google Patents

Abstract

To provide an operation device and so on capable of preventing a waiting time in processing for establishing communication with an operation target from being wastefully elapsed.SOLUTION: An operation device 20 for operating an operation target comprises: a communication device 250 capable of communicating with the operation target; and a control device 200 capable of executing a communication establishing processing of establishing the communication with the operation target through the communication device, and specified item setting processing of receiving setting of a content of a specified item for supporting the operation of the operation target by the operation device after executing the communication establishing processing, through an input part at least during execution of the communication establishing processing.SELECTED DRAWING: Figure 1

Description

The present invention relates to an operating device and an operating system for operating an object.

A remote control system is disclosed that is configured to selectively remotely control multiple target work machines (to switch the target work machine from a first work machine to another work machine) (see, for example, Patent Document 1).

JP 2021-139144 A

However, in an operation system such as that exemplified in Prior Art 1, it takes a certain amount of time to establish communication between the operation device and the operation target so that they can be operated. While the process for establishing this communication is being executed, the operator has to wait for the communication establishment to be completed, resulting in wasted time.

The object of the present invention is to provide an operating device etc. that can prevent unnecessary waiting time in the process of establishing communication with an operating target.

A first aspect of the present invention provides an operating device for operating an operating object. The operating device includes a communication device capable of communicating with the operating object, a control device capable of executing a communication establishment process for establishing the communication with the operating object through the communication device, and a designated item setting process for receiving, at least during the execution of the communication establishment process, through an input unit, the setting of the contents of designated items for supporting the operation of the operating object by the operating device after the execution of the communication establishment process.

With an operating device having this configuration, while a communication establishment process is being executed, it is possible to set the contents of the designated items to assist the operating device in operating the operation target after the communication establishment process is completed. This makes it possible to utilize the waiting time during the execution of the communication establishment process.

The second aspect of the present invention is an operating device according to the first aspect, wherein the control device executes a communication disconnection process for disconnecting communication established between the communication device and the one operating target when the operating target is switched from one operating target to another operating target, and a communication establishment process for establishing communication between the operation device and the other operating target, and the control device preferably accepts the setting of the content of the specified item through the input unit when at least one of the communication disconnection process and the communication establishment process is executed in association with the switching of the operating target through the communication device.

According to the operation device having this configuration, when the operation target is switched from one operation target to another operation target, a communication disconnection process and a communication establishment process are executed. The "communication disconnection process" is a process for disconnecting the communication established between the operation device and the one operation target. The "communication establishment process" here is a process for establishing communication between the operation device and the other operation target. When at least one of the "communication disconnection process" and "the communication establishment process" is executed, it becomes possible to set the contents of the specified items via the input unit. Therefore, the waiting time during the execution of the process for switching the operation target is utilized.

In the third aspect of the present invention, in the operating device according to the first or second aspect, it is preferable that when the operation target is switched from the one operation target to the other operation target having specifications different from those of the one operation target, the control device accepts, via the input unit, the setting of the content of the specified item corresponding to the difference in the specifications.

According to the operation device having this configuration, when an operation target is switched from one operation target to another operation target having specifications different from those of the first operation target, it becomes possible to set, via the input unit, the specified items corresponding to the differences in the specifications. This makes it possible to smoothly operate the other operation target after the operation target is switched.

The fourth aspect of the present invention is an operating device according to any one of the first to third aspects, further comprising a simulation device for operating a virtual operating object corresponding to the operating object in a virtual space, and it is preferable that the control device assists in operating the virtual operating object in the virtual space based on the setting contents of the specified items received through the input unit.

With an operation device having this configuration, it is possible to virtually experience, through the simulation device, the support mode and/or operation mode of the operation target that is predicted to occur according to the settings of the items specified through the input unit. Based on the experience, it is possible to reset or change the settings of the items specified through the input unit.

The operating device according to the fifth aspect of the present invention is preferably an operating device according to any one of the first to fourth aspects, further comprising a storage device that stores the setting contents of the specified items, and an output unit that outputs the setting contents of the specified items stored in the storage device.

The operating device having this configuration allows an operator to check the current settings of the specified items output to the output unit. Based on the output contents, it becomes possible to reset or change the settings of the specified items via the input unit.

The sixth aspect of the present invention is an operating device according to the fifth aspect, which has a function of authenticating an operator, and the output unit preferably outputs the setting contents of the specified items stored in the storage device in association with the authenticated operator.

According to the operation device having this configuration, the contents of the specified items last set by each operator (or the contents of the specified items appropriate for each operator) are stored and output to the output unit. The output contents may be defined as the default contents of the specified items.

The seventh aspect of the present invention is an operating device according to the fifth aspect, which has a function of authenticating an operator, and the storage device preferably stores the setting contents of the specified items received through the input unit in association with the authenticated operator.

With an operating device of this configuration, the contents of the specifications last set by each operator can be stored in a storage device.

The eighth aspect of the present invention is an operating device according to any one of the first to seventh aspects, in which the control device preferably accepts settings for the content of a selected one of the multiple designated items via the input unit.

According to the operation device having this configuration, only the content of one or more designated items selected (through the input unit) from among the multiple designated items is set through the input unit.

The present invention provides an operation system, which is configured with an operation device according to any one of the first to eighth aspects and the operation target.

FIG. 1 is a configuration diagram of an operation system according to an embodiment of the present invention. FIG. 2 is a perspective view of a remote control device according to an embodiment of the present invention. FIG. 3 is a perspective view of a work machine according to an embodiment of the present invention. FIG. 4 is a flowchart of an operation system according to an embodiment of the present invention. FIG. 5 is a flowchart of the authentication processing function of the operation system according to an embodiment of the present invention. FIG. 6 is a flowchart of the operation processing function of the operation system according to one embodiment of the present invention. FIG. 7A is a diagram showing a first authentication screen for an operator in the operation system according to one embodiment of the present invention. FIG. 7B is a diagram showing a second authentication screen for an operator in the operation system according to one embodiment of the present invention. FIG. 8A is a diagram showing a work content designation screen in the operation system according to one embodiment of the present invention. FIG. 8B is a diagram showing an operation target designation screen in the operation system according to one embodiment of the present invention. FIG. 8C is a diagram showing a confirmation screen for the work content and the operation target in the operation system according to one embodiment of the present invention. FIG. 9A is a diagram showing a first lever pattern setting screen in the operation system according to one embodiment of the present invention. FIG. 9B is a diagram showing a second lever pattern setting screen in the operation system according to one embodiment of the present invention. FIG. 10 is a diagram showing a lever sensitivity setting screen in the operation system according to one embodiment of the present invention. FIG. 11 is a diagram showing a screen for setting the seat vibration mode in the operation system according to one embodiment of the present invention. FIG. 12 is a diagram showing a screen for setting an operation assist in the operation system according to one embodiment of the present invention. FIG. 13 is a diagram showing a guidance setting screen in the operation system according to one embodiment of the present invention. FIG. 14 is a diagram showing a depth sense image in the operation system according to one embodiment of the present invention. FIG. 15 is a diagram showing a terrain image in an operation system according to one embodiment of the present invention. FIG. 16A is a diagram showing a first button function setting screen in the operation system according to one embodiment of the present invention. FIG. 16B is a diagram showing a second button function setting screen in the operation system according to one embodiment of the present invention. FIG. 17 is a diagram showing an operation selection screen in an operation system according to one embodiment of the present invention.

(Operation System Configuration)
FIG. 1 is a configuration diagram of an operation system according to an embodiment of the present invention. This operation system has an information processing terminal 10 and a remote control device 20 (operation device), and is configured to remotely operate a work machine 40 (operation target) using the remote control device 20. In another embodiment, the operation system may include the work machine 40. The information processing terminal 10 and the remote control device 20 are configured to be able to communicate with each other via a first communication network. The remote control device 20 and the work machine 40 are configured to be able to communicate with each other via a second communication network. The first communication network and the second communication network may have the same communication standard or different standards. The information processing terminal 10 may be omitted. The remote control device 20 may have the functions of the information processing terminal 10 built in. The "operation target" may be any device having a function of performing work or tasks by remote control, such as a vehicle, a tiller, or an airplane, in addition to the work machine 40.

(Configuration of information processing terminal)
As shown in FIG. 1, the information processing terminal 10 has a terminal control device 100 (control device, control unit), a terminal input interface 110 (input interface, input unit), a terminal output interface 120 (output interface, output unit), and a terminal wireless communication device 150 (communication device) for communicating via a first communication network. The information processing terminal 10 is composed of a smartphone, tablet terminal, or notebook computer carried by an operator or mounted on a remote control device 20. The terminal control device 100 is composed of an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing according to the software on the data. The terminal input interface 110 and the terminal output interface 120 are composed of, for example, a touch panel display.

Also, as described above, the information processing terminal 10 may be a component of the remote control device 20. In this case, the information processing terminal 10 and the remote control device 20 may be configured to be capable of wired or wireless communication, the terminal input interface 110 may constitute a part of the remote input interface 210, and the terminal output interface 120 may constitute a part of the remote output interface 220.

(Configuration of remote control device)
1, the remote operation device 20 has a remote control device 200 (control device, control unit), a remote input interface 210 (input interface, input unit), a remote output interface 220 (output interface, output unit), and a remote wireless communication device 250 (communication device) capable of communicating with the information processing terminal 10 or the work machine 40 via a first communication network and a second communication network. The remote wireless communication device 250 is assigned an operation device identifier (IP address) for identifying the remote wireless communication device 250 in the second communication network and setting it as a transmission destination. The remote control device 200 is composed of an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing according to the software on the data.

The remote input interface 210 has a remote operation mechanism 211 and a face authentication sensor 212. The remote output interface 220 has a remote image output device 221, a remote audio output device 222, and a seat drive device 224.

2 is a perspective view of the remote control mechanism 211 according to this embodiment. The remote control mechanism 211 has the devices and members shown in FIG. 2, which include a travel operation device, a swivel operation device, a boom operation device, an arm operation device, and a tip attachment operation device. Each operation device has an operation lever that receives a rotation operation. The operation lever (travel lever) of the travel operation device is operated to drive a hydraulic travel motor (actuator) that constitutes the travel drive mechanism in the lower travel body 41 of the work machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever may be provided. The operation lever (swivel lever) of the swivel operation device is operated to move a hydraulic swivel motor (actuator) that constitutes the swivel drive mechanism of the work machine 40. The operation lever (boom lever) of the boom operation device is operated to move the boom cylinder 442 of the work machine 40. The operating lever (arm lever) of the arm operating device is operated to move the arm cylinder 444 of the work machine 40. The operating lever (tip attachment lever) of the tip attachment operating device is operated to move the tip attachment cylinder 446 of the work machine 40. An optional pedal that is operated to move the tip attachment of the work machine 40 may be arranged on the side of the travel pedal.

The operation levers constituting the remote operation mechanism 211 are arranged around the seat 24 on which the operator sits, as shown in FIG. 2, for example. The seat 24 has a seat part 241 installed on the upper side of the seat base 240 and a backrest part 242, and is in the form of a high-back chair with armrests. The seat 24 may be in the form of a low-back chair without a headrest, or in the form of a chair without the backrest part 242, or in any other form on which the operator can sit. The seat 24 and/or the side of the seat 24 may be configured to be able to mount the information processing terminal 10. In this case, it is preferable that the touch screen of the information processing terminal 10 is mounted facing the seat 24 so that the display of the information processing terminal 10 can be easily operated and viewed by the operator seated on the seat 24. In addition, a cut-off lever (not shown) may be mounted on the seat 24 and/or the side of the seat 24. The cut-off lever has a function of prohibiting the operation of the work machine (stopping the actuator operation) when tilted from the unlocked position to the locked position.

Left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat 24. One operating lever may serve as multiple operating levers. For example, the left operating lever 2111 provided in front of the left frame of the seat 24 shown in FIG. 2 may function as an arm lever when operated in the forward/backward direction, and as a rotation lever when operated in the left/right direction. Similarly, the right operating lever 2112 provided in front of the right frame of the seat shown in FIG. 2 may function as a boom lever when operated in the forward/backward direction, and as a tip attachment lever when operated in the left/right direction. Such lever patterns may be changed as desired by the operator's operating instructions.

As shown in FIG. 2, the face authentication sensor 212 is composed of a camera and/or a distance measuring sensor disposed in front of the seat 24. The authentication process for the operator is performed based on the image of the operator's face captured by the camera and/or the three-dimensional shape of the operator's face captured by the distance measuring sensor.

The remote image output device 221 is composed of a single or multiple image output devices. For example, as shown in FIG. 2, the remote image output device 221 is composed of a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212, each having a substantially rectangular screen, which are arranged in front of the seat 24, diagonally forward to the left, and diagonally forward to the right. The shape and size of the screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be the same or different. The remote image output device 221 may be composed of a single curved or bendable image output device, or two or four or more image output devices arranged to surround the front of the seat 24.

As shown in FIG. 2, the right edge of the left remote image output device 2211 is adjacent to the left edge of the central remote image output device 2210 such that the screens of the central remote image output device 2210 and the left remote image output device 2211 form a first specified angle (e.g., 120° to 150°). As shown in FIG. 2, the left edge of the right remote image output device 2212 is adjacent to the right edge of the central remote image output device 2210 such that the screens of the central remote image output device 2210 and the right remote image output device 2212 form a second specified angle (e.g., 120° to 150°).

The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel to the vertical direction or inclined to the vertical direction. At least one of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be composed of multiple divided image output devices. For example, the central remote image output device 2210 may be composed of adjacent image output devices above and below, each having a roughly rectangular screen.

The remote audio output device 222 is composed of one or more speakers, and is composed of audio output devices arranged, for example, behind the seat 24, behind the left armrest, and behind the right armrest. The specifications of the audio output devices may be the same or different. The remote audio output device 222 may be configured to output the sound of the work machine 40 collected by a microphone mounted on the work machine 40.

The seat drive device 224 and the seat 24 constitute a "seat device." The seat drive device 224 is configured to tilt or swing the seat 24 relative to the seat base 240 (or integrally with the seat base 240) about the pitch axis and/or the roll axis (about the roll axis as necessary). The seat drive device 224 may be configured to vibrate the seat 24 relative to the seat base 240 (or integrally with the seat base 240) in at least one direction among the pitch axis, the roll axis, and the yaw axis.

(Configuration of the work machine)
As shown in Fig. 1, the work machine 40 has a real machine control device 400, a real machine input interface 410, a real machine output interface 420, and a real machine wireless communication device 450 for communicating via the second communication network. The real machine wireless communication device 450 is assigned a real machine identifier (IP address) for identifying the real machine wireless communication device 450 in the second communication network and making it a transmission destination. Each of the components of the real machine control device 400 is composed of an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing according to the software on the data. The remote control device 200 communicates with a specific work machine 40 selected from a plurality of work machines 40 via the remote wireless communication device 250, thereby making it possible to operate the specific work machine 40 by the remote input interface 210.

Figure 3 is a perspective view of a work machine 40 according to this embodiment. The work machine 40 is, for example, a hydraulic excavator, and as shown in Figure 3, includes a crawler-type lower traveling body 41 constituting a traveling drive mechanism, and an upper rotating body 42 that is rotatably mounted on the lower traveling body 41 via a slewing drive mechanism. A cab 43 (operator's cabin) is provided on the front left side of the upper rotating body 42. A work mechanism 44 is provided on the front right side of the upper rotating body 42. The cab 43 may be configured to be driven to move up and down relative to the upper rotating body 42 by a lifting drive mechanism (not shown) having a link mechanism and an actuator, and may be configured to change the tilt angle relative to the upper rotating body 42 by a tilt drive mechanism (not shown) having an actuator.

The real machine input interface 410 has a real machine operation mechanism 411, a real machine imaging device 412, and a real machine sensor group 416. The real machine operation mechanism 411 has a plurality of operation levers arranged around a seat arranged inside the cab 43 in the same manner as the remote operation mechanism 211. An operating mechanism or robot that receives a signal according to the operation mode of the remote operation lever and moves the real machine operation lever based on the received signal is provided in the cab 43. The real machine operation mechanism 411 may be configured by an operating mechanism that moves a control valve that constitutes a hydraulic circuit mounted on the work machine 40. The real machine imaging device 412 is installed, for example, inside the cab 43, and images the environment in front of the cab 43, including at least a part of the working mechanism 44, through the front window and the left and right side windows. Some or all of the front window and the side windows may be omitted. The real machine imaging device 412 may be mounted outside the cab 43 as long as it images the environment in front of the cab 43, including at least a part of the working mechanism 44.

The actual machine sensor group 416 includes various sensors configured to output signals to the actual machine control device 400 according to parameter values related to the state of the work machine 40. The state of the work machine 40 includes the actual machine attitude detected by the actual machine attitude sensor, that is, the tilt angle (pitch angle) around the pitch axis and/or the tilt angle (roll angle) around the roll axis with respect to the vertical direction of the upper rotating body 42. The actual machine attitude sensor is configured, for example, with a three-dimensional acceleration sensor. The state of the work machine 40 may include the presence or absence of operation of the prime mover (equipment that generates vibrations such as an engine or a work oil pump), as well as the hoisting angle of the boom 441 relative to the upper rotating body 42, the rotation angle of the arm 443 relative to the boom 441, the rotation angle of the tip attachment 445 (lift magnet) relative to the arm 443, the operating oil pressure, the remaining fuel amount, etc. The state of the work machine 40 may include the sound of the work machine 40 collected by a microphone mounted on the work machine 40.

As shown in FIG. 3, the working mechanism 44 (operating mechanism) has a boom 441 whose base end is movably attached to the upper rotating body 42, an arm 443 whose base end is rotatably connected to the tip of the boom 441, and a tip attachment 445 whose base end is rotatably connected to the tip of the arm 443. The working mechanism 44 is equipped with a boom cylinder 442, an arm cylinder 444, and a tip attachment cylinder 446 as actuators composed of extendable hydraulic cylinders.

As the tip attachment 445, in addition to a bucket having a container for storing objects such as soil and sand, a lifting magnet, a grapple that grasps an object by opening and closing multiple opposing gripping claws, a bucket with a hook that can hook an object to a hook attached to the back and lift it, or a breaker with a chisel that is supported by the breaker body so as to be capable of reciprocating and striking the work object may be used. The chisel of the breaker may be configured to be operable by a switch. In addition, a swing mechanism that swings the tip end left and right relative to each base end, or a slide arm that is provided with a sliding extension mechanism on the arm 443 and is configured to be extendable and retractable, may be provided at the base end of the boom 441 and the base end of the arm 443, and may be configured to be operable by a switch or an optional pedal. A dozer having a blade lifting drive mechanism that drives the blade to lift and lower by a dozer cylinder is provided at the front side of the lower traveling body 41, and may be configured to be operable by a switch or an optional pedal.

The boom cylinder 442 is interposed between the boom 441 and the upper rotating body 42 so as to extend and retract when supplied with hydraulic oil, thereby rotating the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to extend and retract when supplied with hydraulic oil, thereby rotating the arm 443 around a horizontal axis relative to the boom 441. The tip attachment cylinder 446 is interposed between the tip attachment 445 and the arm 443 so as to extend and retract when supplied with hydraulic oil, thereby rotating the tip attachment 445 around a horizontal axis relative to the arm 443.

The real machine control device 400 is provided with a switch that can switch between the presence and absence of power supply to the real machine control device 400. The switch switches from a state in which there is no power supply to the real machine control device 400 to a state in which there is power supply to the real machine control device 400, and the real machine control device 400 starts operating. When the real machine control device 400 starts operating, the real machine control device 400 starts the real machine wireless communication device 450, and the real machine wireless communication device 450 connects to the first communication network and the second communication network. In addition, the real machine control device 400 that has started operating starts the real machine imaging device 412 and the real machine sensor group 416. In a state in which there is no power supply to the real machine control device 400 due to the switch, the real machine control device 400 stops operating, the real machine wireless communication device 450 is stopped, and communication with the information processing terminal 10 or the work machine 40 via the first communication network and the second communication network cannot be performed. In addition, when the real machine control device 400 stops operating, the operation of the real machine imaging device 412 and the real machine sensor group 416 stops.

(function)
Fig. 4 is a flowchart of an operation system according to an embodiment of the present invention. Fig. 5 is a flowchart of an authentication processing function of the operation system according to this embodiment. Fig. 6 is a flowchart of an operation processing function of the operation system according to this embodiment. Fig. 7A is a diagram showing a first authentication screen for an operator in the operation system according to this embodiment. Fig. 7B is a diagram showing a second authentication screen for an operator in the operation system according to this embodiment. The functions of the remote operation device 20 and the remote operation system having the above configuration will be described with reference to Figs. 4 to 7B.

When the operation system starts processing, first, the remote control device 200 executes authentication processing (FIG. 4/STEP 200). Specifically, as shown in FIG. 7A, a message Q01 prompting the operator to sit in the seat 24 and an authentication button Q02 are output to the remote output interface 220 (remote image output device 221) (output, display by the remote output interface 220 (same hereinafter)). The various buttons output to the remote output interface 220 (remote image output device 221) form a touch panel type input interface.

When the authentication button Q02 is tapped, etc., as shown in FIG. 7B, a face authentication button Q03 and an identifier authentication button Q04 are output to the remote output interface 220 (remote image output device 221). When the face authentication button Q03 is tapped, for example, a camera and/or a distance sensor mounted on the remote control device 20 and/or the information processing terminal is used to perform face authentication of the operator (and even by querying a database). When the identifier authentication button Q04 is tapped, the operator authentication is performed using the operator identifier (or operator identifier (user ID) and passcode) input through the remote input interface 210. If the authentication is successful, an OK button Q05 is output to the remote output interface 220 (remote image output device 221) together with information that the authentication was successful. If the authentication is unsuccessful, information that the authentication was unsuccessful is output to the remote output interface 220 (remote image output device 221). If the authentication fails, the face authentication button Q03 and the identifier authentication button Q04 may be output again to the remote output interface 220 (remote image output device 221).

Details of the authentication process executed by the remote control device 200 will be described with reference to the flowchart in FIG. 5. The authentication process may be executed steadily or periodically as background processing. Specifically, first, an operator identifier (including captured image data and/or distance measurement data for face authentication) is acquired (FIG. 5/STEP 202). If an operator identifier based on face authentication is not acquired for a certain period of time, an operation history indicating that the operator is away from his/her seat (not seated in the seat 24) is acquired along with the last acquired operator identifier.

It is determined whether or not an operator's change notification has been accepted through the remote input interface 210 (or the change notification input interface (touch button) that constitutes it) (FIG. 5/STEP 240). If the determination result is positive (YES in FIG. 5/STEP 240), the first flag f1 indicating the presence or absence of a change notification is set to "1" (FIG. 5/STEP 241). On the other hand, if the determination result is negative (NO in FIG. 5/STEP 240), the first flag f1 is set to "0" (FIG. 5/STEP 242).

Next, it is determined whether the operator identifier has changed (whether the operator identifier over the most recent multiple times is different from the operator identifier from the previous times) (FIG. 5/STEP 250). If the determination result is positive (YES in FIG. 5/STEP 250), the second flag f2 indicating whether the operator identifier has changed is set to "1" (FIG. 5/STEP 251). For example, if an operator leaves his/her seat and another operator sits in the seat 24, the newly acquired operator identifier is different from the last acquired operator identifier, so the determination result is positive. On the other hand, if the determination result is negative (NO in FIG. 5/STEP 250), the second flag f2 is set to "0" (FIG. 5/STEP 252).

Furthermore, it is determined whether the first flag f1 and the second flag f2 are equal (FIG. 5/STEP 254). If the determination result is negative (NO in FIG. 5/STEP 254), that is, if an operator change report has been made but the operator identifier has not changed, or if an operator change report has not been made but the operator identifier has changed, the authentication flag f indicating the success or failure of authentication is set to "0" (FIG. 5/STEP 262).

On the other hand, if the determination result is positive (FIG. 5/STEP 254: YES), that is, if an operator change has been reported and the operator identifier has changed, or if an operator change has not been reported and the operator identifier has not changed, the success or failure of authentication is determined (FIG. 5/STEP 260). Specifically, the success or failure of authentication is determined depending on whether the latest operator identifier is registered in the storage device constituting the remote control device 200 and/or an external database. If the determination result is positive (FIG. 5/STEP 260: YES), the authentication flag f is set to "1" (FIG. 5/STEP 261). On the other hand, if the determination result is negative (FIG. 5/STEP 260: NO), the authentication flag f is set to "0" (FIG. 5/STEP 262). The authentication process ends when the authentication flag f is set.

The order of the process of determining whether or not an operator change notification has been received and the associated process related to the first flag f1 (see FIG. 5/STEP 240, STEP 241, and STEP 242) and the process of determining whether or not an operator identifier has changed and the associated process related to the second flag f2 (see FIG. 5/STEP 250, STEP 251, and STEP 252) may be reversed, or may be parallel processes. The process of determining whether or not an operator change notification has been received through the remote input interface 210 and the associated process related to the first flag f1 may be omitted (see FIG. 5/STEP 240, STEP 241, STEP 242, and STEP 254). The process of determining whether or not an operator identifier has changed and the associated process related to the second flag f2 may be omitted (see FIG. 5/STEP 250, STEP 251, and STEP 252). The comparison process between the first flag f1 and the second flag f2 may be omitted (see FIG. 5/STEP 254).

Figure 8A is a diagram showing a work content designation screen in the operation system according to this embodiment. After the operator is authenticated by the authentication process in STEP 200 in Figure 4, when the OK button Q05 is tapped, the operation target and work content (scheduled work content) are designated or selected (Figure 4/STEP 204). Specifically, when the OK button Q05 is tapped, as shown in Figure 8A, work content selection windows Q11 and Q12 and a back button Q13 are output to the remote output interface 220 (remote image output device 221). Each of the multiple (e.g., two) work content selection windows Q11 and Q12 displays, for example, the work name, the name of the office involved in the work, the work location, the characteristics of the work target (materials, etc.), and the work period. When the back button Q13 is tapped, the screen shown in Figure 7A or Figure 7B is output to the remote output interface 220 (remote image output device 221).

Figure 8B is a diagram showing an operation target designation screen in the operation system according to this embodiment. When one of the above-mentioned work content selection windows Q11 and Q12 is tapped, as shown in Figure 8B, operation target candidate selection windows Q14 and Q15 and a back button Q16 are output to the remote output interface 220 (remote image output device 221). Each of the multiple (e.g., two) operation target candidate selection windows Q14 and Q15 displays, for example, the model number, name, and specification information of the work machine 40, which is the operation target candidate. The model number, name, and specification information of the operation target candidate are associated with the actual machine identifier for identifying the actual machine wireless communication device 450 possessed by the work machine 40 (operation target) and are registered in the storage device constituting the remote control device 200 and/or an external database. Of the work machines 40 to be operated, only operation target candidates that have been switched to a state in which power is being supplied to the actual machine control device 400 by the switch can be selected, and operation target candidates that have not been switched to a state in which power is being supplied are displayed in a grayed-out state, and windows Q14 and Q15 are output to the remote output interface 220 (remote image output device 221) so that they will not accept selection even if tapped, etc. When the back button Q16 is tapped, etc., the screen shown in FIG. 8A is output to the remote output interface 220 (remote image output device 221).

Depending on the operator authenticated by the authentication process (see FIG. 4/STEP 200), multiple operation target candidates may be searched for from the storage device constituting the remote control device 200 and/or an external database, and multiple operation target candidate selection windows showing the model numbers of the multiple operation target candidates may be output to the remote output interface 220 (remote image output device 221). As a result, a new operation target may be specified by the operator through the remote input interface 210 according to his/her intention from among the operation target candidates output to the remote output interface 220 (remote image output device 221) according to the operator. Note that the operation target candidates to be used by each operator according to the operator identified by the operator identifier are preregistered in the storage device and/or external database.

Depending on the type of work content (scheduled work content) selected from the multiple work content selection windows, multiple operation target candidates may be searched for from the storage device constituting the remote control device 200 and/or an external database, and multiple operation target candidate selection windows showing the model numbers of the multiple operation target candidates may be output to the remote output interface 220 (remote image output device 221). As a result, a new operation target may be specified by the operator through the remote input interface 210 according to his/her intention from among the operation target candidates output to the remote output interface 220 (remote image output device 221) according to the type of planned work content. Note that the storage device and/or external database pre-registers the planned work content of each operator and the operation target candidates to be used when executing the planned work content according to the type of operator identified by the operator identifier.

Figure 8C is a diagram showing a confirmation screen for work content and operation target in the operation system according to this embodiment. When one of the operation target selection windows Q14 and Q15 is tapped or the like, a selection result confirmation window Q17, a back button Q18, and an operation start button Q19 are output to the remote output interface 220 (remote image output device 221) as shown in Figure 8C. The selection result confirmation window Q17 displays, for example, the selected work content and information on the work machine 40 that is the operation target. When the back button Q18 is tapped or the like, the screen shown in Figure 8B is output to the remote output interface 220 (remote image output device 221).

Next, an operation to start using the operation target is accepted (FIG. 4/STEP 206). Specifically, the operation start button Q19 shown in FIG. 8C is tapped, etc.

In response to this, it is determined whether there is an existing operation target for the remote control device 200 (FIG. 4/STEP 208). Specifically, it is determined whether there is a work machine 40 that is an operation target and in a state where communication is established with the remote control device 20. When communication with the operation target is established and/or when the established communication is released, the value of the flag may be changed, and the value of the flag may be used for this determination.

If it is determined that there is no existing operation target (NO in FIG. 4/STEP 208), the remote control device 200 starts the execution of the communication establishment process as an "operation preparation process" (FIG. 4/STEP 210). First, the actual machine identifier corresponding to the work machine 40 selected as the operation target by the designation command is obtained from the actual machine identifiers associated with each of a plurality of operation target candidates registered in the storage device and/or an external database.

Next, the remote control device 200 transmits a communication establishment request signal to the destination identified and indicated by the actual machine identifier corresponding to the actual machine wireless communication device 450 of the work machine 40 selected as the operation target, requesting the establishment of communication with the work machine 40 that is the operation target. The communication establishment request signal includes the operation device identifier corresponding to the remote wireless communication device 250 possessed by the remote operation device 20.

When the actual machine control device 400 mounted on the work machine 40 to be operated receives the communication establishment request signal through the actual machine wireless communication device 450, it transmits an approval signal indicating that the request for communication establishment has been approved to the remote wireless communication device 250 possessed by the remote operation device 20, which is the destination identified and indicated by the operation device identifier contained in the communication establishment request signal.

When the remote control device 200 receives the approval signal through the remote wireless communication device 250, it transmits a start-up signal to start the actual machine imaging device 412 and the actual machine sensor group 416 mounted on the work machine 40 to the destination identified and indicated by the actual machine identifier, and waits until it receives a signal corresponding to the actual machine operating status regarding the image acquired by the actual machine imaging device 412 through the remote wireless communication device 250 and the status of the work machine 40 acquired by the actual machine sensor group 416.

When the actual machine control device 400 receives the activation signal through the actual machine wireless communication device 450, it activates the actual machine imaging device 412 and the actual machine sensor group 416, starts transmitting signals corresponding to the actual machine operating status regarding the image acquired by the actual machine imaging device 412 and the state of the work machine 40 acquired by the actual machine sensor group 416 to a destination identified and indicated by the operation device identifier, and starts receiving the signals corresponding to the operation mode of the remote control lever for controlling the operation of the operating mechanism through the actual machine wireless communication device 450. Note that the signals corresponding to the actual machine operating status include information related to the actual machine identifier corresponding to the actual machine wireless communication device 450.

The remote control device 200 outputs the captured images and actual machine operation status received through the remote wireless communication device 250 to the remote output interface 220. For example, the remote control device 200 outputs the received captured images to the remote image output device 221, outputs the sound of the work machine 40 included in the received actual machine operation status to the remote sound output device 222, and controls the operation of the seat drive device 224 according to the tilt angle included in the received actual machine operation status.

As described above, the actual machine control device 400 executes a process for controlling at least the communication device and establishing the communication with the operation target as an operation preparation process. By executing the communication establishment process, communication is established between the remote wireless communication device 250 of the work machine 40 selected as the operation target and the remote wireless communication device 250 for remotely operating the work machine 40 (operation target) by the remote operation device 20.

When there is a plan to perform communication establishment processing between the work machine 40 to be operated and the remote operation device 20, the remote control device 200 may transmit a communication establishment request signal before the time to execute the communication establishment processing. In this case, the communication establishment request signal includes the operation device identifier corresponding to the remote wireless communication device 250 possessed by the remote operation device 20 and information regarding the time to establish the communication establishment processing. The actual machine control device 400 mounted on the work machine 40 to be operated stores the operation device identifier contained in the communication establishment request signal received through the remote wireless communication device 250 in a storage device mounted on the work machine 40, reads out the operation device identifier from the storage device when it is time to establish the communication establishment processing, and transmits the approval signal indicating that the request for communication establishment has been approved to the remote wireless communication device 250 possessed by the remote operation device 20, which is the transmission destination identified and indicated by the read out operation device identifier.

In addition, the operation device identifier and the actual machine identifier may be assigned something other than an IP address as long as they can identify the remote wireless communication device 250 and the actual wireless communication device 450 in the second communication network. For example, a wireless LAN access point or the like may be assigned to the remote wireless communication device 250 and the actual wireless communication device 450, such as an SSID (Service Set Identifier). The operation device identifier and the actual machine identifier may be configured to correspond to, for example, a plurality of letters, numbers and/or symbols representing the serial numbers of components such as an engine and/or a battery mounted on the work machine 40 together with the actual wireless communication device 450, so that the remote wireless communication device 250 and the actual wireless communication device 450 can be identified in the second communication network. In communication, an authenticator is generated according to a designation method based on the operation device identifier and the actual machine identifier. An authenticator may be generated according to a different designation method depending on the difference in the data or letters, numbers and symbols constituting the operation device identifier and the actual machine identifier. The remote control device 200 then uses the generated authenticator to establish wireless communication with the remote wireless communication device 250 and the actual wireless communication device 450 identified by the communication identifier.

On the other hand, if it is determined that there is an existing operation target for the remote control device 200 (FIG. 4/STEP 208: YES), the remote control device 200 starts the execution of a communication disconnection process as an "operation preparation process" in addition to the communication establishment process (FIG. 4/STEP 212).

In the communication establishment process in the operation preparation process, the remote control device 200 first acquires the actual machine identifier corresponding to the work machine 40 (other operation target) selected as the operation target by the designation command from the actual machine identifiers associated with each of the multiple operation target candidates registered in the storage device and/or external database.

Next, the remote control device 200 transmits a communication establishment request signal to the destination identified and indicated by the actual machine identifier corresponding to the actual machine wireless communication device 450 of the work machine 40 selected as the operation target, requesting the establishment of communication with the work machine 40 that is the operation target.

When the actual machine control device 400 mounted on the work machine 40 to be operated receives the communication establishment request signal through the actual machine wireless communication device 450, it transmits an approval signal indicating that the request for communication establishment has been approved to the remote wireless communication device 250 possessed by the remote operation device 20, which is the destination identified and indicated by the operation device identifier contained in the communication establishment request signal.

When the remote control device 200 receives the approval signal through the remote wireless communication device 250, it transmits a start-up signal to start the actual machine imaging device 412 and the actual machine sensor group 416 mounted on the work machine 40 to the destination identified and indicated by the actual machine identifier, and waits until it receives a signal corresponding to the actual machine operating status regarding the image acquired by the actual machine imaging device 412 through the remote wireless communication device 250 and the status of the work machine 40 acquired by the actual machine sensor group 416.

When the actual machine control device 400 receives the activation signal through the actual machine wireless communication device 450, it activates the actual machine imaging device 412 and the actual machine sensor group 416, starts transmitting signals corresponding to the actual machine operating status regarding the image acquired by the actual machine imaging device 412 and the state of the work machine 40 acquired by the actual machine sensor group 416 to a destination identified and indicated by the operation device identifier, and starts receiving the signals corresponding to the operation mode of the remote control lever for controlling the operation of the operating mechanism through the actual machine wireless communication device 450. Note that the signals corresponding to the actual machine operating status include information related to the actual machine identifier corresponding to the actual machine wireless communication device 450.

The remote control device 200 outputs the captured images and actual machine operation status received through the remote wireless communication device 250 to the remote output interface 220.

As described above, as a communication establishment process in the operation preparation process, the remote control device 200 executes a process to establish the communication with the operation target through the communication device.

In the communication disconnection process in the operation preparation process, the remote control device 200 first stops outputting to the remote output interface 220 captured images and actual machine operating status of the work machine 40 (first work machine) that is currently being operated and received through the remote wireless communication device 250, and further stops transmitting to the work machine 40 (first work machine) a signal corresponding to the operation mode of the remote control lever to the work machine 40 (first work machine).

Next, the remote control device 200 transmits a communication disconnection request signal to request disconnection of the established communication to the destination identified and indicated by the actual machine identifier corresponding to the actual machine wireless communication device 450 of the work machine 40 that is the existing target of operation.

When the actual machine control device 400 mounted on the existing work machine 40 to be operated receives the communication disconnection request signal via the actual machine wireless communication device 450, it executes a process to turn off the power to the actual machine imaging device 412 and the actual machine sensor group 416 that have been activated. Then, the actual machine control device 400 performs a process to cut off the power supply being supplied to the actual machine control device 400.

As described above, the actual machine control device 400 executes a communication disconnection process as an operation preparation process to release the communication establishment state between the remote control device 20 and the work machine 40 in a communication establishment state. In addition, during this communication disconnection process, the power supply state to the actual machine control device 400 mounted on the work machine 40 may be maintained (processing to cut off the power supply supplied to the actual machine control device 400 may not be performed). As a result, the actual machine control device 400 can receive the signal according to the operation mode of the remote control lever for controlling the operation of the operating mechanism through the actual machine wireless communication device 450, and when the actual machine imaging device 412 and the actual machine sensor group 416 are started again, a state in which it is possible to transmit the captured image acquired by the actual machine imaging device 412 and the signal according to the actual machine operating status regarding the state of the work machine 40 acquired by the actual machine sensor group 416 to the remote control device 20 through the actual machine wireless communication device 450 may be maintained.

The communication establishment process and the communication disconnection process may be processed in parallel or in parallel. When the communication establishment process and the communication disconnection process are processed in parallel or in parallel, the remote control device 200 performs the process of acquiring the actual machine identifier corresponding to the work machine 40 (another operation target), while simultaneously starting the process of stopping the output of the captured image and the actual machine operation status of the work machine 40 (one work machine) to the remote output interface 220 and the process of stopping the transmission to the work machine 40 (one work machine) of a signal corresponding to the operation mode of the remote operation lever to the work machine 40 (one work machine). Then, the remote control device 200 proceeds with the communication establishment process and the communication disconnection process by processing them in parallel or in parallel.

Here, either the process of turning off the power supply of the real machine imaging device 412 and the real machine sensor group 416 of the work machine 40 (one work machine) which is an existing operation target, or the process of starting up the real machine imaging device 412 and the real machine sensor group 416 of the work machine 40 (another operation target) may be executed first. That is, the remote control device 200 may be temporarily in a state of receiving captured images and real machine operation status from both the work machine 40 (one work machine) which is an existing operation target and the work machine 40 (another operation target). In this case, the remote control device 200 selects the captured image and the real machine operation status of the work machine 40 (another operation target) by reading the real machine identifier of the transmission source attached to the signal representing the received captured image and real machine operation status, and outputs it to the remote output interface 220. In addition, the start time of the communication establishment process and the start time of the communication disconnection process may be the same or different. The end time of the communication establishment process and the end time of the communication disconnection process may be the same or different. The communication establishment process and the communication disconnection process may be serially processed (sequentially executed).

After the operation preparation process starts, a designated item setting process is executed to set the contents of the designated items to assist the operation of the work machine 40, which is the object of operation by the remote control device 20 (FIG. 4/STEP 214). If it is determined that there is no existing operation object (NO in FIG. 4/STEP 208), the designated item setting process is executed in parallel or in parallel with the execution of the communication establishment process (FIG. 4/STEP 210) as the operation preparation process, and is started, for example, at the same time as the execution of the communication establishment process starts. If it is determined that there is an existing operation object (YES in FIG. 4/STEP 208), the designated item setting process is executed in parallel or in parallel with the execution of the communication establishment process and communication disconnection process (FIG. 4/STEP 212) as the operation preparation process, and is started, for example, at the same time as the execution of the communication establishment process and communication disconnection process starts. Next, the remote control device 200 determines whether the operation preparation process has been completed (FIG. 4/STEP 216). If it is determined that the operation preparation process is not complete (NO in FIG. 4/STEP 216), the designated item setting process continues to be executed (FIG. 4/STEP 214).

The designated item setting process will be described in detail below. FIG. 9A is a diagram showing a first lever pattern setting screen in the operation system according to this embodiment. FIG. 9B is a diagram showing a second lever pattern setting screen in the operation system according to this embodiment. In the designated item setting process, for example, a menu screen for selecting multiple designated items is first output to the remote output interface 220 (remote image output device 221). When "lever pattern" is selected as a designated item on the menu screen, the first lever pattern setting screen shown in FIG. 9A is output to the remote output interface 220 (remote image output device 221).

As shown in FIG. 9A, the first lever pattern setting screen includes a first operation guide window Q21, a lever pattern window Q22, an OK button Q23, and a change button Q24. The first operation guide window Q21 displays information guiding the operation method for setting the lever pattern. The lever pattern window Q22 displays the provisionally set lever pattern. When the OK button Q23 is tapped or the like by the operator, the lever pattern displayed in the lever pattern window Q22 is set for real. When the change button Q24 is tapped or the like by the operator, the second lever pattern setting screen shown in FIG. 9B is output to the remote output interface 220 (remote image output device 221).

The actually set lever pattern may be associated with an operator identifier (including image and distance measurement data for face authentication) and/or an actual machine identifier and stored in a storage device and/or database constituting the remote control device 200. The operator identifier used in the authentication process and/or the actual machine identifier used in specifying the operation target may be used to search for or read out the last actually set lever pattern from the storage device, and then displayed as a provisionally set lever pattern by default in the lever pattern window Q22. The actually set lever pattern may simply be stored in a storage device and/or an external database constituting the remote control device 200, and then read out from the storage device, etc., and then displayed as a provisionally set lever pattern by default in the lever pattern window Q22.

As shown in FIG. 9B, the second lever pattern setting screen includes a second operation guide window Q25 and multiple (e.g., four) lever pattern candidate windows Q26, Q27, Q28, and Q29. The second operation guide window Q21 displays information guiding the selection of one of the lever pattern candidate windows Q26, Q27, Q28, and Q29 in order to set a lever pattern. Each of the lever pattern windows Q26, Q27, Q28, and Q29 displays a lever pattern candidate different from the provisionally set lever pattern. In response to a selective tap or the like on one of the lever pattern candidate windows Q26, Q27, Q28, and Q29, the first lever pattern setting screen is output to the remote output interface 220 (remote image output device 221) (see FIG. 9A). At this time, the selected lever pattern candidate is displayed in the lever pattern window Q22 as the provisionally set lever pattern.

The first lever pattern setting screen and the second lever pattern setting screen may be output to the remote output interface 220 (remote image output device 221) as a single screen or as three or more transitional screens.

Figure 10 is a diagram showing a screen for setting the lever sensitivity in the operation system according to this embodiment. When "lever sensitivity" is selected as a designated item on the menu screen, the lever sensitivity setting screen shown in Figure 10 is output to the remote output interface 220 (remote image output device 221). As shown in Figure 10, the lever sensitivity setting screen includes an operation guide window Q31, a lever sensitivity adjustment bar Q32, and an OK button Q33. The operation guide window Q31 displays information that guides the operation method for setting the lever sensitivity. The lever sensitivity adjustment bar Q32 includes a measure that indicates the lever sensitivity (which determines the amount of displacement of the operation target according to the amount of lever operation or the speed of displacement) and, for example, a triangular slider, and the lever sensitivity is provisionally set by adjusting the position of the slider along the measure. When the OK button Q33 is tapped, the lever sensitivity displayed on the lever sensitivity adjustment bar Q32 is set for real.

The finally set lever sensitivity may be associated with the operator identifier and/or the actual machine identifier and stored in a storage device constituting the remote control device 200 and/or an external database. The operator identifier used in the authentication process and/or the actual machine identifier used in the designation of the operation target may be used to search or read out the last actually set lever sensitivity from the storage device as a provisionally set lever sensitivity, and the default position of the slider in the lever sensitivity adjustment bar Q32 may be adjusted. The finally set lever sensitivity may simply be stored in a storage device constituting the remote control device 200 and/or an external database, and the default position of the slider in the lever sensitivity adjustment bar Q32 may be adjusted after being read out from the storage device, etc.

Figure 11 is a diagram showing a screen for setting the seat vibration mode in the operation system according to this embodiment. When "seat vibration" is selected as a designated item on the menu screen, the seat vibration setting screen shown in Figure 11 is output to the remote output interface 220 (remote image output device 221). As shown in Figure 11, the seat vibration setting screen includes an operation guide window Q41, a seat vibration intensity selection window Q42, and an OK button Q43. The operation guide window Q41 displays information that guides the operation method for setting the vibration intensity (including the presence or absence of vibration) of the seat 24 (or the seat device). The seat vibration intensity selection window Q42 includes a plurality of buttons, and the vibration intensity of the seat device is provisionally set by selectively tapping one of the plurality of buttons. When the OK button Q43 is tapped, the vibration intensity of the seat device corresponding to the one button is set.

The actually set vibration intensity may be associated with the operator identifier and/or the actual machine identifier and stored in a storage device constituting the remote control device 200 and/or an external database. The operator identifier used in the authentication process and/or the actual machine identifier used in the designation of the operation target may be used to search or read out the last actually set vibration intensity from the storage device, and then displayed as a provisional vibration intensity by default in the seat vibration intensity selection window Q42. The actually set vibration intensity may simply be stored in a storage device constituting the remote control device 200 and/or an external database, and then read out from the storage device, etc., and then displayed as a provisional vibration intensity by default in the seat vibration intensity selection window Q42.

FIG. 12 is a diagram showing a screen for setting the operation assist in the operation system according to this embodiment. When "operation assist" is selected as a designated item on the menu screen, the operation assist setting screen shown in FIG. 12 is output to the remote output interface 220 (remote image output device 221). As shown in FIG. 12, the operation assist setting screen includes an operation guide window Q51, an operation assist window Q52, and an OK button Q53. The operation guide window Q51 displays information that guides the operation method for setting the presence or absence of operation assist. The operation assist window Q52 includes an ON button and an OFF button for provisionally setting the presence or absence of operation assist or ON/OFF. When the OK button Q53 is tapped, the operation assist is set for real. When the operation assist is set for real to ON, the operation content assisted by the operation assist may be output to the remote output interface 220 (remote image output device 221). The operation content is, for example, a horizontal pulling operation in which the bucket, which is the tip attachment 445, moves linearly along the ground on which the lower running body 41 of the work machine 40 is in contact, and an operation in which the bucket moves from a first position that is away from the upper rotating body 42 in the forward direction to a second position that is closer to the upper rotating body 42 than the first position. There may be multiple types of operation content assisted by the operation assist, and the configuration may be such that the presence or absence of operation assist can be set for multiple types of operation content.

In addition, a simulation image showing an operation status in which the work machine 40 is assisted by the operation assist may be output to the remote output interface 220 (remote image output device 221). The remote control device 200 executes a process of recognizing an operation mode of the remote input interface 210 (remote operation mechanism 211) by the operator, a process of correcting the recognized operation mode so as to approach a target operation mode, a process of recognizing an operation mode of the work machine 40 in the virtual space based on the corrected operation mode, and a process of outputting an image that simulates a state in which the attitude change of the work machine 40 is reflected according to the recognized operation mode to the remote output interface 220 (remote image output device 221) as the simulation image. For example, when the OK button Q53 is tapped, etc., a simulation image showing an operation status in which the work machine 40, which is the operation target in the virtual space, is assisted by the operation assist is output to the remote output interface 220 (remote image output device 221). Then, when the operator operates the work machine 40 in the virtual space through the remote input interface 210 (remote operation mechanism 211), the work machine 40 moves in accordance with the operation mode of the remote input interface 210 (remote operation mechanism 211) in the simulation image based on the operation mode of the remote input interface 210 (remote operation mechanism 211), and the state in which the position and attitude of the tip attachment 445 and the like change is shown. For this reason, the remote control device 200, the remote input interface 210 (remote operation mechanism 211) and the remote output interface 220 (remote image output device 221) constitute a "simulation device." Here, when the operator operates through the remote input interface 210 (remote operation mechanism 211) to move the bucket closer to a second position that is closer to the upper rotating body 42 than the first position in the virtual space, the state in which the tip of the bucket moves linearly from the first position to the second position is shown in the simulation image. Specifically, when the operator rotates the boom 441 upward while rotating the arm 443 downward and further rotates the bucket tip so that it faces downward through the remote input interface 210 (remote operation mechanism 211), the operation is corrected (reducing the deviation of the bucket tip from the target trajectory) so that the bucket tip does not deviate from the target trajectory that linearly connects the first position to the second position by more than a predetermined distance, and is reflected in the simulation image. The first position, the second position, and the target trajectory may be visualized and shown in the simulation image, and the first position and the second position may be set to any position by the operator's operation (for example, by moving the bucket tip to the position to be the first position or the second position and pressing the decision button to set the position). In addition, after communication is established between the remote operation device 20 and the work machine 40 to be operated and the remote operation of the work machine 40 by the remote operation device 20 is possible, a remote operation simulation of the work machine 40 in the virtual space may be allowed in a state in which the remote operation of the work machine 40 is stopped or temporarily stopped.

Figure 13 is a diagram showing a guidance setting screen in the operation system according to this embodiment. When "Guidance (operation support image)" is selected as a designated item on the menu screen, the operation support image setting screen shown in Figure 13 is output to the remote output interface 220 (remote image output device 221). As shown in Figure 13, the operation support image setting screen includes an operation guide window Q61, a depth sense image window Q62, a terrain image window Q63, and an OK button Q64. The operation guide window Q61 displays information that guides the operation method for setting the display mode of the operation support image in the remote output interface 220 (remote image output device 221). The depth sense image window Q62 includes an ON button and an OFF button for provisionally setting the presence or absence or ON/OFF of the display of the depth sense image in the remote output interface 220 (remote image output device 221). The terrain image window Q63 includes an ON button and an OFF button for temporarily setting whether or not to display the terrain image in the remote output interface 220 (remote image output device 221) or ON/OFF. When the OK button Q64 is tapped, etc., the display of the depth sense image and the display of the terrain image are set permanently.

The set display or non-display of the depth sense image and/or terrain image may be associated with the operator identifier and/or the actual machine identifier and stored in a storage device constituting the remote control device 200 and/or an external database. The operator identifier used in the authentication process and/or the actual machine identifier used in the operation target specification may be used to search or read out the last set display or non-display (ON/OFF) from the storage device, and then displayed as a default in the depth sense image window Q62 and/or the terrain image window Q63. The set display or non-display of the depth sense image and/or the terrain image may simply be stored and stored in a storage device constituting the remote control device 200 and/or an external database, and then read out from the storage device, etc., and then displayed as a default in the depth sense image window Q62 and/or the terrain image window Q63.

Also, a simulation image showing a depth sense image and/or a terrain image may be output to the remote output interface 220 (remote image output device 221). For example, when the display of the depth sense image is set to ON, a simulation image showing an operating situation assisted by the operation assist in the work machine 40, which is the operation target in the virtual space, is output to the remote output interface 220 (remote image output device 221). Then, when the operator operates the work machine 40 in the virtual space through the remote input interface 210 (remote operation mechanism 211), the simulation image shows how the work machine 40 operates according to the operation mode, and the position and attitude of the tip attachment 445 and the like change. For this reason, the remote control device 200, the remote input interface 210 (remote operation mechanism 211) and the remote output interface 220 (remote image output device 221) constitute a "simulation device." Here, in the simulation image showing the operating situation of the work machine 40, a depth sense image for showing the position or depth sense of one or more locations defined in the work mechanism 44 is displayed. 14 is a diagram showing a depth sense image in the operation system according to this embodiment. For example, as shown in FIG. 14, a mark M is displayed on the simulation image. A mark M is displayed indicating the position of a second designated point _p2 as a result of vertically projecting a first designated point _p1 , which is the left-right center point of the base end of the tip attachment 445. The first designated point _p1 may be specified as a point other than the left-right center point of the base end of the tip attachment 445 (such as the tip of the tip attachment 445). The second designated point _p2 may be displayed at a predetermined distance from the first designated point _p1 in the virtual space, or may be displayed at any height in the virtual space.

Figure 15 is a diagram showing a terrain image in the operation system according to this embodiment. When the display of the terrain image is set to ON, a terrain image showing the terrain around (forward of) the work machine 40 is superimposed on the simulation image showing the operating status of the work machine 40. This causes, for example, as shown in Figure 15, a mesh-like terrain image showing the terrain or unevenness of the ground surface to be displayed in the simulation image.

Figure 16A is a diagram showing the first button function setting screen in the operation system according to this embodiment. When "button assignment" is selected as a specification item on the menu screen, the first button function setting screen shown in Figure 16A is output to the remote output interface 220 (remote image output device 221). As shown in Figure 16A, the first button function setting screen includes an operation guide window Q71, a button layout window Q72, a function assignment status window Q73, and an OK button Q74. The operation guide window Q71 displays information that guides the user on how to assign functions to the multiple buttons that make up the remote operation mechanism 211. Functions that can be assigned to the buttons include a function for adjusting the angle of the actual machine imaging device 412 in the left-right direction, a function for adjusting the angle of the actual machine imaging device 412 in the up-down direction, a zoom-in function and a zoom-out function for the actual machine imaging device 412, a function for switching ON/OFF an audio output device (for example, a horn that issues an alarm sound to the surroundings) mounted on the work machine 40, a setting function such as calibration of the operation lever that constitutes the remote operation mechanism 211, a switching function for increasing/decreasing the pressure of the hydraulic oil for the work mechanism 44, a switching function for exciting/releasing the lifting magnet as the tip attachment 445, and a function for canceling the temporary stop state of the work machine 40. The button layout window Q72 shows the layout of a plurality of buttons to which functions can be assigned. For example, the button layout window Q72 shows an arrangement of a total of 10 buttons, with buttons L1, L2, and L3 arranged on the left driving lever 2110, buttons L4 and L5 arranged on the left operating lever 2111, buttons R1, R2, and R3 arranged on the right driving lever 2110, and buttons R4 and R5 arranged on the right operating lever 2112. The function assignment status window Q73 shows a chart showing the functions assigned to each of the multiple buttons. When the OK button Q74 is tapped, etc., the functions assigned to each button shown in the function assignment status window Q73 are finally set.

The functions assigned to each button in this setting may be associated with an operator identifier and/or an actual device identifier and stored in a storage device constituting the remote control device 200 and/or an external database. The operator identifier used in the authentication process and/or the actual device identifier used in specifying the operation target may be used to search or read out the last function assigned to each button in this setting from the storage device, and then displayed as a default in the function assignment status window Q73. The functions assigned to each button in this setting may simply be stored in a storage device constituting the remote control device 200 and/or an external database, and then read out from the storage device, etc., and then displayed as a default in the function assignment status window Q73.

Figure 16B is a diagram showing a second button function setting screen in the operation system according to this embodiment. When one of the multiple buttons shown in the button layout window Q72 (or the function assignment status window Q73) is selectively tapped, the second button function setting screen shown in Figure 16B is output to the remote output interface 220 (remote image output device 221). As shown in Figure 16B, the second button function setting screen includes a button window Q75, a button function window Q76, a button function selection window Q77, and an OK button Q78. The button window Q75 displays (highlights as appropriate) the name and/or position of one button selected from the multiple buttons. The button function window Q76 shows the function assigned to the one button. The button function selection window Q77 shows a list of functions that can be assigned to the one button (in the form of a pull-down menu, etc.). When the OK button Q78 is tapped, the first button function setting screen is output to the remote output interface 220 (remote image output device 221) (see FIG. 16A).

When the work machine 40 is equipped with multiple actual machine imaging devices 412 with different imaging ranges (which may be partially overlapping), "Screen Allocation" may be selected as a specification item on the menu screen, and a correspondence may be set between each of the multiple image output devices (displays or monitors) constituting the remote image output device 221 and the captured images acquired through each of the multiple actual machine imaging devices 412 displayed on each image output device.

As described above, various designated item setting processes are executed. If it is determined that the operation preparation process that was being executed has been completed (YES in FIG. 4/STEP 216), the remote control device 200 in the remote operation device 20 executes remote operation processing for remotely operating the work machine 40 (FIG. 4/STEP 218). When the operation preparation process is completed, if the designated item setting process is in progress for a certain designated item, the designated item setting process may be executed continuously (FIG. 4/STEP 214).

Assuming that communication for remotely operating the work machine 40 (operation target) has been established by the remote operation device 20, the actual machine control device 400 in the work machine 40 transmits the captured image acquired through the actual machine imaging device 412 to the remote operation device 20. The transmitted captured image (or captured image data) may be not only the captured image itself, but also image data representing a simulated environmental image generated based on the captured image.

For example, in the remote operation device 20, the remote control device 200 may establish communication for remotely operating the work machine 40 (operation target) in response to transmitting a designation signal to the work machine 40 via the remote wireless communication device 250. When multiple work machines 40 are multiple operation target candidates, a designation signal may be transmitted to each of the multiple work machines 40. The operator may determine whether or not a designation operation has been performed through the remote input interface 210, and when the determination result is positive, a designation signal may be transmitted. A "designation operation" is, for example, an operation such as tapping on the remote input interface 210 for the operator to designate the work machine 40 that he or she intends to remotely operate.

When the remote control device 20 receives an image via the remote wireless communication device 250, the image or the work environment image is output to the remote image output device 221 (display device) (FIG. 6/STEP 280). As a result, an image that includes, for example, the ground spreading out in front of the cab 43, as well as a bucket corresponding to the tip attachment 445 that is part of the work mechanism 44, the ground, and the like, is output to the remote image output device 221.

When the designated item "button assignment" is selected in the designated item setting process and the function assignment of each button is set, the assignment is applied. This allows the operator to adjust the range and/or scale of the captured image by operating the button to which the angle adjustment and/or zoom-in/zoom-out functions of the actual imaging device 412 are assigned (see FIG. 4/STEP 214, FIG. 16A and FIG. 16B).

When the designated item "guidance" is selected in the designated item setting process and the display of the depth image is set to "Yes (ON)", a mark M as a depth image may be superimposed on the captured image (see FIG. 4/STEP 214, FIG. 13 and FIG. 14). For example, the remote control device 200 calculates the position coordinates of the first designated point _p1 , which is the left-right center point of the base end of the tip attachment 445, from the hoisting angle of the boom 441, the rotation angle of the arm 443, and the rotation angle of the tip attachment 445, and displays the mark M superimposed on the position of the second designated point _p2 obtained by projecting the position coordinates vertically downward. When the designated item "guidance" is selected in the designated item setting process and the display of the terrain image is set to "Yes (ON)", a mesh as a terrain image may be superimposed on the captured image (see FIG. 4/STEP 214, FIG. 13 and FIG. 15). The mesh to be superimposed is generated, for example, based on the distance from the work machine 40 to the surrounding ground surface obtained by a sensing device mounted on the work machine 40 that measures the distance to surrounding objects.

Then, the remote control device 200 recognizes the operation mode of the remote operation mechanism 211 (FIG. 6/STEP 281). Specifically, when the designated item "lever pattern" is selected in the designated item setting process and the lever pattern is set, the operation mode of the remote operation mechanism 211 or the operating levers 2110, 2111, and 2112 that constitute it is recognized according to the lever pattern (see FIG. 4/STEP 214, FIG. 9A, and FIG. 9B). When the designated item "lever sensitivity" is selected in the designated item setting process and the lever sensitivity is set, the operation mode of the remote operation mechanism 211 or the operating levers 2110, 2111, and 2112 that constitute it is recognized according to the lever sensitivity (see FIG. 4/STEP 214 and FIG. 10).

Next, the remote control device 200 judges whether the authentication flag f (see FIG. 5/STEP 261 and STEP 262) is 1 or not (FIG. 6/STEP 282). If the authentication flag f is set to "1" in response to the successful authentication process of the operator (YES in FIG. 6/STEP 282 (YES in FIG. 5/STEP 260 → see STEP 261)), a remote operation command corresponding to the operation mode is transmitted to the work machine 40 through the remote wireless communication device 250 (FIG. 6/STEP 284). If the authentication flag f is set to "0" in response to the failure of the authentication process of the operator (NO in FIG. 6/STEP 282 (NO in FIG. 5/STEP 254, NO in STEP 260 → see STEP 262)), the transmission of the remote operation command corresponding to the operation mode to the work machine 40 through the remote wireless communication device 250 is suspended or only a part of the remote operation command is transmitted (FIG. 6/STEP 286).

Then, the time series of the operation modes of the remote operation mechanism 211 by the operator is sequentially acquired as the operation history of the work machine 40 in association with the operator identifier and/or the actual machine identifier, and is stored and held in the storage device constituting the remote control device 200 and/or an external database (FIG. 6/STEP 288). If the execution of the remote operation process for remotely operating the work machine 40 by the remote control device 200 (FIG. 4/STEP 218) is continued, the operation processing function (FIG. 6/STEP 280 → ... → FIG. 6/STEP 288) is repeated.

In the work machine 40, when the actual machine control device 400 receives a remote operation command via the actual machine wireless communication device 450, the operation of the work mechanism 44 and the like is controlled in accordance with the remote operation command. For example, while holding an object such as soil or sand with a bucket as the tip attachment 445, a predetermined task is performed that involves the operation of the work mechanism 44 and the like in order to displace the real space position of the bucket.

When the designated item "operation assist" is selected in the designated item setting process and the operation assist is set to "Yes (ON)", specific operation content is assisted based on the setting. For example, when a horizontal pulling operation is assisted based on the setting, when the operator operates the bucket to approach a second position closer to the upper rotating body 42 of the work machine 40 (operation target) from a first position away from the upper rotating body 42 in the forward direction through the remote input interface 210 (remote operation mechanism 211), the tip of the bucket moves linearly from the first position to the second position in the work machine 40, and an image or work environment image capturing this movement is output to the remote image output device 221 (display device). Specifically, when the operator rotates the boom 441 upward while rotating the arm 443 downward and further rotates the bucket tip so that it faces downward through the remote input interface 210 (remote operation mechanism 211), the operation is corrected (reducing the deviation of the bucket tip from the target trajectory) so that the bucket tip does not deviate more than a predetermined distance from the target trajectory that linearly connects the first position to the second position, and is reflected in the operation of the working mechanism 44. The first position, second position, and target trajectory may be superimposed on the captured image or the work environment image and output to the remote image output device 221 (display device), and the first position and second position may be set to any position by the operator's operation (for example, the position can be set by moving the bucket tip to the desired first position or second position and pressing a decision button).

In the work machine 40, the actual machine control device 400 transmits the actual machine operation status acquired through the actual machine sensor group 416 to the remote operation device 20. The actual machine operation status includes the actual machine attitude (tilt angle around the pitch axis and/or roll axis with respect to the vertical direction of the upper rotating body 42), whether the prime mover is operating, and the attitude of the work mechanism 44. When the remote operation device 20 receives the actual machine operation status through the remote wireless communication device 250, the remote control device 200 controls the operation of the seat drive device 224 to adjust the attitude of the seat 24 so that the seat attitude is aligned or coincident with the actual machine attitude. At this time, among the frequency components representing the time-varying state of the actual machine attitude, high-frequency components exceeding a threshold value may be cut, and the time-varying state of the seat attitude may be adjusted according to only the low-frequency components below the threshold value.

When the remote control device 200 determines that the prime mover of the work machine 40 is operating based on the actual machine operating status, the seat 24 may be vibrated by the seat drive device 224. The seat 24 vibrates around the pitch axis and/or roll axis with a predetermined minute amplitude and a frequency corresponding to the frequency of the prime mover, so that the attitude of the seat 24 changes slightly based on the designated attitude. Alternatively, the seat 24 vibrates in a direction parallel to at least one of the pitch axis, roll axis, and yaw axis with a predetermined minute amplitude and a frequency corresponding to the frequency of the prime mover, so that the attitude of the seat 24 changes slightly in at least one of the forward/backward, left/right, and up/down directions while being maintained in the designated attitude.

When the designated item "Seat vibration" is selected in the designated item setting process and the vibration intensity of the seat 24 (or seat device) is set, the operating speed and amplitude of the seat drive device 224, and therefore the speed and amount of change in the position of the seat 24, are adjusted according to the intensity (see FIG. 4/STEP 214 and FIG. 11).

In the remote operation device 20, the remote control device 200 determines whether or not the operation end condition of the work machine 40 is satisfied (FIG. 4/STEP 220). The operation end condition is a condition for determining that work is not being performed using the work machine 40 that is the object of operation.

The operation end condition includes, for example, a condition that the operation of the prime mover of the work machine 40 has been stopped (or an operation to stop the operation of the prime mover of the work machine 40 has been performed through the remote input interface 210). The operation end condition may also include a condition that the operation lever, travel pedal, and/or option pedal constituting the remote input interface 210 have been returned to the neutral position or initial position (and that state has been maintained for a certain period of time) to cause the work machine 40 to end a predetermined operation of the operating mechanism such as the working mechanism 44. In addition, the operation end condition may include conditions such as whether the cut-off lever has been operated to the lock position, whether an operation to stop the operation of the prime mover of the work machine 40 has been performed in the remote input interface 210, and whether the switch constituting the remote input interface 210 has been switched from the ON state to the OFF state to cause the work machine 40 to end a predetermined operation of the operating mechanism such as the working mechanism 44, and the like, and may be detected depending on whether multiple conditions are satisfied.

If it is determined that the operation termination condition is not satisfied (NO in FIG. 4/STEP 220), remote operation of the work machine 40 by the remote control device 20 continues (FIG. 4/STEP 218).

On the other hand, if it is determined that the operation termination condition is met (FIG. 4/STEP 220: YES), the remote control device 200 determines the operation content by the operator through the remote input interface 210 (FIG. 4/STEP 222).

Figure 17 is a diagram showing an operation selection screen in the operation system according to this embodiment. Specifically, the operation selection screen shown in Figure 17 is output to the remote output interface 220. The operation selection screen shows an operation restart button Q81, an operation switching button Q82, and an operation end button Q83 that constitute the remote input interface 210. The operation restart button Q81 constitutes an input interface for restarting remote operation of the work machine 40, which is an operation target and in a state of communication establishment with the remote operation device 20, through the remote operation device 20. The operation switching button Q82 constitutes an input interface for starting remote operation of a work machine 40 (second work machine), which is an operation target different from the work machine 40 (first work machine), which is an operation target and in a state of communication establishment with the remote operation device 20, through the remote operation device 20. The operation end button Q83 constitutes an input interface for ending remote operation of the work machine 40, which is an operation target and in a state of communication establishment with the remote operation device 20, through the remote operation device 20.

When the operator taps the operation resume button Q81 (when the operation is an operation to resume remote operation of the work machine 40, which is the operation target and in which communication has been established with the remote control device 20, through the remote control device 20) (FIG. 4/STEP 222-1), remote operation of the work machine 40 by the remote control device 20 continues (FIG. 4/STEP 218).

When the operation switching button Q82 is tapped or the like (when the operation content is an operation for starting remote operation of a work machine 40 (second work machine) that is an operation target other than the work machine 40 (first work machine) that is an operation target and has established communication with the remote operation device 20, via the remote operation device 20) (FIG. 4/STEP 222-2), the authentication process (FIG. 4/STEP 200) and subsequent processes are repeated. In this case, since there is a work machine 40 (first work machine) that is an operation target and has established communication with the remote operation device 20, communication disconnection processing with the first work machine and communication establishment processing with the new operation target work machine 40 (second work machine) are executed as "operation preparation processing" (FIG. 4/STEP 208: YES → see STEP 212).

When the authentication flag f is set to "0" in the authentication process (FIG. 5/NO in STEP 254, NO in STEP 260 → see STEP 262), the operation resume button Q81 and/or the operation switch button Q82 may not be displayed, or operations such as tapping may be prohibited.

When the operation end button Q83 is tapped or the like (when the operation content is an operation to end remote operation of the work machine 40, which is the operation target and in which communication has been established with the remote control device 20, via the remote control device 20) (Fig. 4/STEP 222-3), communication disconnection processing between the remote control device 20 and the work machine 40, which is the operation target, is executed (Fig. 4/STEP 224), and remote operation of the work machine 40 by the remote control device 20 ends.

(Action and Effect)
According to the remote control device 20 that exerts the above-mentioned functions and the operation system configured by the remote control device 20, during the execution of the communication establishment process as the operation preparation process, it becomes possible to set the contents of the designated items for supporting the operation of the work machine 40, which is the operation target by the remote control device 20 after the completion of the communication establishment process (see NO in STEP 208 in FIG. 4 -> STEP 210 -> STEP 214, FIG. 9A, FIG. 9B, FIG. 10, FIG. 11, FIG. 12, FIG. 13, FIG. 16A and FIG. 16B). Therefore, the waiting time during the execution of the communication establishment process between the remote control device 20 and the operation target work machine 40 is utilized. As a result, it is also possible to reduce the stress of the operator during the waiting time. Note that the setting of the contents of the designated items (designated item setting process) may be executed at least during the execution of the communication establishment process.

When the operation target is switched from one work machine 40 (first operation target/first work machine) to another work machine 40 (second operation target/second work machine) through the communication device, a communication disconnection process and a communication establishment process are executed as an operation preparation process (see FIG. 4/STEP 208: YES → STEP 212). When the communication disconnection process and the communication establishment process are executed, it becomes possible to set the contents of the specified items through the remote input interface 210 (see FIG. 4/STEP 214). Therefore, the waiting time during the communication disconnection process and the communication establishment process for switching the operation target is utilized.

The assistance and/or operation modes of the work machine 40 to be operated, which are predicted to occur according to the settings of the specified items through the remote input interface 210, can be simulated through the simulation device (see Figures 14 and 15). Based on the experience, it is possible to reset or change the settings of the specified items through the remote input interface 210. As described above, the simulation device is composed of the remote control device 200, the remote input interface 210 (remote operation mechanism 211), and the remote output interface 220 (remote image output device 221).

The settings of the specified items are stored in a storage device or the like constituting the remote control device 200, and the settings of the specified items stored in the storage device or the like are output to the remote output interface 220 (remote image output device 221) (see Fig. 9A/Q22, Fig. 10/Q32, Fig. 11/Q42, Fig. 12/Q52, Fig. 13/Q62, Q63, Fig. 16A/Q73). The current settings of the specified items output to the remote output interface 220 can be confirmed by the operator or the like. Based on the output contents, it becomes possible to reset or change the settings of the specified items through the remote input interface 210 (see Fig. 9B/Q26-Q29, Fig. 10/Q32, Fig. 11/Q42, Fig. 12/Q52, Fig. 13/Q62, Q63, Fig. 16B/Q75-Q78).

According to the remote control device 20 that performs the above functions and the operation system configured thereby, when the operation target is switched from a work machine 40 (first work machine) as one operation target to a work machine 40 (second work machine) as another operation target, a communication disconnection process and a communication establishment process are executed in parallel (see FIG. 4/STEP 208: YES → STEP 210). The "communication disconnection process" is a process for disconnecting the communication established between the remote control device 20 and the work machine 40 as the one operation target. The "communication establishment process" is a process for establishing communication between the remote control device 20 and the work machine 40 as the other operation target. This allows the operation target to be switched smoothly.

The operator can specify a work machine 40 as a new operation target through the remote input interface 210 according to his/her intention. More specifically, the operator can specify a new operation target work machine 40 after canceling the designation of the work machine 40 that was the previous operation target through the remote input interface 210 according to his/her intention (see FIG. 4/STEP 222-2 →STEP 204 and FIG. 17/Q82 → ... →FIG. 8B). This can avoid a situation in which the operation target is erroneously switched when the operator does not intend to switch the operation target.

When a work machine 40, which is an operation target with established communication, is specified as a new operation target through the remote input interface 210, the execution of the communication disconnection process and communication establishment process is put on hold (FIG. 4/STEP 222 1 →STEP 218 and FIG. 17/Q81). When the work machine 40 specified as the new operation target is the same as the work machine 40 previously specified, the communication already established with that work machine 40 continues to be used. This eliminates the need to execute the communication disconnection process and communication establishment process again when specifying (re-specifying) a work machine 40 as a new operation target, and operation of the operation target can be smoothly resumed.

When the operator, according to his/her intention, issues an instruction to end operation of the work machine 40 to be operated through the remote input interface 210, a communication disconnection process is executed (see FIG. 4/STEP 222-3 →STEP 224 and FIG. 17/Q83). This makes it possible to avoid a situation in which the communication established between the remote control device 20 and the work machine 40 to be operated is erroneously disconnected.

According to the remote control device 20 that performs the above functions and the operation system configured therewith, the operation history of the work machine 40 that is the operation target can be acquired separately for each operator identifier and, in turn, each operator identified by the operator identifier, and can be stored and retained in a database or the like (see FIG. 6/STEP 288). When the operation target of the remote control device 20 is switched from one work machine 40 (first work machine) to another work machine 40 (second work machine) (see FIG. 4/STEP 222-2 → ... →STEP 204 and FIG. 8B/Q14, Q15), the operation history of the other work machine 40 is acquired together with the operator identifier (see FIG. 6/STEP 288). This allows the operation history of each different operation target (work machine 40) by the operator to be acquired separately.

Furthermore, the presence or absence of an instruction to switch the operation object is determined on the condition that no work is being performed using the work machine 40 that is the operation object (see FIG. 4/STEP 220: YES -> STEP 222). This makes it possible to avoid a situation in which, while work is being performed using a work machine 40 that is an operation object, the operation object is switched to another operation object, and the work is stopped against the operator's intention.

When an operator is replaced by another operator, the operation history of the work machine 40 to be operated is acquired together with an operator identifier for identifying the new operator (see FIG. 5/STEP 202 → ... → YES in STEP 250, FIG. 6/STEP 288). This allows the operation history of the work machine 40 to be operated by each different operator to be acquired separately.

In addition, when a change notification from one operator to another operator is accepted through the remote input interface 210 (and when the operator is actually changed), the operation history of the operation target is acquired together with the operator identifier of the other operator (see FIG. 5/STEP 202 → ... → YES in STEP 240, FIG. 6/STEP 288). As a result, the operation history of the work machine 40, which is the operation target, can be acquired separately for each different operator. Furthermore, when a switching instruction from one work machine 40 (first work machine) to another work machine 40 (second work machine) is accepted through the remote input interface 210 (and when the operation target is actually switched), the operation history of the other work machine 40 is acquired together with the operator identifier (see FIG. 5/STEP 202 → ... → YES in STEP 240, FIG. 6/STEP 288). As a result, the operation history of each work machine 40, which is the operation target by the operator, can be acquired separately.

When an operator is replaced by another operator, if authentication of the other operator fails (see FIG. 5/STEP 250: YES → STEP 251 → ... → STEP 260 ... NO → STEP 262), the other operator is prohibited or restricted from operating the work machine 40 as the operation target through the remote operation mechanism 211 (see FIG. 6/STEP 282: NO → STEP 286). As a result, the prohibited or restricted operation history of the other operator and the work machine 40 as the operation target are both acquired and stored in a database or the like.

When an operator is replaced by another operator, if authentication of the other operator fails (see FIG. 5/STEP 250: YES → STEP 251 → ... → STEP 260: NO → STEP 262), switching of the work machine 40 as the operation target in response to a switching instruction from the other operator is prohibited. On the other hand, if authentication of the other operator is successful, switching of the work machine 40 as the operation target in response to a switching instruction from the other operator is permitted.

Other Embodiments of the Invention
When the operation target is switched from one operation target work machine 40 to another operation target work machine 40 having specifications different from the one operation target, the setting of the designated items corresponding to the differences in the specifications may be accepted by the remote control device 200 through the remote input interface 210. For example, when the types of the tip attachments 445 are different, it may be possible to set the assignment mode to each button so that a function specific to the operation of the tip attachment 445 (e.g., a lifting magnet) of the work machine 40 after the switch is assigned to the button. Specifically, when a function specific to the operation of the tip attachment 445, which is a difference in the specifications of the work machine 40 after the switch, is not assigned to the button, it is determined that the operation preparation process is not completed (NO in FIG. 4/STEP 216), and the designated item setting process may be continuously executed (FIG. 4/STEP 214). This reliably assigns a function specific to the operation corresponding to the differences in the specifications to the button, enabling smooth operation of the other work machine 40 after the switch of the operation target.

In the above embodiment, the "simulation device" presents a simulation image showing the operating status of the work machine 40 by displaying the operation assist, depth sense image, and terrain image specified on the menu screen, but the "simulation device" may also be made to allow the user to experience other designated functions that can be specified on the menu screen. For example, when the designated item "lever pattern" is selected in the designated item setting process and the lever pattern is set, the operation mode of the remote control mechanism 211 or the operating levers 2110, 2111, and 2112 that constitute it is recognized according to the lever pattern, and the operation of the work mechanism 44, etc. of the work machine 40 in the virtual space is controlled according to the recognized operation mode. When the designated item "lever sensitivity" is selected in the designated item setting process and the lever sensitivity is set, the operation of the work mechanism 44, etc. of the work machine 40 in the virtual space is controlled according to the operation mode recognized according to the lever sensitivity. When the designated item "seat vibration" is selected in the designated item setting process and the strength of the seat vibration is set, the operating speed and amplitude of the seat drive device 224, and therefore the speed and amount of change in the attitude of the seat 24, are adjusted according to the operating status of the work machine 40 in the virtual space in response to the strength. The operating status of the work machine 40 in the virtual space includes the tilt angle around the pitch axis and/or roll axis of the upper rotating body 42 of the work machine 40 relative to the vertical direction in the virtual space, whether the prime mover of the work machine 40 in the virtual space is operating, and the attitude of the work mechanism 44 of the work machine 40 in the virtual space. When the designated item "button allocation" is selected in the designated item setting process and the button allocation level is set, for example, the button to which the angle adjustment and/or zoom-in function/zoom-out function of the actual machine imaging device 412 of the work machine 40 in the virtual space is assigned is operated by the operator, and a virtual captured image or work environment image with the range and/or scale of the captured image adjusted is output to the remote image output device 221 (display device).

Among the functions of the remote control device 20, except for functions unique to the remote control device 20, such as communication establishment processing (see FIG. 4/STEP 210 and STEP 212), communication disconnection processing (see FIG. 4/STEP 212), and remote control processing (see FIG. 4/STEP 218), authentication processing (see FIG. 4/STEP 200), setting processing of the operation target and work content (see FIG. 4/STEP 204), and designated item setting processing (see FIG. 4/STEP 214) may be executed by the information processing terminal 10. The functions of the remote control device 20, such as communication establishment processing (see FIG. 4/STEP 210 and STEP 212), communication disconnection processing (see FIG. 4/STEP 212), and remote control processing (see FIG. 4/STEP 218), may be executed by a server (management device) capable of communicating with the remote control device 20 and the work machine 40 via the second communication network.

Even after a communication disconnection process is executed between the remote control device 20 and a work machine 40 that is an operation target, the operating status (such as a powered state) of the actual control device 400 and the actual wireless communication device 450 of the work machine 40 may be maintained as it was before the communication disconnection process. This makes it unnecessary to perform a process for restoring the operating status of the actual control device 400 and the actual wireless communication device 450 to the status before the communication disconnection process (such as a process for energizing the actual wireless communication device 450 to return it to a state in which the communication establishment process can be executed) when communication is re-established between the remote control device 20 and the work machine 40 that is an operation target, thereby enabling the communication establishment process to be executed smoothly and quickly. For example, when the communication disconnection process is executed, a power supply state to the actual machine control device 400 mounted on the work machine 40 is maintained (a process of cutting off the power supply supplied to the actual machine control device 400 is not performed), and a power supply state to the actual machine imaging device 412 and the actual machine sensor group 416 is maintained (a process of cutting off the power supply supplied to the actual machine imaging device 412 and the actual machine sensor group 416 is not performed).The actual machine control device 400 then transmits signals corresponding to the actual machine operating status relating to the captured images acquired by the actual machine imaging device 412 and the status of the work machine 40 acquired by the actual machine sensor group 416 to a destination identified and indicated by the operation device identifier, and continues the process of receiving the signals corresponding to the operation mode of the remote operation lever for controlling the operation of the operating mechanism through the actual machine wireless communication device 450. In this case, a signal according to the actual machine operating status is transmitted to a destination identified by the operation device identifier, and the remote control device 20, which is the destination, can determine whether the information related to the actual machine identifier contained in the received signal corresponds to that of the work machine 40 currently being operated by the remote control device 20, thereby determining that the received signal is not that of the current operation target of the remote control device 20. Since the received signal is not that of the current operation target of the remote control device 20, the received signal is discarded. Note that, when the control device in the present invention includes the actual machine control device 400 of the work machine 40, the actual machine control device 400 may perform control so that the operating status (such as the power supply state) of the actual machine control device 400 and the actual machine wireless communication device 450, etc., is maintained as it was before the communication disconnection process.

By executing the communication disconnection process, some of the communications established between the remote control device 20 and the work machine 40 that is the target of operation may be disconnected. For example, when the communication disconnection process is executed, a process of turning off the power of the actual machine imaging device 412 and the actual machine sensor group 416 that have been activated is executed, but the power supply state to the actual machine control device 400 mounted on the work machine 40 is maintained (the process of cutting off the power supply supplied to the actual machine control device 400 is not executed). Then, the actual machine control device 400 transmits the captured image acquired by the actual machine imaging device 412 and the signal corresponding to the actual machine operation status regarding the state of the work machine 40 acquired by the actual machine sensor group 416 to the destination identified and indicated by the operation device identifier, and continues the process of receiving the signal corresponding to the operation mode of the remote control lever for controlling the operation of the operating mechanism through the actual machine wireless communication device 450. In this case, the signal corresponding to the actual machine operation status regarding the state of the work machine 40 acquired by the actual machine sensor group 416 becomes an empty signal with no information, and the signal corresponding to the operation mode of the remote control lever is not received until the work machine 40 becomes an operation target again. The empty signal may be replaced with a signal indicating information indicating that the signal is empty. The empty signal may be transmitted at a transmission period lower than the transmission period in which the signal corresponding to the actual machine operation status acquired by the activated actual machine imaging device 412 and the actual machine sensor group 416 is transmitted to the transmission destination. In this case, the signal corresponding to the actual machine operation status is transmitted to the transmission destination identified and indicated by the operation device identifier, and the remote operation device 20, which is the transmission destination, can determine whether or not the information related to the actual machine identifier included in the received signal corresponds to that of the work machine 40 currently operated by the remote operation device 20, thereby determining that the received signal is not the current operation target of the remote operation device 20. Then, since the received signal is not the current operation target of the remote operation device 20, the received signal is discarded. As a result, the communication volume between the remote control device 20 and the work machine 40 is maintained, although it is low compared to the communication established state, so that when communication is established again between the remote control device 20 and the work machine 40 as the one operation target, the process of restoring the operating status of the real machine control device 400 and the real machine wireless communication device 450, etc. to the status before the communication disconnection process is unnecessary or can be simplified. For example, when the real machine control device 400 receives a communication establishment request signal from the remote control device 200, the real machine control device 400 can transmit a signal including an imaged image and the real machine operating status to the remote control device 200 simply by activating the real machine imaging device 412 and the real machine sensor group 416, and can restore the status before the communication disconnection process. Also, for example, the real machine control device 400 may enable the power supply to the real machine imaging device 412 and the real machine sensor group 416 by not performing the process of turning off the power of the activated real machine imaging device 412 and the real machine sensor group 416 when the communication disconnection process is executed. In this case, the remote control device 200 receives a signal including the captured image and the actual machine operation status before transmitting a communication establishment request signal to the actual machine control device 400, and outputs the captured image and the actual machine operation status to the remote output interface 220 when the communication establishment process is executed. As described above, the communication establishment process can be executed smoothly and quickly.

On the condition that there is no change of operator report through the remote input interface 210 (the first flag f1 is set to "0") (see FIG. 5/STEP 240 NO → STEP 242), the presence or absence of an instruction to switch the work machine 40 as an operation target through the remote input interface 210 may be determined while maintaining the communication established with the work machine 40 as one operation target (see FIG. 4/STEP 222). As a result, if the work machine 40 designated as a new operation target is the same as the previous operation target (see FIG. 4/STEP 222 1), the communication already established between the remote operation device 20 and the work machine 40 is continuously used, and there is no need to execute the communication disconnection process and communication establishment process again, and the operation of the work machine 40 can be smoothly resumed.

The presence or absence of an instruction to switch the work machine 40 as an operation target through the remote input interface 210 may be determined based on the requirements that an operator change has been reported through the remote input interface 210 (the first flag f1 is set to "1") and that the acquired operator identifier has changed from one operator identifier to another operator identifier (the second flag f2 is set to "1") (see FIG. 5/STEP 240: YES → STEP 241 and STEP 250: YES → STEP 251). As a result, when an operator is changed from one operator to another operator, the one operator is prevented from designating the work machine 40 as an operation target to be remotely operated by the other operator, and the other operator can designate the work machine 40 as an operation target to be remotely operated by the one operator according to his/her will.

The communication established between the remote control device 20 and the work machine 40 as the operation target may be disconnected by executing the communication disconnection process, provided that the operator change report has been made through the remote input interface 210 (the first flag f1 is set to "1") (see FIG. 5/STEP 240: YES → STEP 241). This prevents the situation in which the communication establishment state between the remote control device 20 and the work machine 40 as the operation target designated by one operator before the change remains, causing the other operator after the change to feel uncomfortable. For example, the captured image or work environment image received by the remote control device 20 from the work machine 40 as the operation target designated by the one operator is output to the remote image output device 221. Since the captured image or the work environment image is unnecessary information that is not related to the remote operation by the other operator himself, if the captured image or the work environment image continues to be output to the remote image output device 221, the other operator will feel uncomfortable. Here, when a changeover request is made, the communication established between the remote control device 20 and the work machine 40 to be operated is cut off, and the captured image or work environment image is no longer output to the remote image output device 221, preventing the other operator from feeling uncomfortable.

In the above embodiment, when it is determined that an existing operation target exists (YES in FIG. 4/STEP 208) on the condition that an instruction to switch the work machine 40 as the operation target has been given through the remote input interface 210 (see FIG. 4/STEP 222-2 and FIG. 17/Q82), the remote control device 200 starts to execute a communication disconnection process as an "operation preparation process" in addition to the communication establishment process (FIG. 4/STEP 212), but this is not limited to this. For example, after communication established with one work machine 40 (first work machine) is disconnected in response to an operation by the operator through the remote input interface 210, on the condition that an instruction to switch the work machine 40 as the operation target has been given through the remote input interface 210 by the operator (see FIG. 4/STEP 222-2 and FIG. 17/Q82), the communication establishment process may be executed to establish communication with another work machine 40 (second work machine). This allows the process of establishing communication with the work machine 40 (first work machine) and the process of disconnecting communication with the work machine 40 (second work machine) to be performed according to the operator's wishes.

When work is being carried out using the work machine 40 as the operation target (FIG. 4/STEP 218), provided that a change of operator is reported from one operator to another operator via the remote input interface 210 (FIG. 5/STEP 240: YES → see STEP 241), the operation history of the work machine 40 as the operation target may be acquired together with an operator identifier that identifies the other operator (see FIG. 6/STEP 288).

A first aspect of the present invention provides an operating device for operating an operating object. The operating device includes a communication device capable of communicating with the operating object, a control device capable of executing a communication establishment process for establishing the communication with the operating object through the communication device, and a designated item setting process for receiving, at least during the execution of the communication establishment process, through an input unit, the setting of the contents of designated items for supporting the operation of the operating object by the operating device after the execution of the communication establishment process.

With an operating device having this configuration, while a communication establishment process is being executed, it is possible to set the contents of the designated items to assist the operating device in operating the operation target after the communication establishment process is completed. This makes it possible to utilize the waiting time during the execution of the communication establishment process.

The second aspect of the present invention is an operating device according to the first aspect, wherein the control device executes a communication disconnection process for disconnecting communication established between the communication device and the one operating target when the operating target is switched from one operating target to another operating target, and a communication establishment process for establishing communication between the operation device and the other operating target, and the control device preferably accepts the setting of the content of the specified item through the input unit when at least one of the communication disconnection process and the communication establishment process is executed in association with the switching of the operating target through the communication device.

According to the operation device having this configuration, when the operation target is switched from one operation target to another operation target, a communication disconnection process and a communication establishment process are executed. The "communication disconnection process" is a process for disconnecting the communication established between the operation device and the one operation target. The "communication establishment process" here is a process for establishing communication between the operation device and the other operation target. When at least one of the "communication disconnection process" and "the communication establishment process" is executed, it becomes possible to set the contents of the specified items via the input unit. Therefore, the waiting time during the execution of the process for switching the operation target is utilized.

In the third aspect of the present invention, in the operating device according to the first or second aspect, it is preferable that when the operation target is switched from the one operation target to the other operation target having specifications different from those of the one operation target, the control device accepts, via the input unit, the setting of the content of the specified item corresponding to the difference in the specifications.

According to the operation device having this configuration, when an operation target is switched from one operation target to another operation target having specifications different from those of the first operation target, it becomes possible to set, via the input unit, the specified items corresponding to the differences in the specifications. This makes it possible to smoothly operate the other operation target after the operation target is switched.

The fourth aspect of the present invention is an operating device according to any one of the first to third aspects, further comprising a simulation device for operating a virtual operating object corresponding to the operating object in a virtual space, and it is preferable that the control device assists in operating the virtual operating object in the virtual space based on the setting contents of the specified items received through the input unit.

With an operation device having this configuration, it is possible to virtually experience, through the simulation device, the support mode and/or operation mode of the operation target that is predicted to occur according to the settings of the items specified through the input unit. Based on the experience, it is possible to reset or change the settings of the items specified through the input unit.

The operating device according to the fifth aspect of the present invention is preferably an operating device according to any one of the first to fourth aspects, further comprising a storage device that stores the setting contents of the specified items, and an output unit that outputs the setting contents of the specified items stored in the storage device.

The operating device having this configuration allows an operator to check the current settings of the specified items output to the output unit. Based on the output contents, it becomes possible to reset or change the settings of the specified items via the input unit.

The sixth aspect of the present invention is an operating device according to the fifth aspect, which has a function of authenticating an operator, and the output unit preferably outputs the setting contents of the specified items stored in the storage device in association with the authenticated operator.

According to the operation device having this configuration, the contents of the specified items last set by each operator (or the contents of the specified items appropriate for each operator) are stored and output to the output unit. The output contents may be defined as the default contents of the specified items.

The seventh aspect of the present invention is an operating device according to the fifth aspect, which has a function of authenticating an operator, and the storage device preferably stores the setting contents of the specified items received through the input unit in association with the authenticated operator.

With an operating device of this configuration, the contents of the specifications last set by each operator can be stored in a storage device.

The eighth aspect of the present invention is an operating device according to any one of the first to seventh aspects, in which the control device preferably accepts settings for the content of a selected one of the multiple designated items via the input unit.

According to the operation device having this configuration, only the content of one or more designated items selected (through the input unit) from among the multiple designated items is set through the input unit.

The present invention provides an operation system, which is configured with an operation device according to any one of the first to eighth aspects and the operation target.

10. Information processing terminal 100. Terminal control device 110. Terminal input interface 120. Terminal output interface 150. Terminal wireless communication device 20. Remote control device (operation device)
24... Seat 200... Remote control device (control device)
210. Remote input interface 211. Remote operation mechanism 220. Remote output interface 221. Remote image output device 222. Remote sound output device 224. Seat drive device 250. Remote wireless communication equipment (communication device)
40... Work machine 44... Work mechanism 400... Actual machine control device 410... Actual machine input interface 420... Actual machine output interface

Claims (9)

An operation device for operating an operation object,
A communication device capable of communicating with the operation target;
a control device capable of executing a communication establishment process for establishing the communication with the operation target through the communication device, and a designated item setting process for receiving, at least during the execution of the communication establishment process, through an input unit, a setting of a designated item for supporting the operation of the operation target by the operation device after the execution of the communication establishment process;
An operating device comprising: 2. The operating device according to claim 1,
The control device includes:
When the operation target is switched from one operation target to another operation target, a communication disconnection process is executed to disconnect a communication established between the communication device and the one operation target, and a communication establishment process is executed to establish a communication between the communication device and the other operation target;
an operation device that accepts a setting of the content of the specified item through the input unit when at least one of the communication disconnection process and the communication establishment process is executed in association with switching of the operation target through the communication device. 3. The operating device according to claim 2,
The control device accepts, when the operation object is switched from one operation object to another operation object having specifications different from those of the one operation object, the setting of the content of the specified item corresponding to the differences in the specifications through the input unit. 2. The operating device according to claim 1,
A simulation device for operating a virtual operation object corresponding to the operation object in a virtual space,
The control device is an operation device that assists in operating the virtual operation target in the virtual space based on the setting contents of the specified items accepted through the input unit. 2. The operating device according to claim 1,
A storage device that stores the setting contents of the specified items;
an output unit that outputs the setting contents of the specified items stored in the storage device;
An operating device comprising: 6. The operating device according to claim 5,
It has a function to authenticate the operator,
The output unit outputs setting contents of the specified items that are associated with the authenticated operator and stored in the storage device. 6. The operating device according to claim 5,
It has a function to authenticate the operator,
The storage device stores the setting contents of the specified items received through the input unit in association with the authenticated operator. 2. The operating device according to claim 1,
The control device is an operation device that accepts settings, via the input unit, for the content of the designated item selected from the plurality of designated items. 1. An operating system, comprising:
An operating device according to any one of claims 1 to 8;
The operation target;
An operating system comprising:

Images