JP2024090078A - Display device, visibility assistance system, industrial vehicle, control method and program - Google Patents

Abstract

To provide a display device which can check the situation in front of a vehicle without further obstructing a visual field of an operator even in such a situation that the visual field in front the vehicle is obstructed by baggage and the like when the operator faces the front of the vehicle.SOLUTION: A display device, which is worn on the head of a person, comprises: a transparent display part that can display image data; an acquisition part that acquires an image of the front of an industrial vehicle photographed by a camera provided on the industrial vehicle; and a control part that displays the image data including a virtual monitor on which the image is displayed so as to be superimposed on an object loaded on the industrial vehicle that obstructs the front visual field.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a display device, a visibility assistance system, an industrial vehicle, a control method, and a program.

A system is available in which a camera is mounted on the side of the forks of a forklift or on the bottom of the mast, and the image is projected onto a monitor attached near the driver's seat. With this system, even in situations where the forward view is blocked by the load on the forks, the driver can check the monitor to see what is happening ahead. In such systems, the monitor size is sometimes designed to be small to ensure a clear forward view, but the driver must stare at the monitor to see the image displayed.

As a related technique, Patent Document 1 discloses an assistance device that is equipped with a camera that captures an image of an area including the tips of the forks, and displays a virtual fork indicating the position of the forks superimposed on the image captured by the camera on a head-mounted display worn by the driver.

Patent No. 6217472

Even in situations where the driver's forward visibility is obstructed when facing the front of the vehicle, there is a demand for technology that allows the driver to check what is ahead without further obstructing their view.

This disclosure provides a display device, a visibility assistance system, an industrial vehicle, a control method, and a program that can solve the above problems.

The display device of the present disclosure is a display device worn on a person's head, and includes a transparent display unit capable of displaying image data, an acquisition unit that acquires an image of the front of the industrial vehicle captured by a camera mounted on the industrial vehicle, and a control unit that displays the image data, including a virtual monitor on which the image is displayed, superimposed on an object loaded on the industrial vehicle that blocks the forward view and is visible through the display unit.

The visibility assistance system of the present disclosure comprises a camera mounted on an industrial vehicle and the above-mentioned display device.

The industrial vehicle disclosed herein is equipped with a camera and the above-mentioned display device.

The control method disclosed herein is a control method for a display device worn on a person's head that has a transparent display unit capable of displaying image data, and acquires an image of the front of the industrial vehicle captured by a camera mounted on the industrial vehicle, and displays the image data, including a virtual monitor on which the image is displayed, superimposed on an object loaded on the industrial vehicle that blocks the forward view and is visible through the display unit.

The program disclosed herein also causes a computer of a display device worn on a person's head, which has a transparent display unit capable of displaying image data, to function as a means for acquiring an image of the front of the industrial vehicle captured by a camera mounted on the industrial vehicle, and a means for displaying the image data, including a virtual monitor on which the image is displayed, superimposed on an object loaded on the industrial vehicle that blocks the forward view and is visible through the display unit.

The above-mentioned display device, visibility assistance system, industrial vehicle, control method, and program allow the driver to check what is ahead without further obstructing his or her view, even in a situation where the driver's view is obstructed when facing the front of the vehicle.

FIG. 1 is a schematic diagram illustrating an example of a forklift according to an embodiment. 1 is a block diagram showing an example of a display device according to an embodiment; 1A and 1B are diagrams illustrating a focal length and a size of a virtual object according to an embodiment. 10A and 10B are diagrams illustrating a relationship between the driver's posture and the position of a virtual object according to the embodiment. FIG. 2 is a diagram showing a display example of an image captured by a camera according to an embodiment. 1A to 1C are first diagrams illustrating a vertical display angle and a height display position of a virtual monitor according to an embodiment. 11 is a second diagram illustrating the vertical display angle and the height display position of the virtual monitor according to the embodiment. FIG. FIG. 11 is a first diagram illustrating a horizontal display angle of the virtual monitor according to the embodiment. FIG. 11 is a second diagram illustrating the horizontal display angle of the virtual monitor according to the embodiment. FIG. 11 is a first diagram illustrating control of switching between display and non-display of a virtual monitor according to an embodiment. FIG. 11 is a second diagram illustrating control of switching between display and non-display of the virtual monitor according to the embodiment. 11 is a flowchart illustrating an example of display control of the virtual monitor according to the embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device according to an embodiment.

<Embodiment>
Hereinafter, the display device of the present disclosure will be described with reference to the drawings.
(composition)
FIG. 1 is a schematic diagram illustrating an example of a forklift according to each embodiment.
The forklift 1 includes a vehicle body 10, a loading device 11, a control device 18, and a display device 20. The vehicle body 10 includes a drive device and a steering device (not shown) and drives the forklift 1. The loading device 11 includes a fork 12, a mast 13 for raising and lowering the fork 12, and a backrest 14 for preventing the load from falling backward (toward the mast 13), and performs a loading operation by raising and lowering the load loaded on the fork 12. The mast 13 includes an outer mast (not shown) and an inner mast (not shown), and the inner mast is configured to rise and fall relative to the stationary outer mast. The mast 13 is provided with a load sensor 15, which detects the weight of the load loaded on the fork 12. The outer mast is also provided with a proximity sensor 16. The proximity sensor 16 detects whether the fork 12 is loaded with a load. The proximity sensor 16 is provided at a height (a relatively high position) that can detect whether or not the load is loaded to an extent that the driver 100's field of vision is obstructed. The loading device 11 is equipped with a camera 17 that captures an image of the front. For example, a camera 17a is mounted on the upper part of the mast 13 facing diagonally downward so as to capture an image of the tip of the fork 12. Also, for example, a camera 17b is provided on the tip of the fork 12. The camera 17b captures an image of the front of the fork 12. The loading device 11 may be equipped with both the cameras 17a and 17b, or only one of them. In the following, as an example, the camera 17a is described as being equipped. The image captured by the camera 17a (which may be a video or a still image) is transmitted to the display device 20. The control device 18 controls the operation of the forklift 1. For example, when the driver 100 operates the handle 19a or the lift lever 19b, the control device 18 detects the operation and controls the traveling direction of the vehicle and raises and lowers the mast 13. The control device 18 is connected to the load sensor 15 and the proximity sensor 16, and acquires information detected by these sensors. The control device 18 transmits information detected by the load sensor 15 and the proximity sensor 16, and information such as a signal indicating that the driver 100 has operated the handle 19a or the lift lever 19b, to the display device 20. The display device 20 is an AR (Augmented Reality) glass or an MR (Mixed Reality) glass. The driver 100 wears the display device 20 and drives the forklift 1. In this specification, the direction in which the loading device 11 of the forklift 1 is provided as shown in FIG. 1 is called the front, and the opposite direction is called the rear, and the left and right directions when facing forward are called the left side and the right side, respectively.

Figure 2 is a block diagram showing an example of a display device according to an embodiment. The display device 20 is AR glasses or MR glasses (hereinafter, both will be referred to as AR glasses) that include a display device and a computer. The display device 20 is communicatively connected to the control device 18 and the camera 17, and acquires information from these devices. The display device 20 displays an image captured by the camera 17a. As shown in the figure, the display device 20 includes a sensor unit 21, a display unit 22, a transmission/reception unit 23, a control unit 24, a display determination unit 25, and a memory unit 26.

The sensor unit 21 includes a sensor for head tracking of the driver 100 (e.g., multiple visible light cameras and an IMU (Inertial Measurement Unit)), a sensor for eye tracking of the driver 100 (e.g., an infrared camera), a depth sensor, a microphone, etc.

The display unit 22 constitutes the lens (screen) of the AR glasses, and is composed of, for example, a transparent holographic display (the outside world can be seen through the lens). When no virtual object is projected onto the lens, the driver 100 can see the real world as it is. A virtual object is image data projected onto the display unit 22. When a virtual object is displayed on the display unit 22, the driver 100 sees the virtual object superimposed on the real world.

The transmission/reception unit 23 transmits and receives data to and from other devices. For example, the transmission/reception unit 23 transmits and receives data to and from other devices via wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). For example, the transmission/reception unit 23 receives images captured by the camera 17. For example, the transmission/reception unit 23 receives from the control device 18 information detected by the load sensor 15 and the proximity sensor 16, a signal indicating that the driver 100 has operated the steering wheel 19a or the lift lever 19b, and the like.

The control unit 24 performs display control of the display unit 22, etc. The control unit 24 recognizes the spatial shape around the driver 100 based on the information detected by the sensor unit 21, creates three-dimensional map information of the surrounding space, and generates a virtual space. The control unit 24 places a virtual object at a desired position in the virtual space based on the created map information. The virtual object is displayed to the driver 100 superimposed on an object existing at that position in the real world. The virtual object can be displayed at (1) a predetermined position in the space (the virtual object is fixedly present at a certain position), (2) a predetermined position based on an object existing in the surrounding space (for example, if the object is the vehicle body 10, the virtual object moves following the running of the vehicle body 10), or (3) a predetermined position based on the driver 100 (for example, the virtual object is present at a position a predetermined distance away in the line of sight of the driver 100). In the case of (1) or (2), the virtual object is recognized by the driver 100 as if it actually exists at that position. The virtual object can be placed at any position in the virtual space, and the placement position can be freely changed by moving it closer to or farther away from the driver 100. For example, if the virtual object is placed far away, the driver 100 sees the virtual object as being far away, and if it is placed close to the driver 100, the driver 100 sees the virtual object as being close to the driver. Furthermore, the control unit 24 calculates the position of the driver's 100's head when the driver 100 assumes a posture, such as bending over or leaning to the right or left from the driver's seat, based on the information detected by the sensor unit 21. Then, the control unit 24 generates and displays a virtual object based on the positional relationship between the driver's 100's head and the virtual object so that the virtual object appears to be in the same way as if it actually exists in the real world.

The AR glasses have the functions of freely arranging virtual objects, arranging a virtual object at a certain position in the virtual space so that the virtual object appears to exist at that position, and changing the appearance of the virtual object according to the driver's 100 posture. The user sets the desired virtual object for the display device 20 (the developer develops the virtual object using an AR development tool and implements the developed program in the display device 20). The control unit 24 then places the virtual object at the desired position according to the setting. In this embodiment, the control unit 24 displays the virtual object at any position within the range occupied by the luggage loaded on the fork 12 (corresponding to (2) above). For example, if a virtual object is placed in the range where the luggage is loaded on the fork 12, the virtual object can be displayed superimposed on the luggage. The range where the luggage is loaded is originally blocked by the luggage, so even if the virtual object is displayed superimposed on the luggage, the driver's 100's visibility is not further impaired. In this embodiment, a virtual monitor VM that displays an image captured by camera 17a as a virtual object is placed somewhere within the area occupied by the luggage. This allows the image of the front captured by camera 17a to be displayed within the field of view of driver 100 without blocking the view, even in a situation where the view ahead is blocked by luggage.

3A shows a virtual object 31a placed a distance L1 ahead of the driver 100, a display range (range of the field of view that fits in the display unit 22) 31b placed a distance L1 ahead of the driver 100, a virtual object 32a placed a distance L2 ahead of the driver 100, and a display range 32b placed a distance L2 ahead. If a real monitor that is not a virtual object is placed on the back side of the mast 13 (for example, a distance L2 ahead of the driver 100), the monitor will be hidden by the mast 13 and will not be visible, but in the case of an AR virtual monitor VM, even if it is placed on the back side of the mast 13, the virtual monitor VM will not be hidden by the mast because it is displayed on the AR glasses. AR technology also has an occlusion function that displays the virtual monitor VM as if it is hidden by the mast 13, but in this embodiment, the occlusion function is not used to prioritize the information displayed on the virtual monitor VM. As shown in FIG. 3, when two virtual objects 31a and 32a of different sizes (virtual object 32a is larger in the virtual space) are placed at positions of distances L1 and L2, respectively, the driver 100 can see both of them as the same size due to the nature that the farther an object is, the smaller it appears. In other words, when a virtual object is placed farther away, the size of the virtual object is made larger, and when it is placed closer, the size of the virtual object is made smaller. The specific size of the virtual object can be calculated based on the placement position (distance from the driver 100) and the size at which it is displayed on the display unit 22, and by changing the size of the virtual object according to the distance from the driver 100, it is possible to make the virtual object always appear as a constant size to the driver 100. Assuming that the virtual object is not hidden by the mast 13 and is not visible, the difference between the two virtual objects 31a and 32a displayed as the same size to the driver 100 is only the placement position, in other words, the focal length. By controlling the placement position and size of the virtual object placed in the virtual space in this way, it is possible to freely change only the focal length without changing the display size of the virtual object as seen by the driver 100.

By utilizing this property, when the virtual monitor VM is placed near the tip of the fork 12, the focus when looking at the tip of the real fork 12 and when looking at the virtual monitor VM are almost the same, and the driver 100 can see both at the same time. In other words, in order to show the image captured by the camera 17a without forcing the driver 100 to focus, it is preferable to place the virtual object as close as possible to the focal position of the driver 100. When the forklift 1 is moved forward with a load loaded, the driver 100 may want to check a shelf or the like further forward of the tip of the fork 12 from the driver's seat. In this case, if the virtual monitor VM is also placed slightly forward of the tip of the fork 12, the virtual monitor VM appears to be present at that position. For the driver 100 who is visually checking the area further forward of the tip of the fork 12, the virtual object appears to be present at the exact position where the driver's line of sight is aligned, so there is no need to refocus to check the virtual monitor VM. However, in a similar situation where the forklift 1 is moving forward with a load, as shown in FIG. 3B, the driver 100 may want to lean forward to the right and check the positional relationship between the load and the shelf or the like located further forward from the side. In this case, if the virtual monitor VM is placed in front of the tip of the fork 12, the shelf or the like to be checked and the virtual monitor VM will appear to overlap, making it difficult to check the positional relationship between the load and the shelf. Therefore, in this embodiment, the virtual monitor VM is placed in front of the tip of the fork 12 so as not to block the driver 100's field of vision even when the driver 100 takes a posture as shown in FIG. 3B. Also, if the virtual monitor VM is placed too far forward (for example, VM1 in FIG. 3B), the virtual monitor VM will fall out of the field of vision H1 when the driver 100 takes a posture as shown in FIG. 3B, and the driver 100 will have to move his or her line of sight to check the virtual monitor VM1. In order to prevent such a situation from occurring, in this embodiment, the virtual monitor VM is placed in front of the backrest 14. In other words, the virtual monitor VM is placed somewhere in the range H2. This allows the image captured by the camera 17a to be displayed within the field of view of the driver 100 without blocking the driver's 100 field of view, even if some focusing burden is imposed. Figure 4 shows an example of the virtual monitor VM displayed on the display unit 22.

Next, control of the vertical display angle and the height direction display position of the virtual monitor VM will be described with reference to FIGS. 5A and 5B.
The display surface P in FIG. 5A shows the display position in the front-rear direction and the vertical angle of the virtual monitor VM. The control unit 24 places the virtual monitor VM at a predetermined position in the range H2, and tilts the display surface of the virtual monitor VM upward by an angle θ so as to be perpendicular to the line of sight of the driver 100. The line of sight of the driver 100 may be calculated using the eye tracking function of the sensor unit 21, or a predetermined angle may be set according to the head position of the driver 100. In addition, the size (vertical size) of the virtual monitor VM may be set as large as possible so that the driver 100 can better recognize the state behind the luggage 50 as seen from the driver 100 (for example, size 51 in FIG. 5A), or the size may be set to match the size of the luggage so that the driver 100's field of vision is not obstructed even a little by the virtual monitor VM (for example, size 52 in FIG. 5A). The vertical size of the virtual monitor VM may be determined in advance based on the size of an average load of luggage in a warehouse or the like, or, for example, the control unit 24 may recognize the boundary between the luggage 50 and other areas (e.g., the ground) from an image recognized by a visible light camera provided in the sensor unit 21, calculate the size of the virtual monitor VM such that the upper end of the virtual monitor VM as seen by the visible light camera does not exceed the corner 53 of the luggage 50, and display the virtual monitor VM at the calculated size. Also, even if the virtual monitor VM exceeds the range of the luggage, the image captured by the camera 17a may be displayed on the virtual monitor VM as a see-through image (an image displayed in such a way that the real world on the other side of the image can be seen through when viewed from the driver 100) so that the real world on the other side of the virtual monitor VM can be recognized.

Figure 5B shows an example of the display of the virtual monitor VM when the driver 100 is bent over. When the driver 100 is bent over, the control unit 24 slides the virtual monitor VM downward to reduce the size in the vertical direction or reduce the overall size. In this case, the control unit 24 also adjusts the angle of the display surface of the virtual monitor VM so that it is perpendicular to the line of sight of the driver 100. The control unit 24 may set the size of the virtual monitor VM as large as possible (for example, size 51' in Figure 5B) or may set it to a size that matches the size of the luggage (for example, size 52' in Figure 5A).

Next, control of the horizontal display angle of the virtual monitor VM will be described with reference to FIGS. 6A and 6B.
The display surface P in FIG. 6A indicates the display position in the front-rear direction and the horizontal angle of the virtual monitor VM. The control unit 24 places the virtual monitor VM at a predetermined position in the range H2 and directs the display surface of the virtual monitor VM so as to be perpendicular to the line of sight of the driver 100. The control unit 24 may set the size (horizontal size) of the virtual monitor VM as large as possible (for example, size 61 in FIG. 6A), or may set it to a size that fits within the luggage 60 (for example, size 62 in FIG. 5A). The horizontal size of the virtual monitor VM may be determined in advance based on the average size of luggage loaded in a warehouse or the like, or, for example, the control unit 24 may recognize the boundary between the luggage 60 and other objects from an image recognized by a visible light camera provided in the sensor unit 21, calculate a size that fits within the luggage 60, and display the virtual monitor VM at the calculated size. The control unit 24 may also display the image captured by the camera 17a as a transparent image.

6B shows an example of the display of the virtual monitor VM when the driver 100 leans out to the right of the driver's seat. When the driver 100 moves to the right, the control unit 24 adjusts the horizontal angle of the virtual monitor VM around point O according to the driver's 100 posture. For example, the control unit 24 may rotate the virtual monitor VM horizontally so that the line of sight of the driver 100 and the display surface of the virtual monitor VM are perpendicular to each other, or may rotate the virtual monitor VM counterclockwise around point O by a predetermined angle according to the amount of movement of the driver 100 to the right. The control unit 24 may set the size of the virtual monitor VM as large as possible (for example, size 61' in FIG. 5B) or may set the size according to the size of the luggage (for example, size 62' in FIG. 5A). However, when the size of the virtual monitor VM is displayed large, the control unit 24 does not display the virtual monitor VM in an area overlapping the illustrated range 63' so as not to obstruct the view toward the tip of the fork 12. For example, the control unit 24 may hide a predetermined range on the right side of the virtual monitor VM, shrink the horizontal size, or slide the display position in the direction of the arrow 64' in accordance with the amount of rightward movement of the driver 100 so as not to block the field of view of the range 63'. The control unit 24 may also hide the range on the opposite side (left side) of the virtual monitor VM that is outside the angle of view of the camera 17a.

In this way, the position, angle, and range of the virtual monitor VM displayed on the display unit 22 are controlled according to the posture of the driver 100. In addition, the display and non-display of the virtual monitor VM can be switched automatically or by the movement or operation of the driver 100. For example, in a scene where the virtual monitor VM is in the way, the virtual monitor VM can be hidden.

The display determination unit 25 determines whether to display or hide the virtual monitor VM. When the display determination unit 25 determines that the virtual monitor VM should be displayed, the control unit 24 displays (places) the virtual monitor VM, and when the display determination unit 25 determines that the virtual monitor VM should be hidden, the control unit 24 hides the virtual monitor VM (does not place the virtual monitor VM in the virtual space). For example, the display determination unit 25 determines that the virtual monitor VM should be displayed when the front of the vehicle is obscured by luggage, and otherwise determines that the virtual monitor VM should be hidden. An example of the determination process of the display determination unit 25 is described below.

<When automatically detecting a state in which the vehicle's front is not visible>
(A) The display determination unit 25 acquires information detected by the proximity sensor 16 (detects whether the outer mast on which the proximity sensor 16 is provided is in close proximity to the luggage loaded on the forks 12. If no luggage is loaded, it is considered that no luggage is loaded until it reaches a height that can be detected by the proximity sensor 16) from the control device 18 via the transmission/reception unit 23. If the proximity sensor 16 detects that the outer mast is in close proximity to the luggage, the display determination unit 25 determines to display the virtual monitor VM, and if not, determines to hide the virtual monitor VM.

(B) The display determination unit 25 acquires information detected by the load sensor 15 (weight of the luggage loaded on the forks 12) from the control device 18 via the transmission/reception unit 23. When the weight detected by the load sensor 15 is equal to or greater than a predetermined threshold, the display determination unit 25 determines to display the virtual monitor VM, and when the height is less than the threshold, determines to hide the virtual monitor VM.

(C) The sensor unit 21 uses a visible light camera or the like to capture an image of the area around the mast 13 when no luggage is loaded, and records the image or the shape of the area around the mast 13 when no luggage is loaded in the memory unit 26. The display determination unit 25 then compares the image of the area around the mast 13 captured by the sensor unit 21 with the image recorded in the memory unit 26 each time, and determines whether luggage has been loaded based on, for example, the difference between the images. For example, if the average difference between each pixel in the height range that blocks the view of the captured image is equal to or greater than a threshold value, the display determination unit 25 determines that luggage has been loaded up to that height. If it is determined that luggage has been loaded, the display determination unit 25 determines that the virtual monitor VM should be displayed, and if not, determines that the virtual monitor VM should not be displayed.

<When display/non-display is determined by driver operation>
(D) The control unit 24 displays an operation panel (virtual object) in a virtual space based on, for example, the gestures or voice of the driver 100. This operation panel is provided with a virtual switch that switches between displaying and hiding the virtual monitor VM. When the driver 100 instructs display by operating the virtual switch, the display determination unit 25 determines that the virtual monitor VM is to be displayed. When the driver 100 instructs not to display by operating the virtual switch, the display determination unit 25 determines that the virtual monitor VM is to be hidden. A technology that enables operations to be performed on the operation panel displayed in the virtual space in this way can be realized by MR.

(E) A button for switching the display or non-display of the virtual monitor VM is provided on the handle 19a or the lift lever 19b, and when this button is operated, the display determination unit 25 determines that the virtual monitor VM is to be displayed or hidden.

(F) A two-dimensional marker (e.g., a QR code (registered trademark)) is attached to the front pipe of the head guard of the mast 13 (driver's side of the outer mast). For example, as illustrated in FIG. 7A, a two-dimensional marker 71 containing information corresponding to display (ON) is attached to the mast 13 on the right side as viewed from the driver's side, and a two-dimensional marker 72 containing information corresponding to non-display (OFF) is attached to the mast 13 on the left side as viewed from the driver's side. When the driver 100 stares at either of the two-dimensional markers 71, 72 (for example, for a predetermined period of time), the display determination unit 25 determines that the virtual monitor VM is to be displayed or hidden. Whether the driver 100 is staring at either of the two-dimensional markers 71, 72 can be detected, for example, by an eye tracking sensor of the sensor unit 21. It is possible to recognize whether display or non-display has been instructed based on whether the user has simply gazed at the two-dimensional marker 71 on the right side or the two-dimensional marker 72 on the left side. However, in order to prevent erroneous operation, it is preferable to use different information contained in the left and right two-dimensional markers, and the display determination unit 25 determines whether display or non-display is required based on the content read from the two-dimensional marker. Also, the switching between display and non-display using the two-dimensional marker may be enabled only when it is determined by the detection methods (A) to (C) described above that luggage is loaded to a certain height. This makes it possible to prevent the virtual monitor VM from being displayed when the user happens to gaze at the two-dimensional marker 71 that corresponds to the display even when no luggage is loaded, or when the virtual monitor VM is not needed even when luggage is loaded.

(G) Also, instead of attaching a QR code (registered trademark) to the mast 13, etc., a virtual switch for display and non-display may be placed near a position in the virtual space corresponding to the mast 13. For example, as illustrated in FIG. 7B, a virtual switch 71' for display (ON) is placed on the right side, and a virtual switch 72' for non-display (OFF) is placed on the left side. When the driver 100 gazes at any of the virtual switches, the display determination unit 25 determines that the virtual monitor VM should be displayed or hidden. In this case, too, the switching between display and non-display using the virtual switch may be enabled only when it is determined that luggage is loaded to a certain height using the detection methods (A) to (C) described above.

(H) The display determination unit 25 may also use voice recognition to determine whether to display or hide the virtual monitor VM. For example, when the driver 100 utters "display," the microphone of the sensor unit 21 detects the utterance, and the display determination unit 25 recognizes through voice recognition that an instruction to display the virtual monitor VM has been given. The display determination unit 25 then determines that the virtual monitor VM should be displayed. The same applies to the case of not displaying the virtual monitor VM. In this case, too, the virtual switch may be set to enable switching between display and non-display only when it is determined that luggage is loaded to a certain height using the detection methods (A) to (C) described above.

The memory unit 26 stores information detected by the sensor unit 21, information received by the transceiver unit 23, virtual objects, CAD information of the forklift 1 or various types of forklifts, etc.

(motion)
Next, control of the display device 20 will be described with reference to FIG.
FIG. 8 is a flowchart illustrating an example of display control of the virtual monitor according to the embodiment.
When the driver 100 wearing the display device 20 sits in the driver's seat of the forklift 1, the driver starts up the display device 20 and performs calibration (step S1). In the calibration, the positional relationship between the driver 100 and the forklift 1 is calculated. For example, the control unit 24 recognizes the handle and other structures of the forklift 1 using the visible light camera of the sensor unit 21, and estimates the type of vehicle of the forklift 1 from the shape and appearance of the photographed handle, etc. The control unit 24 reads out CAD information corresponding to the estimated type of vehicle from the CAD information recorded in the storage unit 26, and grasps the overall shape of the forklift 1. In addition, for example, the control unit 24 calculates the positional relationship between the driver 100 and the steering wheel, that is, where the driver 100 sits in the driver's seat, how high the driver's 100's head is, or the positional relationship between the driver 100 and each part of the forklift 1 (horizontal distance, vertical distance, etc.) from the read CAD information of the forklift 1 by analyzing an image of the steering wheel captured by the visible light camera of the sensor unit 21 or measuring the distance to the steering wheel by the depth sensor of the sensor unit 21. This makes it possible to grasp the exact distance between the driver 100 and the tip of the fork 12 or the backrest 14, and to calculate, for example, the position and angle of the virtual monitor VM, and the position and angle of the virtual monitor VM when the driver 100 leans out to the right. In addition, for example, a two-dimensional marker may be provided at a predetermined position on the forklift 1 (e.g., the outer mast), the two-dimensional marker may be recognized by the sensor unit 21, and the positional relationship between the driver 100 and each part of the forklift 1 may be calculated in a similar manner to that described above based on the positional relationship with the two-dimensional marker, the type of forklift 1 indicated by the two-dimensional marker, and CAD information for that type of vehicle.

When the calibration is completed, the driver 100 performs a predetermined operation on the display device 20 to activate a function for displaying the virtual monitor VM. When the function is activated, the control unit 24 acquires an image captured by the camera 17 from the camera 17 through the transmission/reception unit 23 (step S2). Next, the display determination unit 25 determines whether to display or not display the virtual monitor VM by the determination method described in (A) to (G) above (step S3). If the display determination unit 25 determines to display the virtual monitor VM (step S4; Yes), the control unit 24 places the virtual monitor VM at a predetermined position in the virtual space (step S5). The predetermined position is, for example, between the tip of the fork 12 and the backrest 14 in the front-rear direction, and is a range that overlaps with the height of the luggage in the vertical direction. The height of the luggage may be grasped by a predetermined image recognition from an image captured by the camera 17a or the visible light camera of the sensor unit 21, or a certain height may be set in advance. The control unit 24 displays the virtual monitor VM, for example, superimposed on the luggage loaded on the fork 12, in as large a size as possible. Also, as described with reference to Figures 5A to 6B, the control unit 24 changes the direction of the display surface of the virtual monitor VM according to the line of sight of the driver 100, and changes the vertical or horizontal position and display size of the virtual monitor VM according to the posture of the driver 100.
In step S3, when the display determination unit 25 determines that the display should be made non-display (step S4; No), the process proceeds to step S6.

Next, the control unit 24 determines whether or not to terminate the function of displaying the virtual monitor VM (step S6). For example, if the driver 100 performs an operation to terminate the function of displaying the virtual monitor VM, the control unit 24 determines that the function of displaying the virtual monitor VM is to be terminated, and otherwise determines that the function is not to be terminated. If it is determined that the function is not to be terminated (step S6; No), the processing from step S2 onwards is repeated. If it is determined that the function is to be terminated (step S6; Yes), the processing flow of FIG. 8 is terminated.

In the above embodiment, the visibility assistance system including the camera 17 and the display device 20 is applied to a forklift 1, but the visibility assistance system can also be applied to industrial vehicles other than forklifts, such as excavators, bulldozers, cranes, and reach stackers.

(effect)
Conventionally, a system has been provided in which a real monitor is provided in front of the driver's seat and an image of the front of the fork 12 is displayed on the monitor. In the case of this system, the monitor obstructs the view during driving, and it is necessary to stare at the monitor in order to check the image displayed on the real monitor, which is designed to be small to ensure a forward view. In contrast, in this embodiment, the driver is made to wear AR glasses (display device 20), a virtual monitor VM is placed in a virtual space, and an image of the front of the fork 12, etc. is displayed on the virtual monitor VM. In addition, since the virtual monitor VM can be placed at a free position, it is superimposed on the luggage blocking the view in front of the vehicle, so that the driver 100 can check the front without further blocking the view. In addition, since the virtual monitor VM can be displayed larger than the real monitor, the driver 100 does not need to stare at it while facing the front of the vehicle, and can check the image of the front projected on the luggage. In addition, by placing the virtual monitor VM somewhat forward (between the backrest 14 and the tip of the fork 12), the focal length can be set farther than when a real monitor is provided in the driver's seat. This makes it easier for the driver 100 who drives the forklift 1 while visually checking the surroundings of the vehicle to focus, and makes it possible to prevent the driver 100 from being burdened by checking the image displayed on the virtual monitor VM. In addition, by turning off the virtual monitor VM when it is not necessary, it is possible to prevent the driver 100 from being obstructed by the view.

FIG. 9 is a diagram illustrating an example of a hardware configuration of the display device according to the embodiment.
The computer 900 includes a CPU 901 , a main memory device 902 , an auxiliary memory device 903 , an input/output interface 904 , and a communication interface 905 .
The above-mentioned display device 20 is implemented in a computer 900. The above-mentioned functions are stored in the auxiliary storage device 903 in the form of a program. The CPU 901 reads the program from the auxiliary storage device 903, loads it in the main storage device 902, and executes the above-mentioned processing in accordance with the program. The CPU 901 also reserves a storage area in the main storage device 902 in accordance with the program. The CPU 901 also reserves a storage area in the auxiliary storage device 903 for storing data being processed in accordance with the program.

A program for implementing all or part of the functions of the display device 20 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform processing by each functional unit. The term "computer system" here includes hardware such as an OS and peripheral devices. In addition, if a WWW system is used, the term "computer system" also includes a homepage providing environment (or display environment). In addition, the term "computer-readable recording medium" refers to portable media such as CDs, DVDs, and USBs, and storage devices such as hard disks built into a computer system. In addition, if the program is distributed to the computer 900 via a communication line, the computer 900 that receives the program may expand the program into the main storage device 902 and execute the above processing. In addition, the program may be for implementing part of the functions described above, and may further be capable of implementing the functions described above in combination with a program already recorded in the computer system.

As described above, several embodiments of the present disclosure have been described, but all of these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and gist of the invention.

<Additional Notes>
The display device, the visibility assistance system, the industrial vehicle, the control method, and the program described in each embodiment can be understood, for example, as follows.

(1) The display device of the first aspect is a display device 20 worn on the head of a person (driver 100 or passenger) and includes a transparent display unit 22 capable of displaying image data, an acquisition unit (transmitter/receiver unit 23) that acquires an image of the front of the industrial vehicle captured by a camera 17 installed on the industrial vehicle, and a control unit 24 that displays the image data, including a virtual monitor on which the image is displayed, by superimposing it on an object (luggage) carried by the industrial vehicle that blocks the forward view and is visible through the display unit.
Using AR technology, an object blocking the driver's 100 forward view is utilized and an image of the front of the industrial vehicle captured by a camera is displayed thereon, allowing the driver to check what is happening ahead even when the driver's forward view is blocked.

(2) A second aspect of the present invention is a display device according to (1), in which the control unit changes the position and/or angle at which the image data is displayed depending on the posture of the person.
This makes it easier for the driver 100 to check the virtual monitor.

(3) A display device according to a third aspect is the display device of (1) to (2), in which the control unit controls the area in which the image data is displayed so that it does not extend beyond the object.
This can prevent the virtual monitor VM from making it harder to see ahead.

(4) A display device according to a fourth aspect is a display device according to any one of (1) to (3), further comprising a display determination unit that determines whether to display or hide the image data, and when the display determination unit determines that the image data should be displayed, the control unit displays the image data, and when the display determination unit determines that the image data should be hidden, the control unit does not display the image data.
This allows the virtual monitor VM to be displayed only when necessary. For example, in a situation where the virtual monitor VM blocks the view and becomes an obstacle, the virtual monitor VM can be hidden.

(5) A display device according to a fifth aspect is the display device of (4), in which the display determination unit determines to display the image data when it detects that the industrial vehicle is holding the object.
This makes it possible to display the virtual monitor VM only when the maintenance device is in operation.
When an industrial vehicle has an object blocking the view, it becomes difficult to see ahead, so the virtual monitor is displayed when it detects that an object blocking the view is detected. As a result, when the view ahead is blocked, the virtual monitor VM is automatically displayed, and the driver can check the view ahead on the virtual monitor VM.

(6) A sixth aspect of the display device is a display device according to any one of (4) to (5), wherein the display determination unit detects that the industrial vehicle is carrying the object based on any one of a change in weight detected by a mechanism for the industrial vehicle to carry the object, a change in an image captured around the mechanism, and detection information from a proximity sensor provided around the mechanism for detecting the approach of the object.
This makes it possible to detect whether the industrial vehicle has an object blocking the view.

(7) A display device according to a seventh aspect is a display device according to any one of (4) to (6), wherein the control unit displays a virtual switch that accepts instructions to switch between displaying and hiding the image data, and the display determination unit determines whether to display or hide the image data based on the person's operation of the switch.
This makes it possible to toggle the visibility of virtual monitors on and off without having to set up a hardware switch.

(8) A display device according to an eighth aspect is a display device according to any one of (4) to (7), wherein the display determination unit determines whether to display or not display the image data based on whether the person is staring at a first marker associated with the display of the image data or a second marker associated with the non-display of the image data.
This allows the driver to switch the virtual monitor on and off by simply looking at the screen, even if their hands are occupied.

(9) The visibility assistance system according to the ninth aspect includes a camera mounted on an industrial vehicle and a display device as set forth in (1) to (8).

(10) The industrial vehicle according to the tenth aspect is equipped with a camera and a display device according to (1) to (8).

(11) The control method according to the eleventh aspect is a control method for a display device worn on a person's head, which has a transparent display unit capable of displaying image data, and which acquires an image of the front of the industrial vehicle captured by a camera mounted on the industrial vehicle, and displays the image data, including a virtual monitor on which the image is displayed, superimposed on an object of the industrial vehicle that is visible through the display unit and blocks the forward view.

(12) The program according to the twelfth aspect causes a computer of a display device worn on a person's head, having a transparent display unit capable of displaying image data, to function as a means for acquiring an image of the front of the industrial vehicle captured by a camera mounted on the industrial vehicle, and a means for displaying the image data, including a virtual monitor on which the image is displayed, superimposed on an object of the industrial vehicle that is visible through the display unit and blocks the forward view.

Reference Signs List 1: Forklift 10: Body 11: Cargo handling device 12: Fork 13: Mast 14: Backrest 15: Load sensor 16: Proximity sensor 17: Camera 18: Control device 19a: Handle 19b: Lift lever 20: Display device 21: Sensor section 22: Display section 23: Transmitter/receiver 24: Control section 25: Display determination section 26: Memory section 100: Driver VM: Virtual monitor 900: Computer 901: CPU
902: Main memory device 903: Auxiliary memory device 904: Input/output interface 905: Communication interface

Claims (12)

A display device that is worn on a person's head,
a transparent display capable of displaying image data;
an acquisition unit that acquires an image of a front portion of the industrial vehicle captured by a camera provided on the industrial vehicle;
a control unit that displays the image data including a virtual monitor on which the image is displayed, superimposed on an object that is loaded on the industrial vehicle and blocks a forward view and is visible through the display unit;
A display device comprising: The control unit changes a position and/or an angle at which the image data is displayed in accordance with the posture of the person.
The display device according to claim 1 . The control unit controls a range in which the image data is displayed so as not to extend beyond the object.
The display device according to claim 1 or 2. a display determination unit that determines whether the image data is to be displayed or not;
Further equipped with
The control unit displays the image data when the display determination unit determines that the image data is to be displayed, and does not display the image data when the display determination unit determines that the image data is to be hidden.
The display device according to claim 1 or 2. the display determination unit determines to display the image data when detecting that the industrial vehicle has the object.
The display device according to claim 4. the display determination unit detects that the industrial vehicle has the object based on any one of a change in weight detected by a mechanism for the industrial vehicle to have the object, a change in an image captured around the mechanism, and detection information by a proximity sensor provided around the mechanism to detect the approach of the object.
The display device according to claim 5 . the control unit displays a virtual switch that receives an instruction to switch between displaying and not displaying the image data;
the display determination unit determines whether the image data is to be displayed or not based on the person's operation of the switch;
The display device according to claim 4. The display determination unit determines whether the image data is displayed or not based on whether the person gazes at a first marker associated with display of the image data or a second marker associated with non-display of the image data.
The display device according to claim 4. A camera installed in an industrial vehicle;
The display device according to claim 1 or 2;
A visual aid system. A camera; the display device according to claim 1 or 2;
An industrial vehicle equipped with A method for controlling a display device to be worn on a person's head, the display device having a transparent display unit capable of displaying image data, comprising the steps of:
An image of a front portion of the industrial vehicle is acquired by a camera installed in the industrial vehicle;
The image data including the virtual monitor on which the image is displayed is displayed in a manner superimposed on an object that is loaded on the industrial vehicle and blocks the forward view and is visible through the display unit.
Control methods. A computer of a display device mounted on a person's head, the computer having a transparent display capable of displaying image data,
A means for acquiring an image of a front portion of the industrial vehicle captured by a camera provided on the industrial vehicle;
a means for displaying the image data including a virtual monitor on which the image is displayed, superimposed on an object that is loaded on the industrial vehicle and blocks the forward view and is visible through the display unit;
A program to function as a

Images