JP2017058840A - Display system, display device control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display system capable of controlling a display device worn on a user's head by a more intuitive input operation, a control method of the display device, and a program. A head-mounted display device 100 includes a head-mounted display device 100 having a display unit mounted on a user's head, and an operation device 3 configured as a separate body from the display unit. Be prepared. The head-mounted display device 100 corresponds to the operation device 3 and performs AR display related to the outside view transmitted through the image display unit 20 by the image display unit 20. [Selection diagram] Fig. 1

Description

  The present invention relates to a display system including a display device, a control method for the display device, and a program.

  2. Description of the Related Art Conventionally, as an input operation method for a display device such as a head mounted display (HMD) used by a user wearing on the head, the device is attached to a finger, and the movement of the finger is detected by this device. A method has been proposed (see, for example, Patent Documents 1 and 2). Patent Document 1 discloses a configuration for inputting text using a device worn on a finger. Patent Document 2 discloses a configuration in which an operation equivalent to that of a mouse is performed using an HMD and a light emitter that is worn on a finger.

JP 2000-284886 A JP 2000-29619 A

When an input operation equivalent to a text input or a mouse is performed in the HMD, if there is an erroneous input, a result contrary to the user's intention is brought about. For this reason, the user pays attention to the input character or the cursor indicating the indicated position, and performs an operation so as to be accurate and error-free. For this reason, the user is required to pay careful attention to the display on the display device and carefully operate it, and a method capable of controlling the HMD with a more intuitive operation is desired.
The present invention has been made in view of the above circumstances, and provides a display system, a display device control method, and a program capable of controlling a display device mounted on a user's head by a more intuitive input operation. The purpose is to do.

In order to achieve the above object, a display system according to the present invention includes a display device that is mounted on a user's head and has a transmissive display unit that allows the user to see through the outside scene, and the display unit. And an operation object configured as a separate body, and the display device includes a control unit that performs AR display related to the outside scene that is transmitted through the display unit by the display unit, corresponding to the operation object. It is characterized by providing.
According to the present invention, since the display device mounted on the user's head performs AR display related to the outside scene in response to the operation of the operation device separate from the display device, intuitive input operation Can control the AR display.

The present invention is characterized in that, in the display system, the display device displays the control target corresponding image associated with the operation target in an AR display by the display unit in association with the operation target.
According to the present invention, it is possible to display information related to the operation of the operation target in a form that is easy to grasp intuitively.

In the display system according to the present invention, the operation target includes an operation unit that receives an operation of the user, and a communication unit that transmits data indicating the operation received by the operation unit to the display device. The control unit of the display device performs AR display by the display unit based on an operation of the operation device.
According to the present invention, a display device mounted on a user's head can quickly detect an operation of an operation device separate from the display device, and can perform an AR display corresponding to the operation.

According to the present invention, in the display system, the control unit of the display device displays an AR display related to the outside scene that is transmitted through the display unit in a mode associated with the operation unit based on an operation of the operation device. Is performed by the display unit.
According to the present invention, the operation on the operation device can be reliably detected in the display device. Further, it is possible to reduce the load on the display device related to the operation detection.

According to the present invention, in the display system, the control unit detects an operation of the operation unit based on data transmitted from the operation device, and the display unit displays the operation corresponding to the detected operation. It is characterized by changing the AR display.
According to the present invention, the AR display can be changed by operating the operation device while the AR display is being performed.

In the display system according to the present invention, the control unit may be capable of setting the allocation of the AR display to the operation unit.
In this case, the control unit may be capable of executing a process for updating the allocation of the AR display to the operation unit.

In the display system according to the present invention, the control unit of the display device obtains a relative position between the operation device and the display device, and performs AR display on the display unit based on the obtained relative position. And
According to the present invention, display can be controlled corresponding to the relative position between the operation device and the display device.

In the display system according to the aspect of the invention, the operation unit includes a touch operation unit that detects a contact operation, the touch operation unit includes a plurality of operation areas, and the control unit is operated by the touch operation unit. The AR display assigned to the operation area is displayed by the display unit.
According to the present invention, various operations for AR display can be performed by performing a touch operation with an operation device.

The present invention is characterized in that, in the display system, the operation unit of the operation device detects a movement of the operation device as an operation.
According to the present invention, the display device can be operated by moving the operation device.

The present invention is characterized in that, in the display system, the operation device includes an imaging unit, and data including a captured image of the imaging unit is transmitted to the display device by the communication unit.
According to the present invention, it is possible to capture an image with an operation device separate from the display device and display the captured image on the display device.

In order to achieve the above object, a display system according to the present invention includes a display device that is mounted on a user's head and has a transmissive display unit that allows the user to see through the outside scene, and the display unit. And a control unit that performs display control including a process of transitioning a control screen displayed by the display unit based on an operation on the operation target. It is characterized by providing.
According to the present invention, the screen is changed according to the operation of the operation device separate from the display device with respect to the display device mounted on the user's head. Can be controlled.

In the display system according to the present invention, the operation target is an operation device including an operation unit that receives an operation of the user.
According to the present invention, it is possible to detect an operation on an operation device separate from the display device.

In the display system according to the aspect of the invention, the operation device includes a communication unit that transmits data indicating the operation accepted by the operation unit to the display device, and the control unit of the display device transmits the data from the operation device. The operation of the operation device is detected based on the data to be processed.
According to the present invention, the operation on the operation device can be reliably detected in the display device. Further, it is possible to reduce the load on the display device related to the operation detection.

In the display system according to the aspect of the invention, the operation unit included in the operation device detects an operation involving rotation, and the control unit of the display device transmits data indicating the operation accompanied by rotation from the operation device. In this case, the screen displayed by the display unit is changed corresponding to the rotation direction of the operation.
According to the present invention, since the screen displayed corresponding to the rotation direction is changed by an operation involving rotation, a more intuitive operation can be realized. Further, since the direction of operation corresponds to the direction of transition of the screen, for example, a groping operation can be performed.

In the display system according to the aspect of the invention, the control unit of the display device performs display control including any of enlargement, reduction, and rotation of an image displayed by the display unit based on data transmitted from the operation device. It is characterized by performing.
According to the present invention, it is possible to control display of an image displayed on a display device through an intuitive operation.

In the display system according to the aspect of the invention, the display device transmits output data to the operation device, and the operation device receives the output data transmitted from the display device at the communication unit, and receives the output data. The output data is output.
According to the present invention, the operation device can be used as an apparatus for outputting data.

The present invention is characterized in that, in the display system, the operation device includes an imaging unit, and data including a captured image of the imaging unit is transmitted to the display device by the communication unit.
According to the present invention, it is possible to capture an image with an operation device separate from the display device and display the captured image on the display device.

According to the present invention, in the display system, the control unit of the display device receives data including a captured image from the operation device, and determines a relative position between the operation device and the display device based on the received data. It is characterized by seeking.
According to the present invention, the relative position between the operation device and the display device can be quickly obtained.

The present invention is characterized in that, in the display system, the operation unit of the operation device detects a movement of the operation device as an operation.
According to the present invention, the display device can be operated by moving the operation device.

The present invention is characterized in that, in the display system, the control unit of the display device obtains a relative position between the operation device and the display device, and controls display of the display unit based on the obtained relative position. And
According to the present invention, display can be controlled corresponding to the relative position between the operation device and the display device.

In the display system according to the present invention, the display device includes an imaging unit, and the control unit obtains a relative position between the operation device and the display device based on a captured image of the imaging unit. And
According to the present invention, the relative position between the operation device and the display device can be quickly obtained.

The present invention is characterized in that, in the above display system, the display unit is a transmissive display unit that allows the user to visually recognize an outside scene.
According to the present invention, the user can control the display of the display device by operating the operation device while looking at the outside scene. Since it can be operated intuitively without gazing at only the display device, it is suitable when the display device is used while looking at the outside scene.

The present invention is characterized in that, in the above display system, the operation device has a mounting portion to be mounted on the user's body.
ADVANTAGE OF THE INVENTION According to this invention, more intuitive input operation can be performed using the operation device with which a user's body is mounted | worn, and a display apparatus can be controlled.

In the display system according to the aspect of the invention, when the control unit included in the display device detects the operation target, the control target corresponding image associated with the operation target corresponds to the operation target. And displaying on the display unit.
ADVANTAGE OF THE INVENTION According to this invention, the information regarding the operation target separate from a display apparatus can be provided in a user-friendly manner.

In the display system according to the aspect of the invention, when the control unit included in the display device detects an operation on the operation target object that displays the control target corresponding image, the control unit corresponds to the operation content and the control target corresponding image. It is characterized by executing a process to perform.
According to the present invention, since the display device executes processing in response to an operation on the operation target, an intuitive operation can be realized using the operation target.

The present invention is characterized in that, in the display system, the display device includes a detection unit that detects an operation on the operation target.
According to the present invention, the display device can quickly detect an operation on the operation target. Moreover, even if the operation target does not have a communication function, the display device can execute processing corresponding to the operation of the operation target.

In the display system according to the aspect of the invention, the display device includes an imaging unit, and the control unit detects the operation target based on a captured image of the imaging unit, and detects an operation on the operation target. It is characterized by.
According to the present invention, even if the operation target does not execute communication, the display device can detect the operation target and detect an operation on the operation target.

In the display system according to the present invention, the display device includes a data storage unit that stores data for AR display for AR display of an operation image corresponding to the operation unit included in the operation target, The control unit can change the AR display data stored in the data storage unit.
According to the present invention, since the data can be changed for the function of performing the AR display corresponding to the operation target, the data can be changed to correspond to a new operation target or an unknown operation target. Thereby, the restrictions regarding the operation target object which is the target of AR display can be reduced, and the convenience can be improved.

In the display system, the AR display data stored in the data storage unit includes data that associates the operation image with the operation unit included in the operation target, and the control unit includes: In the AR display data stored in the data storage unit, the correspondence between the operation image and the operation unit included in the operation object can be changed.
According to the present invention, the correspondence between the operation object and the AR display can be changed.

In order to achieve the above object, the present invention is implemented by a control unit that controls a display device that is mounted on the user's head and has a transmissive display unit that allows the user to see through the outside scene. The control unit detects an operation target configured as a separate body from the display unit, detects an operation unit included in the operation target, and operates the detected operation target An operation image is displayed at a position corresponding to the unit, an operation related to the operation image is detected, and processing corresponding to the operation image from which the operation has been detected is executed.
According to the present invention, the display device mounted on the user's head displays an operation image corresponding to an operation target separate from the display device, and according to an operation related to the operation image. Execute the process. For this reason, it becomes possible to make a display apparatus perform a process using an operation target object, and intuitive operation with respect to a display apparatus is attained.

In order to achieve the above object, the present invention is a program executable by a computer for controlling a display device having a display unit mounted on a user's head, which is separate from the display unit. When the configured operation target receives an operation of the user, the display unit performs AR display related to an outside scene that is transmitted through the display unit based on the operation of the operation target. To do.
According to the present invention, when a display device having a display unit mounted on a user's head performs AR display related to an outside scene, the operation of an operation target separate from the display device is reflected. For this reason, the AR display can be controlled by an intuitive input operation.

In order to achieve the above object, a method for controlling a display device according to the present invention includes: a display device having a display unit mounted on a user's head; an operation target configured separately from the display unit. In a control method using an object, the operation unit of the operation object receives an operation of the user, and the display device is related to an outside scene that is transmitted through the display unit based on the operation of the operation object. The AR display is performed by the display unit.
According to the present invention, when a display device having a display unit mounted on a user's head performs AR display related to an outside scene, the operation of an operation target separate from the display device is reflected. For this reason, the AR display can be controlled by an intuitive input operation.

In order to achieve the above object, the present invention is a program executable by a computer for controlling a display device having a display unit mounted on a user's head, which is separate from the display unit. When a configured operation target receives an operation of the user, display control including a process of transitioning a screen displayed by the display unit is performed based on the operation of the operation target. .
According to the present invention, for a display device having a display unit mounted on the user's head, an operation object separate from the display device is operated and the screen is changed. Thus, the display device can be controlled.

In order to achieve the above object, the server device of the present invention is connected to a communication line, and transmits the above program through the communication line.
According to the present invention, for a display device having a display unit mounted on the user's head, an operation object separate from the display device is operated and the screen is changed. Thus, the display device can be controlled.

In order to achieve the above object, a method for controlling a display device according to the present invention includes: a display device having a display unit mounted on a user's head; an operation target configured separately from the display unit. In a control method for controlling using an object, the user's operation is accepted by the operation object, data indicating the accepted operation is transmitted to the display unit, and transmitted from the operation object by the display device Based on the data, display control including a process of transitioning the screen displayed by the display unit to another screen is performed.
According to the present invention, for a display device having a display unit mounted on the user's head, an operation object separate from the display device is operated and the screen is changed. Thus, the display device can be controlled.

  The present invention can also be realized in various forms other than the head-mounted display device. For example, a head-mounted display device control method, an information system having a head-mounted display device, a head-mounted display device control method, a computer program for realizing the information system, and the computer program recorded The present invention can be realized in the form of a recording medium, a server device for distributing a computer program, a data signal including the computer program and embodied in a carrier wave.

Explanatory drawing which shows schematic structure of the display system which concerns on embodiment. Explanatory drawing which shows the structure of an operation device. The functional block diagram of an operation device. The functional block diagram of a head mounted type display apparatus. The flowchart which shows operation | movement of an operation device. The flowchart which shows operation | movement of a head mounting | wearing type display apparatus. Explanatory drawing which shows the example of AR display for a setting. The figure which shows the example of the process performed according to operation of an operation device. The flowchart which shows operation | movement of a head mounting | wearing type display apparatus. The flowchart which shows another example of operation | movement of a head mounted type display apparatus. The figure which shows the example of the process performed according to operation of an operation device. The figure which shows another example of the process performed according to operation of an operation device. Explanatory drawing which shows the structural example of the operation target object. Explanatory drawing which shows another structural example of an operation target object. The flowchart which shows operation | movement of a head mounting | wearing type display apparatus. Explanatory drawing which shows the example of AR display corresponding to an operation target object. Explanatory drawing which shows another example of AR display corresponding to an operation target object. The figure which shows the structural example of the data which a memory | storage part memorize | stores. The flowchart which shows the process which edits AR content data. The flowchart which shows the process which downloads AR content data. The flowchart which shows the process which downloads AR content data.

[System configuration]
FIG. 1 is a diagram showing a schematic configuration of a display system 1 according to an embodiment to which the present invention is applied.
The display system 1 includes a head-mounted display device 100 and an operation device 3 that are worn on a user's body.
As shown in FIG. 1, the head-mounted display device 100 includes an image display unit 20 that is mounted on the user's head. The head-mounted display device 100 includes an image display unit 20 (display unit) that allows the user to visually recognize a virtual image while being mounted on the user's head, and a control device 10 that controls the image display unit 20. I have. The image display unit 20 is a mounting body to be mounted on the user's head, and has a glasses-shaped frame 2 (main body) in the present embodiment. The control device 10 also functions as a controller for the user to operate the head-mounted display device 100. The control device 10 is accommodated in the pocket of the user's clothes or attached to the belt on the user's waist.
The operation device 3 is a so-called wearable device that can be worn on the user's body. In the present embodiment, the operation device 3 has a watch-like shape that is worn on the user's arm.

[Operation device configuration]
FIG. 2 is an explanatory diagram showing a configuration of the operation device 3, FIG. 2A is a front view of a main part of the operation device 3, and FIGS. 2B and 2C are diagrams showing display examples on the LCD 303. . FIG. 3 is a functional block diagram of the operation device 3. The configuration of the operation device 3 will be described with reference to FIGS.

  The operation device 3 has a band portion 300 having a shape similar to that of a wristwatch band. The band part 300 has a fixing part such as a buckle (not shown), and can be wound and fixed around, for example, a user's forearm. In the band portion 300 of the operation device 3, a substantially disc-shaped plane portion 300A is formed at a position corresponding to a dial of a watch. The plane portion 300A is provided with a bezel 301, an LCD 303, a button 305, a crown-type operation element 307, a plurality of buttons 309, and an optical unit 310.

  The bezel 301 is a ring-shaped operation element, and is disposed on the peripheral portion of the flat surface portion 300A. The bezel 301 is provided so as to be rotatable in the circumferential direction with respect to the band unit 300. As will be described later, the operation device 3 includes a mechanism for detecting the rotation direction and the rotation amount of the bezel 301. The mechanism that instructs the band unit 300 to rotate the bezel 301 may include a memory that generates a notch sound when the bezel 301 is rotated.

  The LCD 303 is an LCD (Liquid Crystal Display) that displays characters and images. A touch panel 304 (touch operation unit) shown in FIG.

  The button 305 is a push button type switch disposed outside the bezel 301. For example, as shown in FIGS. 1 and 2A, the button 305 is positioned below the bezel 301 when viewed from the user in the mounted state. The button 305 is larger than the crown-type operation element 307 and the button 309 described later, and can be operated by groping.

  The crown-type operation element 307 is an operation element having a shape imitating a crown of a wristwatch, and can be rotated as indicated by an arrow in the drawing. The operation device 3 includes a mechanism for detecting the rotation direction and the rotation amount of the crown-type operation element 307 when the user rotates the crown-type operation element 307. In addition, the mechanism that instructs the band unit 300 to rotate the crown-type operation element 307 may include a memory that generates a notch sound when the crown unit 307 is rotated.

The button 309 is a push button type switch provided on the outer peripheral portion of the plane portion 300A. The number of buttons 309 is not particularly limited, and in the present embodiment, an example in which four buttons 309 are provided is shown.
Different functions can be assigned to the buttons 309, and the functions assigned to the buttons 309 can be displayed on the LCD 303.

  FIG. 2B shows an example in which the function of the button 309 is displayed on the LCD 303. In this example, function displays 331, 332, 333, and 334 that indicate functions assigned to the four buttons 309 are displayed on the LCD 303. The display positions of the function displays 331, 332, 333, and 334 correspond to the position of the function display target button 309. For example, a function display 331 displayed on the upper right portion of the LCD 303 indicates a function assigned to the button 309 located on the upper right portion of the plane portion 300A.

  Further, the LCD 303 can display selectable functions by rotating the bezel 301. FIG. 2C shows an example in which functions assigned to the operation of the bezel 301 are displayed on the LCD 303. In this example, function displays 341, 342, 343, and 344 indicating four functions that can be selected by rotating the bezel 301 are displayed on the LCD 303. The display positions of the function displays 341, 342, 343, and 344 correspond to the rotational position of the bezel 301. That is, when the bezel 301 is rotated, the function displays 341, 342, 343, and 344 are selected by switching in order. For example, when the bezel 301 is rotated clockwise (CW), the function displays 341, 342, 343, and 344 are sequentially switched and selected.

  A function display 345 indicates a function assigned to the button 305, which is a “decision” function in the example of FIG. When the button 305 is operated in a state where any of the function displays 341, 342, 343, and 344 is selected by the rotation of the bezel 301, the selection of the function displays 341, 342, 343, and 344 is determined. . Then, the function selected by the rotation of the bezel 301 is executed.

  The function assigned to the button 309 and the function selected by the rotation of the bezel 301 are not limited to one, and more functions can be switched and assigned. In this case, the operation device 3 can be configured to be able to switch functions. In this case, the function display 331, 332, 333, 334 or the function display 341, 342, 343, 344 of the LCD 303 may be switched as the function is switched.

The optical unit 310 includes two cameras 311 (imaging unit) and an LED 313. The camera 311 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like. The imaging ranges of the two cameras 311 included in the operation device 3 may be different ranges or the same range. In the present embodiment, the two cameras 311 can capture both eyes of the user wearing the operation device 3, and can detect the user's line of sight based on the captured image. Therefore, one of the two cameras 311 is configured to capture the right eye of the user who views the plane portion 300A from the front, and the other can capture the left eye.
An LED (Light Emitting Diode) 313 is an infrared LED that emits infrared light. The LED 313 is used as illumination when the camera 311 captures an image. The operation device 3 can transmit an optical signal by turning on or blinking the LED 313.

FIG. 3 shows the configuration of the control system of the operation device 3.
The operation device 3 includes a control unit 350 that controls each unit of the operation device 3. The control unit 350 includes a CPU, ROM, RAM, and the like (not shown), and realizes the function of the operation device 3 by executing a program stored in the ROM.
The control unit 350 includes a wireless communication unit 361 (communication unit) and an NFC (Near Field Communication) unit 362 as functional units for communication. The wireless communication unit 361 performs wireless data communication conforming to a standard such as a wireless LAN (WiFi (registered trademark)), Miracast (registered trademark), Bluetooth (registered trademark), or the like. The NFC unit 362 performs short-range wireless communication based on the NFC standard.

The control unit 350 is connected to the LCD 303, the touch panel 304, the two cameras 311, and the LED 313. The control unit 350 controls the LCD 303 to display images and characters. The touch panel 304 is arranged so as to overlap the surface of the LCD 303 and detects a touch operation on the surface of the LCD 303. For the touch panel 304, for example, a capacitance type sensor or a pressure sensitive sensor can be used. The control unit 350 detects a contact operation using the touch panel 304 and specifies an operation position. The touch panel 304 may detect a single contact operation or may correspond to a so-called multi-touch operation that can simultaneously detect operations at a plurality of points.
The control unit 350 controls each of the two cameras 311 to execute imaging and generate captured image data. Further, the control unit 350 turns on / off the current flowing through the LED 313, and turns on or blinks the LED 313 at an arbitrary timing.

  A microphone 365, a button operation unit 366, a rotation detection unit 367, a 9-axis sensor 368, a vibrator 369, and a speaker 370 are connected to the control unit 350. The microphone 365 may be, for example, a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone. When the microphone 365 has directivity, the microphone 365 collects sound in the front direction of the plane portion 300A, for example, at or near the plane portion 300A (FIG. 2). The microphone 365 may be a bone conduction microphone that contacts the user's body. The control unit 350 acquires an analog audio signal of the audio collected by the microphone 365 and generates digital audio data.

  The button operation unit 366 detects the operation of the buttons 305 and 309 and outputs an operation signal corresponding to the operated button to the control unit 350. The rotation detection unit 367 detects the rotation of the bezel 301 and the rotation of the crown type manipulator 307. The rotation detection unit 367 outputs a signal indicating the rotation direction and the rotation amount of the bezel 301 to the control unit 350, and similarly outputs a signal indicating the rotation direction and the rotation amount to the control unit 350 for the crown-shaped operation element 307. Here, the rotation detection unit 367 may output a signal indicating the rotation direction every time the bezel 301 rotates by a predetermined amount. In this case, since the control unit 350 counts the output signal of the rotation detection unit 367, the control unit 350 can acquire the rotation amount, and the rotation detection unit 367 does not need to integrate the rotation amount. Similarly, for the rotation operation of the crown-type operation element 307, the rotation detection unit 367 may output a signal indicating the rotation direction every time the crown-type operation element 307 rotates by a predetermined amount.

The 9-axis sensor 368 is a unit including an inertial sensor, and in this embodiment, includes a 3-axis acceleration sensor, a 3-axis angular velocity sensor, and a 3-axis magnetic sensor. The axis on which the nine-axis sensor 368 detects can be set, for example, in the vertical direction, the horizontal direction, and the direction orthogonal to the plane including the plane portion 300A in the plane including the plane portion 300A.
Vibrator 369 includes a motor (not shown) that rotates under the control of control unit 350. A weight is eccentrically attached to the rotation shaft of the motor, and the operation device 3 is vibrated as the motor rotates. The speaker 370 outputs audio based on the audio signal output from the control unit 350.

  Power is supplied to each unit of the operation device 3 including the control unit 350 from a power supply unit 360 having a primary battery or a secondary battery. The power supply unit 360 may include a circuit that controls charging of the secondary battery together with the secondary battery.

The control unit 350 realizes the functions of the operation detection unit 351, the output control unit 352, the imaging control unit 353, the communication control unit 354, and the sound processing unit 355 by executing a program.
The operation detection unit 351 detects the operation of the operator included in the operation device 3 based on the signals input from the button operation unit 366 and the rotation detection unit 367, and specifies the operation content. Specifically, the operation detection unit 351 detects the operation of the button 305, the crown type operation element 307, and the button 309. Further, the operation detection unit 351 detects an operation such as shaking or moving the operation device 3 based on a signal input from the 9-axis sensor 368. Further, the operation detection unit 351 detects a touch operation on the LCD 303 based on a signal input from the touch panel 304 and specifies an operation position. Here, the operation detection unit 351 can specify the function assigned to the operation position by specifying the image displayed on the LCD 303 at the operation position.

The output control unit 352 executes output in response to data received by the communication control unit 354 by the wireless communication unit 361 or the NFC unit 362 or an operation detected by the operation detection unit 351. The output control unit 352 displays characters and images on the LCD 303. The output control unit 352 operates the motor of the vibrator 369 to vibrate the operation device 3.
The imaging control unit 353 controls the camera 311 to execute imaging, and acquires captured image data.
The communication control unit 354 transmits / receives data to / from an external device through the wireless communication unit 361 or the NFC unit 362. The communication control unit 354 can execute data communication with the head-mounted display device 100 through the wireless communication unit 361. Further, when the NFC unit 362 detects a device capable of near field communication, the communication control unit 354 transmits and receives data to and from this device. Further, the communication control unit 354 can transmit data using an optical signal by blinking the LED 313. In this case, the communication control unit 354 may be configured to detect and receive an optical signal with the camera 311.

  The sound processing unit 355 generates digital sound data of the sound collected by the microphone 365 and performs analysis as necessary. For example, the voice processing unit 355 may analyze the user's voice collected by the microphone 365 and output an operation instruction to the operation detection unit 351 when detecting a voice instructing the operation. In this case, the user can operate the operation device 3 by voice. In addition, the audio processing unit 355 outputs an audio signal to the speaker 370 to output audio.

  The control unit 350 can execute a plurality of operation modes including a clock mode and a UI mode as operation modes of the operation device 3. The clock mode is an operation mode in which the control unit 350 measures the current time and displays the current time on the LCD 303. The clock mode has an advantage that the power consumption is small because the operation device 3 is operated as a clock. The UI mode is an operation mode in which the operation device 3 functions as a user interface (UI) for operating the head-mounted display device 100. In the UI mode, the operation device 3 transmits data to the head-mounted display device 100 based on an operation on the operation device 3, and the head-mounted display device 100 operates according to this data. Switching from the clock mode to the UI mode and switching from the UI mode to the clock mode are performed under the control of the control unit 350, as will be described later.

[Configuration of head-mounted display device]
In the head-mounted display device 100, the frame 2 of the image display unit 20 includes a right holding unit 21 and a left holding unit 23. The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 contacts the right ear or the vicinity thereof on the user's head, and the left holding unit 23 contacts the user's left ear or the vicinity thereof and holds the image display unit 20 on the user's head. To do.

The frame 2 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61 ( An imaging unit) and a microphone 63 are provided.
The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. The right display drive unit 22 and the left display drive unit 24 are collectively referred to simply as “display drive unit”, and the right optical image display unit 26 and the left optical image display unit 28 are simply referred to as “optical image display unit”. Also called.

The display driving units 22 and 24 include liquid crystal displays 241 and 242 (Liquid Crystal Display, hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252, and the like.
The right optical image display unit 26 and the left optical image display unit 28 include light guide plates 261 and 262 (FIG. 2) and a light control plate 20A. The light guide plates 261 and 262 are formed of a light transmissive resin or the like, and guide the image light output from the display driving units 22 and 24 to the user's eyes. The light control plate 20A is a thin plate-like optical element, and is disposed so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. As the light control plate 20A, various materials such as a material that has almost no light transmission, a material that is nearly transparent, a material that attenuates the amount of light and transmits light, and a material that attenuates or reflects light of a specific wavelength may be used. it can. By appropriately selecting the optical characteristics (light transmittance, etc.) of the light control plate 20A, the external light amount incident on the right optical image display unit 26 and the left optical image display unit 28 from the outside is adjusted to visually recognize the virtual image. You can adjust the ease. In the present embodiment, a case will be described in which at least a light control plate 20A having such a light transmittance that a user wearing the head-mounted display device 100 can visually recognize an outside scene is used. The light control plate 20A protects the right light guide plate 261 and the left light guide plate 262, and suppresses damage to the right light guide plate 261 and the left light guide plate 262, adhesion of dirt, and the like.
The dimming plate 20A may be detachable from the right optical image display unit 26 and the left optical image display unit 28, or may be mounted by replacing a plurality of types of dimming plates 20A, or may be omitted. .
Then, the head-mounted display device 100 causes the image light of the image to be processed inside and the external light to overlap and enter the user's eyes, and the user can see the outside scene through the dimming plate 20A, An image of image light is visually recognized over the outside scene. Thus, the head-mounted display device 100 functions as a see-through display device.

The camera 61 is disposed at the boundary between the right optical image display unit 26 and the left optical image display unit 28. In a state where the user wears the image display unit 20, the position of the camera 61 is substantially in the middle of both eyes of the user in the horizontal direction and above the eyes of the user in the vertical direction. The camera 61 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like, and may be a monocular camera or a stereo camera.
The camera 61 images at least a part of the outside view in the front side direction of the head-mounted display device 100, in other words, the user's view direction when the head-mounted display device 100 is mounted. The width of the angle of view of the camera 61 can be set as appropriate, but the imaging range of the camera 61 is a range that includes the outside world visually recognized by the user through the right optical image display unit 26 and the left optical image display unit 28. preferable. Furthermore, it is more preferable that the imaging range of the camera 61 is set so that the entire field of view of the user through the light control plate 20A can be imaged.
The camera 61 executes imaging under the control of the control unit 140 (FIG. 4), and outputs captured image data to the control unit 140.

  The image display unit 20 is connected to the control device 10 via the connection unit 40. The connection unit 40 includes a main body cord 48, a right cord 42, a left cord 44, and a connecting member 46 that are connected to the control device 10. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the housing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and is connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24.

  The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. A microphone 63 is provided in the vicinity of the earphone plug 30. The earphone plug 30 to the microphone 63 are combined into one cord, and the cord branches from the microphone 63 and is connected to each of the right earphone 32 and the left earphone 34.

  For example, as shown in FIG. 1, the microphone 63 is arranged so that the sound collection unit of the microphone 63 faces the user's line of sight, collects sound, and sends the sound signal to the sound processing unit 187 (FIG. 3). Output. For example, the microphone 63 may be a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone.

  The right cord 42, the left cord 44, and the main body cord 48 only need to be capable of transmitting digital data, and can be constituted by, for example, a metal cable or an optical fiber. Further, the right cord 42 and the left cord 44 may be combined into a single cord.

  The image display unit 20 and the control device 10 transmit various signals via the connection unit 40. The end of the main body cord 48 opposite to the connecting member 46 and the control device 10 are provided with connectors (not shown) that fit together. By fitting the connector of the main body cord 48 and the connector of the control device 10 or removing this fitting, the control device 10 and the image display unit 20 can be contacted and separated.

  The control device 10 controls the head-mounted display device 100. The control device 10 includes switches including an enter key 11, a lighting unit 12, a display switching key 13, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. In addition, the control device 10 includes a track pad 14 that is operated by a user with fingers.

  The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control device 10. The lighting unit 12 includes a light source such as an LED, and notifies the operating state (for example, power ON / OFF) of the head-mounted display device 100 according to the lighting state of the light source. The display switching key 13 outputs, for example, a signal instructing switching of the image display mode in response to the pressing operation.

  The track pad 14 has an operation surface for detecting a contact operation, and outputs an operation signal according to an operation on the operation surface. The detection method on the operation surface is not limited, and an electrostatic method, a pressure detection method, an optical method, or the like can be adopted. The luminance switching key 15 outputs a signal instructing increase / decrease of the luminance of the image display unit 20 in response to the pressing operation. The direction key 16 outputs an operation signal in response to a pressing operation on a key corresponding to the up / down / left / right direction. The power switch 18 is a switch for switching on / off the power of the head-mounted display device 100.

FIG. 4 is a functional block diagram of each part constituting the head-mounted display device 100.
The head-mounted display device 100 includes an interface 125 for connecting various external devices OA that are content supply sources. As the interface 125, for example, an interface corresponding to wired connection such as a USB interface, a micro USB interface, a memory card interface, or the like may be used, and the interface 125 may be configured by a wireless communication interface. The external device OA is an image supply device that supplies an image to the head-mounted display device 100, and a personal computer (PC), a mobile phone terminal, a portable game machine, or the like is used.

The control device 10 includes a control unit 140, an input information acquisition unit 110, a storage unit 120, a transmission unit (Tx) 51, and a transmission unit (Tx) 52.
The input information acquisition unit 110 is connected to the operation unit 111. The operation unit 111 is connected to each operation element of the control device 10, detects an operation of each operation element, and outputs an operation signal corresponding to the detected operation element to the input information acquisition unit 110. The input information acquisition unit 110 acquires the input content input by the operation of the control device 10 based on the signal input from the operation unit 111.

The storage unit 120 is a nonvolatile storage device, and stores various computer programs and data related to these programs. The storage unit 120 may store still image data or moving image data to be displayed on the image display unit 20.
The control device 10 includes a power supply unit 130 including a primary battery and a secondary battery, and supplies power from the power supply unit 130 to each unit of the control device 10 and the image display unit 20.

  A three-axis sensor 113, a voice recognition unit 114, a GPS 115, and a communication unit 117 are connected to the control unit 140. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 acquires the detection value of the triaxial sensor 113. The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and calculates the current position of the control device 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. The GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140.

The communication unit 117 executes wireless data communication conforming to a standard such as a wireless LAN (WiFi (registered trademark)), Miracast (registered trademark), or Bluetooth (registered trademark).
When the external device OA is wirelessly connected to the communication unit 117, the control unit 140 acquires content data from the communication unit 117 and causes the image display unit 20 to display an image. On the other hand, when the external device OA is wired to the interface 125, the control unit 140 acquires content data from the interface 125 and causes the image display unit 20 to display an image. The communication unit 117 and the interface 125 function as a data acquisition unit DA that acquires content data from the external device OA.

  The control unit 140 stores a CPU (not shown) for executing a program, a RAM (not shown) for temporarily storing a program and data executed by the CPU, and a basic control program and data executed by the CPU in a nonvolatile manner. ROM (not shown). The control unit 140 controls each unit of the head-mounted display device 100 by executing a control program by the CPU. The control unit 140 corresponds to a computer that reads and executes a computer program stored in the storage unit 120 and realizes various functions of the control unit 140. The control unit 140 functions as an operating system (OS) 150, an image processing unit 160, a display control unit 170, an imaging processing unit 181, a device control unit 183, an AR display control unit 185, and an audio processing unit 187.

The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). The image processing unit 160 stores the converted digital image signal in the RAM of the control unit 140 for each frame as image data of the target image (Data in the figure). This image data is, for example, RGB data.
Note that the image processing unit 160 may perform a resolution conversion process for converting the resolution of the image data into a resolution suitable for the right display driving unit 22 and the left display driving unit 24 as necessary. The image processing unit 160 also performs image adjustment processing for adjusting the brightness and saturation of image data, 2D image data is created from 3D image data, or 2D / 3D conversion processing is performed to generate 3D image data from 2D image data. May be executed.

  The image processing unit 160 transmits the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data stored in the RAM via the transmission units 51 and 52, respectively. The transmission units 51 and 52 function as a transceiver and execute serial transmission between the control device 10 and the image display unit 20. The image data Data transmitted via the transmission unit 51 is referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is referred to as “left eye image data”.

  The display control unit 170 generates a control signal for controlling the right display drive unit 22 and the left display drive unit 24, and generates and generates image light by each of the right display drive unit 22 and the left display drive unit 24 based on the control signal. Control injection. Specifically, the right LCD 241 is controlled to be turned on / off by the right LCD control unit 211, and the right backlight 221 is turned on / off by the right backlight control unit 201. In addition, the display control unit 170 controls the driving of the left LCD 242 by the left LCD control unit 212 and the driving of the left backlight 222 by the left backlight control unit 202.

  The audio processing unit 187 acquires an audio signal included in the content, amplifies the acquired audio signal, and outputs the amplified audio signal to the right earphone 32 and the left earphone 34. In addition, the sound processing unit 187 acquires sound collected by the microphone 63 and converts it into digital sound data. The voice processing unit 187 may perform a preset process on the digital voice data.

  The image display unit 20 includes an interface 25, a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, and a left light guide plate 262 as a left optical image display unit 28. A camera 61, a vibration sensor 65, and a nine-axis sensor 66.

  The vibration sensor 65 is configured using an acceleration sensor, and is built in the vicinity of the end ER of the right optical image display unit 26 in the right holding unit 21 as shown in FIG. When the user performs an operation of hitting the end ER (knock operation), the vibration sensor 65 detects vibration caused by this operation and outputs the detection result to the control unit 140. Based on the detection result of the vibration sensor 65, the control unit 140 detects a knocking operation by the user.

The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). When the image display unit 20 is mounted on the user's head, the control unit 140 can detect the movement of the user's head based on the detection value of the 9-axis sensor 66. For example, the control unit 140 can estimate the magnitude of the tilt and the tilt direction of the image display unit 20 based on the detection value of the 9-axis sensor 66.
The detection axes of the acceleration sensor and the angular velocity sensor built in the nine-axis sensor 66 can be, for example, three axes of X, Y, and Z shown in FIG. In this example, the left-right direction is the X axis, the front-rear direction is the Y-axis, and the vertical direction is the Z-axis with respect to the head of the user wearing the image display unit 20. More specifically, when the head-mounted display device 100 is worn, the image display unit 20 is in a horizontal position that is perceived by the user with respect to the left and right eyes. In a standard mounting position of the head-mounted display device 100, the detection axis (X axis, Y axis, Z axis) of the 9-axis sensor 66 is aligned with the left, right, front, back, and top perceived by the user. is there. The acceleration sensor of the 9-axis sensor 66 detects acceleration in the X-axis direction, the Y-axis direction, and the Z-axis direction. The angular velocity sensor included in the nine-axis sensor 66 detects rotation (pitch) around the X axis, rotation (yaw) around the Y axis, and rotation (roll) around the Z axis.

The interface 25 includes a connector to which the right cord 42 and the left cord 44 are connected. The interface 25 outputs the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data transmitted from the transmission unit 51 to the corresponding reception units (Rx) 53 and 54. Further, the interface 25 outputs a control signal transmitted from the display control unit 170 to the corresponding reception units 53 and 54, the right backlight control unit 201, or the left backlight control unit 202.
The interface 25 is an interface for connecting the camera 61, the vibration sensor 65, and the 9-axis sensor 66. The detection result of vibration by the vibration sensor 65 and the detection result of acceleration (three axes), angular velocity (three axes), and geomagnetism (three axes) by the nine-axis sensor 66 are sent to the control unit 140 via the interface 25.

  The right display drive unit 22 includes the right backlight 221, the right LCD 241, and the right projection optical system 251 described above. The right display driving unit 22 includes a receiving unit 53, a right backlight (BL) control unit 201 that controls the right backlight (BL) 221, and a right LCD control unit 211 that drives the right LCD 241.

  The reception unit 53 operates as a receiver corresponding to the transmission unit 51 and executes serial transmission between the control device 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data input via the reception unit 53. .

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes the left backlight 222, the left LCD 242 and the left projection optical system 252 described above. The left display driving unit 24 includes a receiving unit 54, a left backlight control unit 202 that drives the left backlight 222, and a left LCD control unit 212 that drives the left LCD 242.

  The reception unit 54 operates as a receiver corresponding to the transmission unit 52 and executes serial transmission between the control device 10 and the image display unit 20. The left backlight control unit 202 drives the left backlight 222 based on the input control signal. The left LCD control unit 212 drives the left LCD 242 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right-eye image data Data input via the reception unit 54. . The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as a right “image light generation unit”. Similarly, the left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are collectively referred to as a left “image light generation unit”.

The imaging processing unit 181 controls the camera 61 to execute imaging, and acquires captured image data.
The device control unit 183 controls the communication unit 117 to perform data communication with the operation device 3 and analyzes data received from the operation device 3. Further, when the blinking pattern of the LED 313 provided in the operation device 3 corresponds to a specific pattern based on the captured image data of the camera 61, the device control unit 183 decodes this pattern and receives data.

  The AR display control unit 185 controls the image processing unit 160 and the display control unit 170 based on the AR content data, and causes the image display unit 20 to display an image for AR display. For example, the storage unit 120 stores the AR content data. In addition, when the AR content data 123 includes audio data, the AR display control unit 185 controls the audio processing unit 187 so that the audio of the content is output from the right earphone 32 and the left earphone 34.

  The AR display control unit 185 displays the AR content while the user is looking at the object through the image display unit 20. The AR display control unit 185 provides information about the object by performing AR display that displays an image, characters, or the like at a position corresponding to the object, or the shape of the object seen through the image display unit 20 Change how it looks. The AR content includes image and character data displayed at a position corresponding to the object. In addition, the AR content may include data for specifying an object and data regarding the display position of an image or a character. The display position of the AR content may be a position overlapping the target object or may be around the target object. The object may be an object, may be a real estate such as a building, may be a moving body such as an automobile or a train, or may be a living organism such as a human being or an animal. The AR display control unit 185 detects an object located in the user's field of view from the captured image data acquired by the imaging processing unit 181. Then, the AR display control unit 185 determines the display position of the AR content corresponding to the detected object, and displays the AR content.

  It is desirable to display the AR content so as to overlap the position where the user visually recognizes the target object or according to the position where the user visually recognizes the target object. For this reason, the AR display control unit 185 detects the image of the target object from the captured image data of the imaging processing unit 181, and in the imaging range of the camera 61 based on the positional relationship between the detected target image and the entire captured image. Identify the position of the object. The AR display control unit 185 determines the display position of the AR content corresponding to the position of the object based on the positional relationship between the imaging range of the camera 61 and the display area of the image display unit 20.

  The audio processing unit 187 acquires an audio signal included in the content, amplifies the acquired audio signal, and outputs the amplified audio signal to the right earphone 32 and the left earphone 34. In addition, the sound processing unit 187 acquires sound collected by the microphone 63 and converts it into digital sound data. The voice processing unit 187 may perform a preset process on the digital voice data.

[Display system operation]
FIG. 5 is a flowchart showing the operation of the display system 1, and particularly shows the operation of the operation device 3.
The operation device 3 operates in the clock mode while the display system 1 is not operated (step S11). When an operation for instructing the transition to the UI mode is performed in the timepiece mode (step S12), the control unit 350 of the operation device 3 performs imaging with the cameras 311 and 311 to detect the user's line of sight (step S12). S13). The control unit 350 performs imaging and analysis of the captured image until a line of sight is detected (step S13; NO).

  When detecting the line of sight (step S13; YES), the controller 350 determines whether or not the head-mounted display device 100 is ON (step S14). In step S <b> 14, for example, the wireless communication unit 361 communicates with the communication unit 117 of the head-mounted display device 100 to detect the operating state of the head-mounted display device 100, thereby performing the determination. Further, for example, the camera 311 may perform imaging under the control of the imaging control unit 353, and the operation state of the head-mounted display device 100 may be determined based on the captured image. For example, when the image displayed by the image display unit 20 passes through the light control plate 20A and appears in the captured image, it may be determined that the head-mounted display device 100 is ON. In addition, the image display unit 20 may be provided with a light emitting unit that emits light according to the operating state, and the operating state may be determined by specifying the light emitting state of the light emitting unit from the captured image.

  When it is determined that the head-mounted display device 100 is not ON (step S14; NO), the control unit 350 performs communication by the wireless communication unit 361 and instructs the head-mounted display device 100 to start. Data is transmitted (step S15). The control unit 350 receives control data indicating activation from the head-mounted display device 100, and proceeds to step S17. When it is determined that the head-mounted display device 100 is ON (step S14; YES), the control unit 350 proceeds to step S17.

  In step S17, control unit 350 determines whether or not authentication of head-mounted display device 100 has been completed. When used in combination with the head-mounted display device 100, the control unit 350 authenticates the head-mounted display device 100 and operates as a user interface of the head-mounted display device 100 that has been successfully authenticated. . If it is determined that the authentication has not been completed (step S17; NO), the control unit 350 transmits / receives authentication data to / from the head-mounted display device 100 that has determined whether or not the authentication is ON in step S14. To authenticate (step S18). When data necessary for authentication is transmitted / received, the earphone plug 30 performs pairing with the head-mounted display device 100, that is, one-to-one association that operates as a user interface (step S19), and proceeds to step S20. . This association may be one-to-many. When the head-mounted display device 100 has been authenticated (step S17; YES), the control unit 350 proceeds to step S20.

In step S20, control unit 350 displays an operation screen on LCD 303 and stands by for the operation. The operation screen is, for example, the screen illustrated in FIG. 2B or 2C.
Control unit 350 waits for an operation, and when operation is detected by operation detection unit 351 (step S21), operation data indicating the detected operation is transmitted to head-mounted display device 100 (step S22).

Thereafter, the control unit 350 determines whether or not an end condition is satisfied (step S23). When the end condition is satisfied (step S23; YES), the control unit 350 shifts to the clock mode (step S24) and ends the process. If the end condition is not satisfied (step S23; NO), the process returns to step S21.
The end condition is, for example, that the operation for instructing the end of the UI mode is performed by the touch panel 304, the button 305, the crown-type operation element 307, or the button 309, and the control data for instructing the end from the head-mounted display device 100. , Etc.

In the display system 1, the head-mounted display device 100 can display an AR image corresponding to the operation by the operation device 3.
FIG. 6 is a flowchart showing the operation of the head-mounted display device 100, and particularly shows the operation of performing AR display corresponding to the operation device 3.
The control unit 140 causes the device control unit 183 to receive and analyze the operation data transmitted from the operation device 3 by the communication unit 117 (step S31), and specify the type of operation (step S32). The data received in step S31 is operation data (step S22 in FIG. 5) transmitted by the operation device 3 in response to the operation. The device control unit 183 determines whether or not the instruction corresponding to the operation indicated by the operation data is an instruction related to the operation setting of the operation device 3 (step S33). When the instruction is other than the operation setting (step S33; NO), the device control unit 183 executes a process corresponding to the operation specified in step S32 (step S34), and ends this process. In step S34, for example, the AR display control unit 185 starts / ends the AR display according to the process of starting the imaging of the camera 61 according to the operation data instructing the start of imaging or the operation data instructing the start or end of the AR display. .

  On the other hand, when the instruction corresponding to the operation indicated by the operation data is an instruction related to operation setting (step S33; YES), the AR display control unit 185 performs AR display for operation setting (step S35). Here, when the operation device 3 transmits operation data in response to an operation on the operation device 3 during AR display, the device control unit 183 receives the operation data, and sets and sets the operation data based on the received operation data. Update the settings.

7A and 7B show an example of AR display for operation setting, FIG. 7A shows an example of AR display for setting the operation of the bezel 301, and FIG. 7B shows the setting for operation of the button 309. An example of AR display is shown.
In the example of FIG. 7A, the actual operation device 3 is visually recognized in the user's visual field VR, and the AR content is displayed by the image display unit 20 at a position corresponding to the operation device 3. The AR contents are assignment candidate displays A1, A2, A3, A4 and setting candidates A11, A12, A13, A14. The assignment candidate displays A1, A2, A3, and A4 are images showing candidate functions to be assigned to the rotation of the bezel 301. In this example, the assignment candidate displays A1, A2, A3, and A4 include text indicating combinations of functions that can be selected by rotating the bezel 301. The assignment candidate display A1 includes four character strings (text) of “Home”, “Return”, “Menu”, and “App end”. The allocation candidate display A2 includes four character strings of “forward”, “return”, “imaging”, and “imaging end”. When the operation of rotating the bezel 301 is performed on the operation device 3, any of the allocation candidates A1 to A4 can be selected in order.

  In the example of FIG. 7A, the allocation candidate A2 is selected. For this reason, four character strings of “forward”, “return”, “imaging”, and “imaging end” are arranged so as to overlap the operation device 3 visible in the center of the visual field VR. These character strings are candidate functions to be assigned, that is, setting candidates A11, A12, A13, and A14. When the assignment candidate display A2 is switched to another assignment candidate display by operating the bezel 301, the character strings of the setting candidates A11, A12, A13, and A14 are changed. Therefore, the user can select and set the function assigned to the rotation operation of the bezel 301 from the assignment candidate displays A1, A2, A3, and A4. When the button 305 is operated on the operation device 3, the assignment candidate display selected by the operation of the bezel 301 is confirmed and the setting is updated.

  In the example of FIG. 7B, assignment candidate displays A21, A22, A23, and A24 and setting candidates A31, A32, A33, and A34 are displayed as AR contents at positions corresponding to the actual operation device 3. The assignment candidate displays A21, A22, A23, and A24 are images showing function candidates to be assigned to the four buttons 309. In this example, the functions executed when each of the four buttons 309 are operated are shown. Contains text. The allocation candidate display A21 includes four character strings (text) of “Home”, “Return”, “Menu”, and “App end”. The allocation candidate display A22 includes four character strings of “forward”, “return”, “imaging”, and “imaging end”. When the operation of rotating the bezel 301 is performed on the operation device 3, any of the allocation candidates A21 to A24 can be selected in order.

  In the example of FIG. 7B, allocation candidate A22 is selected. For this reason, four character strings of “forward”, “return”, “imaging”, and “imaging end” are arranged so as to overlap the operation device 3 visible in the center of the visual field VR. These character strings are candidates for functions to be assigned, that is, setting candidates A31, A32, A33, and A34, and are displayed at positions corresponding to buttons 309 to which functions are assigned. When the assignment candidate display A22 is switched to another assignment candidate display by the operation of the bezel 301, the character strings of the setting candidates A31, A32, A33, and A34 are changed. Therefore, the user can select and set the function assigned to the rotation operation of the bezel 301 from the assignment candidate displays A21, A22, A23, and A24. When the button 305 is operated on the operation device 3, the assignment candidate display selected by the operation of the bezel 301 is confirmed and the setting is updated.

  The examples of FIGS. 7A and 7B show an operation of setting a function to be executed by the head-mounted display device 100 corresponding to the operation data transmitted by the operation device 3. Since the assignment of functions can be set and updated, various functions of the head-mounted display device 100 can be operated by the operation device 3. Moreover, the setting can be easily performed by using the AR display.

  FIG. 8 is an explanatory diagram illustrating an example of an operation performed by the head-mounted display device 100 in response to an operation of the operation device 3. FIG. 8 shows an example in which the display mode of the AR display displayed on the image display unit 20 is changed according to the operation of the operation device 3 under the control of the AR display control unit 185. FIG. 8A shows an example of AR display by the image display unit 20, and FIG. 8B shows an example in which the display mode of the AR display is changed.

In the example of FIG. 8A, a stage, performers, and audience seats are visible as an actual outside scene in the visual field VR. In this example, the character string A51 related to the performance being performed on the stage is displayed so as to overlap the actual outside scene. The character string A51 is, for example, text indicating a dialogue of the stage, and AR is displayed at a position overlapping the stage.
The head-mounted display device 100 changes the display size of the character string A51 according to the operation of the operation device 3, and displays the reduced size as shown in FIG. 8B, for example. In this operation, the device control unit 183 analyzes the operation data transmitted by the operation device 3, and when the device control unit 183 detects that the operation corresponding to the reduction of the AR display has been performed, the AR display control unit 185 Change the display size.

  In the head-mounted display device 100, as described with reference to FIGS. 6 and 7, the operation performed by the head-mounted display device 100 can be associated with the operation on the operation device 3. This association is not limited to the operation of the button 309 and the rotation operation of the bezel 301. For example, an operation of pressing the button 305 and an operation of rotating the crown-type operation element 307 may be used. Further, for example, an operation of touching the touch panel 304 with a finger may be included.

Specifically, the display image may be enlarged in response to an operation of turning the bezel 301 to the right, and the display image may be reduced in response to an operation of turning the bezel 301 to the left. Further, the display image is enlarged when a finger contact with the center of the touch panel 304 is detected, and the display image is reduced when a finger contact with the peripheral edge of the touch panel 304 (for example, near the lower left frame) is detected. Also good. Further, when the finger touching the touch panel 304 rotates in a clockwise direction or in a counterclockwise direction while being in contact, the display image may be rotated in accordance with the rotation.
In response to these operations, the head-mounted display device 100 performs not only enlargement / reduction of an image (including characters) displayed on the image display unit 20, but also rotation, copying, page turning, and the like. Also good.

  Furthermore, the operation device 3 can transmit the detection value of the 9-axis sensor 368 to the head-mounted display device 100. Further, the operation device 3 may include a GPS detection unit, and the current position detected by the GPS detection unit may be transmitted to the head-mounted display device 100. In this case, the control unit 140 can detect the relative position between the operation device 3 and the head-mounted display device 100 based on the detection value received from the operation device 3 and the detection value of the three-axis sensor 113.

  Then, the control unit 140 changes display such as enlargement, reduction, and rotation of the display image in response to the movement of the operation device 3 or the change in the relative position between the operation device 3 and the head-mounted display device 100. You may perform, and you may perform fade-in display and fade-out display of an image. In this case, the display can be changed by an operation of bringing the operation device 3 close to or away from the head-mounted display device 100.

Operation | movement of the display system 1 is not limited to said example, For example, the following operations can be performed.
By the operation of rotating the bezel 301 in the operation device 3, it is possible to realize an operation of sequentially switching and selecting a plurality of candidates. According to this operation, for example, when selecting a file or folder, or when selecting a virtual key on the virtual keyboard, the bezel 301 may be rotated. The bezel 301 can be operated intuitively and is located on the outer peripheral portion of the flat surface portion 300A of the operation device 3, so that it can be operated by groping. Therefore, the user can operate the bezel 301 while paying attention to the AR display displayed by the image display unit 20. Furthermore, since the selection can be confirmed by a button 305 which is provided at a position apart from the plane portion 300A and is larger than other switch type operation elements, the operation can be performed more easily. Since the position and size of the button 305 are different from those of the button 309 and the crown-type operation element 307, the operation by groping is easy like the bezel 301. Therefore, an intuitive and reliable operation can be performed while viewing the image displayed by the image display unit 20.

  Further, the control unit 140 may perform an operation of changing a screen displayed by the image display unit 20 in accordance with an operation of rotating the bezel 301. It is conceivable that the operating system 150 of the head-mounted display device 100 has a configuration in which a plurality of screens are switched and changed like iOS (registered trademark) or Android (registered trademark). In this case, the operation of the bezel 301 can be used more effectively by changing the screen according to the operation of rotating the bezel 301.

  In the operation of selecting one candidate from a plurality of candidates, instead of the operation of the bezel 301, it may be possible to select by voice. In this case, when a unique name, number, code, or the like assigned to the candidate is designated by voice, the operation device 3 detects this voice with the microphone 365 and transmits the detected voice data to the head-mounted display device 100. Then, the control unit 140 may perform the setting.

  It is also possible for the head-mounted display device 100 to operate the operation device 3 by transmitting control data to the operation device 3. For example, the device control unit 183 of the control unit 140 may transmit control data instructing to start detection of the touch panel 304 and the 9-axis sensor 368 or to shift to a non-detection state. In this case, execution / non-execution of sensing in the operation device 3 can be controlled by the head-mounted display device 100, so that an erroneous operation can be prevented and consumption of the power supply unit 360 can be prevented. In addition, the device control unit 183 may transmit control data instructing to start and stop imaging of the camera 311. In addition, the device control unit 183 may transmit control data for controlling lighting, extinguishing, and blinking of the LCD 303.

  The operation device 3 can also perform output based on data transmitted from the head-mounted display device 100. For example, the device control unit 183 may transmit text data and image data through the communication unit 117, and the operation device 3 may display text data and image data received from the head-mounted display device 100 on the LCD 303. In this case, the operation state (status) of the head-mounted display device 100 is displayed on the LCD 303 so that the user can easily grasp the operation state of the head-mounted display device 100. For example, it is useful when the user wants to know the operating state of the head-mounted display device 100 without wearing the head-mounted display device 100. The operation device 3 may output sound from the speaker 370 according to data received from the head-mounted display device 100, or may generate vibration by moving the vibrator 369.

  Further, the device control unit 183 transmits control data instructing execution of imaging and transmission of captured image data to the operation device 3, and the operation device 3 performs imaging and displays the captured image data on the image display unit 20. You can also. In this case, imaging can be performed from an angle that cannot be captured by the camera 61 included in the head-mounted display device 100, and the captured image can be displayed on the image display unit 20. Further, the control unit 140 may analyze the captured image data.

  In the display system 1, even if the NFC unit 362 of the operation device 3 reads data recorded on a non-contact type IC card or an RFID tag, the read data is transmitted from the wireless communication unit 361 to the communication unit 117. Good. In this case, the balance or the like of the IC card type electronic money can be displayed by the image display unit 20.

  Thus, in the display system 1, the see-through head-mounted display device 100 and the clock-type operation device 3 are combined, and the head-mounted display device 100 is operated by operating the operation device 3. As a result, hands-free operation is realized except during operation, and operation can be performed with natural movement even during operation. The operation device 3 can be used as a wristwatch by operating in the clock mode. By using the operation device 3 as a user interface for operation, hands-free operation is realized except during operation, and operation can be performed with natural movement even during operation. In addition, the display on the image display unit 20 allows the user to see information related to the operation device 3, and it is not necessary to look directly at the LCD 303. Therefore, there is an advantage that it is not necessary to use a hand to view the display on the LCD 303.

  Further, when the touch panel 304 supports a so-called multi-touch operation, the display image of the head-mounted display device 100 is enlarged or reduced in response to an operation for expanding or reducing the interval between a plurality of operation points on the touch panel 304. Alternatively, a rotating operation may be performed.

  In the display system 1, the head-mounted display device 100 can detect a relative position with the operation device 3. Although various methods can be used for this position detection, in the present embodiment, the head-mounted display device 100 images the range including the operation device 3 and specifies the relative position based on the captured image data. An example will be described.

FIG. 9 is a flowchart showing the operation of the head-mounted display device 100, and particularly shows position detection processing for detecting the relative position between the head-mounted display device 100 and the operation device 3.
The control unit 140 controls the camera 61 by the imaging processing unit 181 and executes imaging (step S51). The control unit 140 acquires captured image data of the camera 61, analyzes the acquired captured image data, and detects an image of the operation device 3 (step S52). Thereby, in the captured image of the camera 61, the position where the operation device 3 is reflected and the shape of the image of the operation device 3 can be detected.

  Subsequently, the control unit 140 calculates the relative position between the operation device 3 and the camera 61 based on the position and size of the image of the operation device 3 in the captured image (step S53). That is, the control unit 140 calculates the position of the operation device 3 with respect to the central axis of the angle of view of the camera 61, for example, the optical axis of the imaging lens of the camera 61, from the image position of the operation device 3. In addition, the control unit 140 obtains the distance between the operation device 3 and the imaging surface of the camera 61 from the size of the image of the operation device 3 in the captured image. Here, the control unit 140 may perform the calculation using the imaging conditions (zoom magnification, etc.) of the camera 61.

In step S53, after obtaining the relative position between the camera 61 and the operation device 3, the control unit 140 converts the obtained relative position into the relative position between the head-mounted display device 100 and the operation device 3. Good.
Since the head-mounted display device 100 and the operation device 3 are larger than the coordinates in the captured image, it is realistic to set a reference point. That is, the relative position between the head-mounted display device 100 and the operation device 3 is obtained as the relative position between the reference point set in the head-mounted display device 100 and the reference point set in the operation device 3. . When the reference point in the head-mounted display device 100 is a point that overlaps the optical axis on the imaging surface of the camera 61, the relative position obtained in step S53 remains as it is between the operation device 3 and the head-mounted display device 100. Relative position. When the reference point of the head-mounted display device 100 is at a position different from the imaging surface of the camera 61, the control unit 140 may perform the above-described conversion. The reference point in the operation device 3 may be the center of the bezel 301, for example.

  The control unit 140 identifies the orientation of the LCD 303 of the operation device 3 based on the shape of the image of the operation device 3 in the captured image (step S54). In step S <b> 54, the control unit 140 can calculate the orientation based on the shape of the bezel 301 of the operation device 3 in the captured image, the entire shape of the operation device 3 including the band unit 300, or the shape of the LCD 303. Also, for example, as shown in FIG. 2A, when the operation device 3 displays a graphic such as an arrow on the LCD 303 or when the time is displayed on the LCD 303, the display image of the LCD 303 is used. You may ask for direction.

  Thereafter, the control unit 140 includes data indicating the relative position between the operation device 3 calculated in step S53 and the head-mounted display device 100, and the operation device 3 corresponding to the head-mounted display device 100 specified in step S54. Data indicating the direction is generated. The control unit 140 stores the relative position data including the generated data in the storage unit 120 (step S55).

For example, in the operation shown in step S35 of FIG. 6, the AR display control unit 185 causes the operation device 3 to be displayed while the user of the head-mounted display device 100 is looking at the operation device 3 through the image display unit 20. AR content is displayed so that it appears to overlap. Here, the AR display control unit 185 determines the relative position between the operation device 3 and the head-mounted display device 100 detected by the operation shown in FIG. 9 and the direction of the operation device 3 with respect to the head-mounted display device 100. Based on the above, the display position of the AR content is determined. In this process, the AR display control unit 185 determines the display position of the AR content corresponding to the position of the object based on the positional relationship between the imaging range of the camera 61 and the display area of the image display unit 20. This positional relationship can be obtained from the relative position data stored in the storage unit 120.
The AR display control unit 185 may adjust the display size of the AR content according to the distance between the operation device 3 and the head-mounted display device 100. In addition, the AR display control unit 185 may adjust the display position of the AR content in accordance with the direction of the LCD 303 of the operation device 3 with respect to the head-mounted display device 100, or deform an image included in the AR content. May be.

  The method for detecting the relative position of the head-mounted display device 100 with the operation device 3 is not limited to the method shown in FIG. 9. For example, the operation device 3 and the head-mounted display device 100 perform data communication. And mutual detection results may be used. An example of this operation is shown in FIG.

FIG. 10 is a flowchart showing the operation of the display system 1, (A) shows the operation of the operation device 3, and (B) shows the operation of the head-mounted display device 100.
The operation in FIG. 10 is executed when the operation device 3 is in the UI mode, for example.
The control unit 350 of the operation device 3 detects the user's line of sight by performing imaging with the cameras 311 and 311 (step S61). The control unit 350 performs imaging and analysis of the captured image until a line of sight is detected (step S61; NO).

  When the line of sight is detected (step S61; YES), the control unit 350 performs imaging with the cameras 311 and 311 (step S62), and acquires captured image data (step S63). The control unit 350 transmits the captured image data by the wireless communication unit 361 (step S64). In step S62, imaging may be performed using only one of the cameras 311 and 311. Alternatively, both of the two cameras 311 and 311 may be captured, and both captured image data may be acquired and transmitted. Good.

  The control unit 140 receives captured image data from the operation device 3 through the communication unit 117 (step S71), analyzes the received captured image data, and detects an image of the head-mounted display device 100 (step S72). . Thereby, in the captured image, the position where the head-mounted display device 100 is reflected and the shape of the image of the head-mounted display device 100 can be detected.

  Subsequently, the control unit 140 calculates the relative position between the operation device 3 and the head-mounted display device 100 based on the position and size of the image of the head-mounted display device 100 in the captured image (step S73). For example, the control unit 140 calculates the position of the head-mounted display device 100 with respect to the center axis of the angle of view of the cameras 311 and 311 from the position of the image of the head-mounted display device 100. In addition, the control unit 140 obtains the distance between the head-mounted display device 100 and the operation device 3 from the size of the image of the head-mounted display device 100 in the captured image. Here, the control unit 140 may perform the calculation using the imaging conditions (such as zoom magnification) of the cameras 311 and 311.

In step S <b> 73, the control unit 140 may perform detection with higher accuracy using captured images of the plurality of cameras 311 and 311. For example, the parallax of the plurality of captured images may be calculated, and the distance between the operation device 3 and the head-mounted display device 100 may be obtained from the parallax.
Thereafter, the control unit 140 generates data indicating the relative position between the operation device 3 calculated in step S73 and the head-mounted display device 100 (step S74). The control unit 140 stores relative position data including the generated data in the storage unit 120.

  As described above, the operation device 3 and the head-mounted display device 100 may be linked to perform processing for obtaining a relative position between the operation device 3 and the head-mounted display device 100. In the operation of FIG. 10, the processing of steps S72 to S73 may be executed by the control unit 350, and the control unit 350 may generate data indicating the relative position and transmit the data to the head-mounted display device 100.

  As described with reference to FIGS. 9 and 10, the head-mounted display device 100 can detect the position of the operation device 3 with respect to the head-mounted display device 100. The head-mounted display device 100 can display AR content corresponding to the position of the operation device 3. Furthermore, by repeatedly executing the detection of the relative position, the movement of the head-mounted display device 100 can be obtained with the operation device 3 as a reference. Further, for example, the screen displayed on the image display unit 20 may be changed in accordance with the position and orientation of the operation device 3. In this case, changing the relative position between the operation device 3 and the head-mounted display device 100 corresponds to an operation on the head-mounted display device 100.

For example, in step S35 of FIG. 6, the AR display control unit 185 appears to overlap the operation device 3 while the user of the head-mounted display device 100 is looking at the operation device 3 through the image display unit 20. Thus, the AR content is displayed. Here, the AR display control unit 185 displays the AR content display position based on, for example, the relative position between the operation device 3 and the head-mounted display device 100 and the orientation of the operation device 3 with respect to the head-mounted display device 100. To decide. In this process, the AR display control unit 185 determines the display position of the AR content corresponding to the position of the object based on the positional relationship between the imaging range of the camera 61 and the display area of the image display unit 20. This positional relationship can be obtained from relative position data stored in the storage unit 120, for example.
The AR display control unit 185 may adjust the display size of the AR content according to the distance between the operation device 3 and the head-mounted display device 100. In addition, the AR display control unit 185 may adjust the display position of the AR content in accordance with the direction of the LCD 303 of the operation device 3 with respect to the head-mounted display device 100, or deform an image included in the AR content. May be.

  Further, the control unit 140 determines the detection axes (X, Y, Z axes in FIG. 1) of the 9-axis sensor 66 and the LCD 303 based on the relative position and orientation with the operation device 3 specified by the operation of FIG. Correlation with the direction (inclination) may be performed. Thereby, the display image on the image display unit 20 and the display image on the LCD 303 can be changed in conjunction with both the tilt of the head-mounted display device 100 and the tilt of the LCD 303.

  In addition, as an example in which the operation device 3 and the head-mounted display device 100 operate in conjunction with each other, for example, a marker is displayed on the LCD 303 under the control of the control unit 350, and this marker is displayed on the head-mounted display device. The aspect which 100 recognizes is mentioned. The control unit 140 detects a marker from the captured image of the camera 61. When the marker displayed on the LCD 303 is a marker that can include information such as a barcode or a two-dimensional code, the control unit 140 can acquire information from the reading result. The control unit 140 can perform an operation of establishing communication with the operation device 3 by the communication unit 117 based on the acquired information. Further, when there are a plurality of operation devices 3, it is possible to identify these operation devices 3. In this case, an operation for detecting the relative position between the specific operation device 3 and the head-mounted display device 100 and an operation for establishing communication with the specific operation device 3 are possible.

[Another configuration example of the operation device]
Here, the case where the operation objects 4 and 5 are used as another configuration example of the operation object will be described.
In the description of the display system 1 of the above embodiment, the case where the user uses the operation device 3 that can communicate with the head-mounted display device 100 is illustrated.
The head-mounted display device 100 is not limited to the operation device 3 and can execute processing corresponding to the operation of the operation object even when the user operates an operation object that does not have a communication function, for example.

FIG. 13 is an external view showing a configuration of the operation target object 4 as a configuration example of the operation target object.
Although the operation target 4 is a wristwatch type device like the operation device 3, a communication function like the operation device 3 is not indispensable. The operation target 4 is, for example, a commonly used watch.
The operation target 4 illustrated in FIG. 13 is an analog type wristwatch including a crown 401, a bezel 402, a dial 403, a long hand 404, a short hand 405, an index 406, and a date window 407 in a wristwatch body case. The crown 401 can be rotated at the normal position. By pulling the crown 401 rightward in the figure and rotating it, the long hand 404 and the short hand 405 are rotated, the date display of the date window 407 is sent, and the date and time are adjusted. Can be done.

  A wristwatch that includes a liquid crystal display panel at the position of the dial 403 and displays images of the long hands 404, the short hands 405, and the index 406 on the liquid crystal display panel may be used. The index 406 may be a number, a symbol, or any other shape that serves as a mark for the positions of the long hand 404 and the short hand 405. Further, the date window 407 is not essential, and the date window 407 may be omitted. In addition, the operation target 4 includes a button 408 that can be pushed. In the example of FIG. 13, four buttons 408 are provided, but the number and position of the buttons 408 can be arbitrarily changed. Also, the bezel 402 is not essential, but in the example of FIG. 13, the bezel 402 can be manually rotated to the left and right by the user's operation, as indicated by the arrows in the figure.

  The operation object 4 is shown as an example of a general wristwatch. For example, instead of the dial 403, a digital wristwatch including a liquid crystal display panel that displays numerals in seven segments is used as the head-mounted display device 100. It is good also as a structure which recognizes as an operation target object. The shapes of the dial 403 and the case are not limited to a circle, and may be a triangle or a rectangle.

FIG. 14 is a diagram illustrating another example of the operation target object. 14 includes a crown 501, a bezel 502, and a button 508 in the main body case of the wristwatch, similarly to the operation target 4 (FIG. 13). Further, the operation target 5 includes a display surface 503 at the same position as the dial 403 of the operation target 4. The display surface 503 is a display screen configured by a liquid crystal display panel, an organic EL panel, or electronic paper.
The crown 501 may be configured to be manually rotated or may be a push button. The bezel 502 may be configured to be manually rotated, or may be a fixed decoration. The button 508 may be a push button.

  The operation target 5 displays a marker 510 on the display surface 503. The marker 510 includes a plurality of marks 511. The position of the mark 511 is a predetermined position, and the display color, pattern, etc. of the plurality of marks 511 may be different. The operation target 5 normally displays the date, hour, minute, second, day of the week, etc. on the display surface 503, and displays the marker 510 according to the operation of the button 508 or the like. Moreover, it may replace with the display surface 503 of the operation target object 5, and may provide the dial 403 (FIG. 13), and the marker 510 is comprised with the adhesive seal | sticker (sticker), and this seal | sticker is used for the dial 403 or the display surface. You may stick to 503. In addition, when the operation target 5 includes an imaging unit (not shown) and the image display unit 20 is recognized by a captured image of the imaging unit (not shown), the marker 510 is placed on the display surface 503 by the control of the operation target 5. It may be displayed.

  Although the operation target 4 in FIG. 13 includes four buttons 408, the number and positions of the buttons 408 can be arbitrarily changed. The same applies to the button 508 provided in the operation target 5. The buttons 408 and 508 may be touch-sensitive operation units, the bezels 402 and 502 are not essential, and the shapes of the operation objects 4 and 5 such as belts are arbitrary.

FIG. 15 is a flowchart showing the operation of the head-mounted display device 100, and shows the operation when the user uses the operation objects 4 and 5.
The control unit 140 controls the camera 61 by the imaging processing unit 181 to execute imaging (step S101), and acquires captured image data. The control unit 140 causes the AR display control unit 185 to detect the image of the operation target 4 or the operation target 5 from the captured image (step S102). The process in which the AR display control unit 185 detects the images of the operation objects 4 and 5 can be realized by using, for example, data indicating the feature amount of the image of the operation objects 4 and 5 stored in the storage unit 120 in advance.

  The control unit 140 determines the operation target detected in step S102 as the operation target used for the operation (step S103). When a plurality of operation objects 4 and 5 are detected in the captured image of the camera 61, for example, an operation object to be used for the operation may be selected and determined by an operation on the control device 10 by the user.

  The control unit 140 causes the AR display control unit 185 to display an operation image (control target corresponding image) corresponding to the operation target determined in step S103 (step S104). The operation image is an image corresponding to the shape or the like of the operation target, and is prepared in advance for each operation target and stored in the storage unit 120.

FIG. 16 is a diagram illustrating an example of a state in which an operation image is displayed, and illustrates a state in which an operation image corresponding to the operation target 4 is displayed.
In the user's visual field VR, the operation object 4 in real space is seen through the image display unit 20. The AR display control unit 185 displays an operation image including the operation displays B11 to B19 so that the AR display control unit 185 can be seen at a position corresponding to the operation target 4. Each of the operation displays B11 to B19 corresponds to a portion (control object) of the operation target object 4. Each of the operation displays B11 to B19 may include an arrow indicating a corresponding portion as shown in FIG. In addition, each of the operation displays B11 to B19 guides the processing contents executed by the operation on the operation target object 4 by text and images.

  In the example of FIG. 16, the operation display B <b> 11 indicates that the display magnification is enlarged by operating the long hand 404. The operation display B12 indicates that the display magnification is reduced by operating the short hand 405. The positions of the long hand 404 and the short hand 405 change with time. For this reason, the AR display control unit 185 specifies the positions of the long hand 404 and the short hand 405 from the image captured by the camera 61, and changes the display positions of the operation displays B11 and B12 according to the positions of the long hand 404 and / or the short hand 405. You may adjust. The operation display B13 indicates that the head-mounted display device 100 displays a menu screen by an operation of rotating the crown 401, and the screen display is changed as described with reference to FIGS. 11A and 11B, for example. Show. The operation display B14 indicates that confirmation is instructed by operating the date window 407, and the operation display B15 indicates that a history is displayed by operating the index 406 at the 6 o'clock position. The operation display B16 indicates that the camera 61 performs imaging by operating the index 406 at the 12 o'clock position. The operation display B17 indicates that the operation on the operation target object 4 is locked by the operation on the 6 o'clock position of the bezel 402. The operation displays B18 and B19 indicate that the display is rotated right or left by an operation of rotating the bezel 402.

FIG. 17 is a diagram illustrating another example of the state in which the operation image is displayed, and illustrates a state in which the operation image corresponding to the operation target object 5 is displayed.
In the user's visual field VR, the operation object 5 in real space is seen through the image display unit 20. The AR display control unit 185 displays an operation image including the operation displays B31 to B39 so that the AR display control unit 185 can be seen at a position corresponding to the operation target 5. Each of the operation displays B31 to B39 corresponds to a mark 511 (control object) of the marker 510. Each of the operation displays B31 to B39 may include an arrow indicating a corresponding mark 511 as shown in FIG. Further, each of the operation displays B31 to B39 guides the processing content executed by the operation on the mark 511 by text or an image.

  In the example of FIG. 17, the operation display B31 indicates that the camera 61 performs imaging by operating the mark 511. The operation display B32 indicates that the operation on the operation target object 4 is locked by the operation of the mark 511. The operation display B33 indicates that confirmation is instructed by operating the mark 511. The operation displays B34 and B35 indicate that the display is rotated right or left by the operation of the mark 511. The operation display B36 indicates that the display magnification is reduced by operating the mark 511. The operation display B37 indicates that the display magnification is enlarged by operating the mark 511. The operation display B38 indicates that the home screen is displayed by operating the mark 511. The operation display B39 indicates that the head-mounted display device 100 displays a menu screen by operating the mark 511, and transitions the screen display as described with reference to FIGS. 11A and 11B, for example.

  The operation image illustrated in FIG. 17 can be referred to as an operation image corresponding to the operation object 5, but can also be referred to as an operation image corresponding to the marker 510. That is, the control unit 140 may be configured to display the operation image of FIG. 17 when the marker 510 is detected from the captured image of the camera 61. In this case, the configuration of the button 508 and the like included in the operation target 5 is not particularly limited. For example, a marker 510 is attached to the user's arm, and the marker 510 (or the arm to which the marker 510 is attached) is the operation target. It can be a thing. Therefore, any object that is separate from the image display unit 20 and has the marker 510 in a state where the image can be captured by the camera 61 can be used as the operation object.

In the state in which the operation image including the operation displays B11 to B19 in FIG. 16 is displayed, the operation on the operation target 4 is performed, for example, by superimposing the user's fingers on the operation target 4 portion. In this case, the control unit 140 uses the device control unit 183 to detect the image of the user's finger from the captured image of the camera 61 and identify the portion of the operation target 4 that is close to the position of the fingertip. Thereby, it is assumed that the specified part has been operated, and the control unit 140 executes processing shown in the operation displays B11 to B19.
In the state where the operation image including the operation displays B31 to B39 in FIG. 17 is displayed, the operation on the operation target 5 is performed by, for example, overlapping the user's finger on the mark 511. In this case, the control unit 140 uses the device control unit 183 to detect the image of the user's finger from the image captured by the camera 61, identifies the mark 511 close to the position of the fingertip, and positions the mark 511 in the marker 510. Is identified. Thereby, it is assumed that the specified mark 511 has been operated, and the control unit 140 executes the processing shown in the operation displays B31 to B39.

  After displaying the operation image illustrated in FIG. 16 or 17 in step S104 of FIG. 15, the control unit 140 starts detecting the operation (step S105). The control unit 140 determines whether or not an operation has been performed by the device control unit 183 (step S106), and when operated (step S106; YES), the process indicated by the operation image corresponding to the operated position or portion. Is executed (step S107). After executing this process, the control unit 140 proceeds to step S108. If no operation corresponding to the operation image is detected (step S106; NO), the control unit 140 proceeds to step S108.

  In step S108, the control unit 140 determines whether or not to end the operation using the operation image. Here, when the end of the operation is instructed by the user's operation or when it is detected that the operation target is not visually recognized in the user's visual field VR (step S108; YES), the control unit 140 This process is terminated. Moreover, when not ending operation (step S108; NO), the control part 140 returns to step S106.

In the present embodiment, when the head-mounted display device 100 recognizes the operation device 3 or the operation object 4 or 5 based on the captured image of the camera 61 or the like, the AR display control unit 185 stores the AR content from the storage unit 120. The data is read and displayed.
The AR display for operation setting shown in FIG. 7A and the operation images shown in FIGS. 16 and 17 are examples of AR content.

  FIG. 18 illustrates a configuration example of data related to the AR content among the data stored in the storage unit 120. The storage unit 120 stores not only the AR content data 123 and the target device data 127 shown in FIG. 18 but also other data, but the illustration is omitted. The storage unit 120 that stores the AR content data 123 and the target device data 127 corresponds to a data storage unit.

Data related to the AR content includes AR content data 123 and target device data 127. The AR content data 123 includes AR element display data 123a, position attribute data 123b, and correspondence processing data 123c.
The AR content data 123 is data for displaying the AR content, and corresponds to data for displaying the AR display for operation setting in FIG. 7A and the operation images in FIGS. 16 and 17. The AR element display data 123a is data for displaying an element of an AR content for operation (an image for operation). The element of the AR content is a character or an image displayed in association with the operation device or the operation unit of the operation target, and may be accompanied by an additional image such as an arrow. In the example of FIG. 7A, allocation candidate displays A1, A2, A3, and A4 and setting candidates A11, A12, A13, and A14 correspond to elements. In the example of FIG. 7B, assignment candidate displays A21, A22, A23, A24 and setting candidates A31, A32, A33, A34 correspond to elements. In the example of FIG. 16, the operation displays B11 to B19 correspond to elements, and in the example of FIG. 17, the operation displays B31 to B39 correspond to elements. The AR element display data 123a includes, for example, image data for displaying the element, text data, data for specifying display attributes (color, size, font, background, etc.) of the image and text, and the like.

  The position attribute data 123b is data for designating the display position of the element. The element display position is set for each element included in the AR element display data 123a. The position attribute data 123b may be data that designates the display position of an element at a fixed position in the right display drive unit 22 and the left display drive unit 24. Further, the position attribute data 123b may be data that specifies the display position of an element as a relative position with respect to the operation device 3 or the operation objects 4 and 5. Further, the position attribute data 123b may be data that designates a display position of an element as a relative position with respect to the operation unit 3 or the operation objects 4 and 5. For example, the position attribute data 123b can be data that designates the display positions of the setting candidates A11 to A14 and A31 to A34 in FIGS. 7A and 7B as relative positions to the bezel 301 of the operation device 3. . For example, the display positions of the operation displays B11 to B19 in FIG. 16 are the relative positions of the operation target 4 with respect to the crown 401, bezel 402, dial 403, long hand 404, short hand 405, index 406, date window 407, button 408, and the like. May be specified as. For example, the display position of the operation displays B31 to B39 in FIG. 17 may be data specifying the relative position of the operation target 5 with respect to the marker 510, the mark 511, and the like.

The corresponding process data 123c is data that defines a process to be executed by the control unit 140 when an element included in the AR element display data 123a is selected or designated by a user operation. In the example of FIG. 7A, the control unit 140 performs a process of sequentially switching a plurality of candidates according to the operation of the operation unit (for example, the bezel 301) of the operation device 3, and the screen displayed by the image display unit 20 is changed. The operation etc. to perform are performed. In addition, even if the device control unit 183 performs a process of transmitting control data instructing to start detection of the touch panel 304 and the 9-axis sensor 368 or to shift to a non-detection state according to the operation of the operation device 3. Good.
In the example of FIG. 16, the control unit 140 enlarges the display magnification according to the operation on the long hand 404 and reduces the display magnification according to the operation on the short hand 405. Further, a menu screen is displayed according to an operation of rotating the crown 401, and the screen display is changed. Further, a process for accepting a confirmation instruction is performed according to the operation of the date window 407, and a history is displayed according to the operation for the index 406 at the 6 o'clock position. Further, an image is taken by the camera 61 by an operation on the index 406 at the 12 o'clock position, and the operation on the operation target 4 is locked according to the operation on the 6 o'clock position of the bezel 402. Further, the display is rotated clockwise or counterclockwise according to the operation of rotating the bezel 402.
In the example of FIG. 17, in accordance with the operation of the mark 511, the control unit 140 captures the camera 61, locks the operation on the operation target 4, accepts a confirmation instruction, rotates the display, enlarges / reduces the display magnification, Processing such as home screen display, menu screen display, and screen display transition is performed.
The corresponding process data 123c includes data designating these processes.

The target device data 127 is data indicating an operation device or an operation target corresponding to the AR content data 123. For example, image feature amount data for detecting the operation device or the operation target from the captured image of the camera 61, and the like. including.
The target device data 127 is referred to when the control unit 140 detects the operation device 3 in step S52 (FIG. 9), for example, and detects the operation objects 4 and 5 in step S102 (FIG. 15). The control unit 140 can detect or recognize the operation device or the operation target based on the target device data 127, identify the corresponding AR content, and display the AR content based on the AR content data 123.

Here, the storage unit 120 can store a plurality of AR content data 123. Further, data regarding a plurality of AR contents may be included in one AR content data 123. The target device data 127 is data for designating a corresponding operation device or operation target for each AR content, and a specific mode of data association is arbitrary. For example, the same number of target device data 127 as the AR content data 123 may be stored in the storage unit 120. Further, the target device data 127 may include data for designating a corresponding operation device or operation target for each of the plurality of AR content data 123. Further, the target device data 127 may include data indicating an operation device or an operation target corresponding to a plurality of AR contents included in the AR content data 123.
Further, the target device data 127 may be included in the AR content data 123. In this case, the AR content data 123 is configured to include AR element display data 123a, position attribute data 123b, correspondence processing data 123c, and target device data 127 in association with each AR content.

The type, position, number, and the like of elements in the AR content can be changed according to the user's operation, and the AR content data 123 can be edited by reflecting this change.
FIG. 19 is a flowchart showing a process for editing the AR content data 123.
The control unit 140 starts editing the AR content data 123 in accordance with a user operation (step S121). In the following description, the user's operation may be either an operation received by the operation unit 111 or a gesture operation by an indicator such as the user's hand detected by the camera 61.

  The control unit 140 detects an operation device or an operation target, for example, similarly to the processing executed in steps S51 to S52 (step S122). The control unit 140 displays the AR content corresponding to the detected operation device or operation target based on the AR content data 123 (step S123). Here, when an unknown operation device or operation target is targeted, the AR content data 123 corresponding to the operation device or operation target is not stored in the storage unit 120. In this case, the control unit 140 may detect that it is an unknown operation device or operation target, or may be instructed to be an unknown operation device or operation target by a user operation. The control unit 140 displays the AR content using the general-purpose AR content data 123 stored in the storage unit 120 in advance, or the AR content data 123 specified by the user's operation or prior settings.

When an instruction to change (move), add, or delete the position of the AR content is input by the user's operation for the element of the AR content displayed in step S123, the control unit 140 changes the position of the AR content according to the input. Addition and deletion are performed (step S124).
Further, when a process to be executed by the control unit 140 corresponding to the element of the AR content is designated by the user's operation for the element of the AR content being displayed, the control unit 140 executes the control unit 140 according to the input. The process to be performed is designated or changed (step S125).
Note that the execution order of the processes of steps S124 and S125 can be changed, and the processes of steps S124 and S125 may be executed in parallel.

  The control unit 140 determines whether or not the editing of the AR content data 123 has been completed (step S126). If the editing has not ended (step S126; NO), the control unit 140 returns to step S124. When the editing is completed (step S126; YES), the control unit 140 generates target device data 127 that sets the operation device or the operation target detected in step S122 as the target device (step S127). The control unit 140 associates the AR content data 123 reflecting the edited contents of steps S124 and S125 with the target device data 127 generated in step S127, and stores it in the storage unit 120 (step S128). Exit.

  As described above, the display mode of the AR content displayed corresponding to the operation device 3, the operation target object 4, 5, etc., the type, position, number, and element of the AR content are executed by the user's operation. You can specify and change the process. For this reason, it is possible to display AR content that suits the user's application and preferences, and to further improve convenience.

  The head-mounted display device 100 may download the AR content data 123 and the target device data 127 by communicating with an external device through the communication unit 117. In this case, the head-mounted display device 100 communicates with an external device that relays communication between the head-mounted display device 100 and another communication network, such as a smartphone or a router, via the communication network. You may download from a server apparatus (not shown). The head-mounted display device 100 may download the AR content data 123 and the target device data 127 from an external device (for example, a personal computer or a smartphone) connected by the communication unit 117.

FIG. 20 is a flowchart illustrating a process in which the head-mounted display device 100 downloads AR content data.
The control unit 140 starts downloading in response to a user operation or access from an external device (step S141). The control unit 140 downloads and acquires the AR content data 123 (step S142). The control unit 140 stores the AR element display data 123a, the position attribute data 123b, and the corresponding processing data 123c of the acquired AR content data 123 in the storage unit 120 (Step S143). If necessary, the control unit 140 may perform setting so that the stored data can be used in step S143.

  The control unit 140 determines whether or not the downloaded device data includes target device data 127 or data that specifies a corresponding operation device or operation target (step S144). If the corresponding data is included (step S144; YES), the control unit 140 stores the target device data 127 in association with the AR content data 123 in the storage unit 120 (step S145), and ends this process. To do.

  When the target device data 127 or the data specifying the corresponding operation device or operation target is not included (step S144; NO), the control unit 140 starts processing for generating the target device data 127. . That is, the control unit 140 executes imaging with the camera 61 to acquire captured image data (step S146), and generates target device data 127 based on the captured image data (step S147). Thereby, the operation device or the operation target in the imaging range of the camera 61 is set as the operation device or the operation target corresponding to the downloaded AR content data 123.

  The control unit 140 determines whether or not to adjust the correspondence between the AR content data 123 and the target device data 127 (Step S148). This adjustment is necessary for associating the AR content data 123 with the operation device or operation target imaged in step S146. Specifically, in step S147, the control unit 140 extracts an image of the operation device or the operation target from the captured image data, detects the operation unit, calculates the positional relationship of the operation unit, and stores the target device data 127. Generate. Which element of the AR content data 123 is associated with each detected operation unit is affected by the detection state of the operation unit. Therefore, for example, the control unit 140 determines whether or not to adjust the association between the operation unit and the AR content element based on a user instruction. Here, a message requesting an instruction from the user may be displayed.

  When the adjustment is performed (step S148; YES), the control unit 140 performs a process of editing the AR content data (step S149). In step S149, for example, the processing described in FIG. 19 (steps S121 to S128) can be executed. Thereafter, the control unit 140 associates the edited AR content data 123 with the target device data 127 generated in step S147 and edited in step S149, and stores it in the storage unit 120 (step S150). Exit. If no adjustment is made (step S148; NO), the control unit 140 does not execute step S149, proceeds to step S150, and associates the target device data 127 generated in step S147 with the downloaded AR content data 123. Is stored in the storage unit 120.

  In the operation device 3, the control unit 350 displays an image on the LCD 303 based on data stored in advance. The operation target 5 displays a marker 510 by a function of a control unit (not shown) based on data stored in advance. The display data is not limited to fixed data, and may be configured to be downloaded and added from the head-mounted display device 100 or other external devices, for example.

For example, the operation target 5 may be configured such that the marker 510 can be edited or added. As described with reference to FIGS. 14 and 17, the operation target 5 displays the marker 510 on the display surface 503 configured by an LCD or an electronic paper display. The marker 510 can be changed as long as the head-mounted display device 100 can recognize the marker 510.
The head-mounted display device 100 detects a mark 511 on the marker 510 displayed by the operation target 5 as an operation unit, and displays operation displays B31 to B39 as AR contents at positions corresponding to the mark 511. The operation displays B31 to B39 are associated with the color of the mark 511, for example. The control unit 140 identifies the mark 511 based on the color of the mark 511, and displays the operation displays B31 to B39 at positions corresponding to the color marks 511 associated in advance. In this case, the operation displays B31 to B39 are displayed regardless of the position of the mark 511 in the marker 510.

In this example, the position attribute data 123b includes data specifying the color of the mark 511. In detail, data specifying that the image of the AR element display data 123a is displayed at a position corresponding to the mark 511 of a specific color is included.
Accordingly, the plurality of markers 510 having the same color of the mark 511 and different positions of the mark 511 can cause the head-mounted display device 100 to execute the same function. In this case, the user can selectively use a plurality of markers 510 whose positions of the marks 511 are different and have the same color without changing the process of causing the head-mounted display device 100 to execute.

In this case, for example, the marker 510 may be switched and displayed by operating the button 508 provided in the operation target 5. In addition, the operation target 5 performs short-range wireless communication with the communication unit 117 of the head-mounted display device 100 and switches the marker 510 based on control data transmitted by the head-mounted display device 100. Also good.
This is merely an example, and the head-mounted display device 100 may be configured to identify the mark 511 based on the pattern (texture) of the mark 511, or may be configured to identify the mark 511 based on the shape of the mark 511. In this case, as long as the pattern and shape of the mark 511 do not change, the markers 510 having different positions of the mark 511 can be used properly without affecting the function of the head-mounted display device 100.
In this case, the display on the display surface 503 can be switched so as to change the clothes according to the preference of the user who uses the operation target 5. When acquiring the data of the marker 510, the operation target 5 may be connected to the head-mounted display device 100 or an external device by wireless communication or a communication cable.

  Furthermore, it is good also as a structure which changes the marker 510 in the range which affects the function which the head mounted display apparatus 100 performs. In this case, since the function executed by the head-mounted display device 100 is changed, it is preferable to change the AR content data 123 and the target device data 127 also in the head-mounted display device 100. For example, the head-mounted display device 100 downloads the AR content data 123 and target device data 127 from an external device (not shown), and operates the marker 510 corresponding to the AR content data 123 and target device data 127. The structure which the target object 5 acquires from the head mounted display apparatus 100 is mentioned. This example will be described with reference to FIG.

  FIG. 21 is a flowchart illustrating an operation in which the head-mounted display device 100 and the operation target 5 download the AR content data 123 and related data. 21A shows the operation of the head-mounted display device 100, and FIG. 21B shows the operation of the operation target 5. The operation target 5 that performs the operation shown in FIG. 21 has a configuration including a communication unit (not shown) that communicates with the head-mounted display device 100. For example, a wireless communication unit (not shown) that performs wireless communication with the communication unit 117 or a communication interface (not shown) including a connector that is wired to the communication unit 117 and the interface 125 is provided.

  The control unit 140 starts downloading in response to a user operation or access from an external device (step S161). The control unit 140 downloads and acquires the AR content data 123 (step S162). The control unit 140 stores the AR element display data 123a, the position attribute data 123b, and the corresponding processing data 123c of the acquired AR content data 123 in the storage unit 120 (Step S163). If necessary, the control unit 140 may perform setting so that the stored data can be used in step S163.

  The control unit 140 stores target device data 127 associated with the downloaded device data in association with the target device data 127 in the storage unit 120 in association with the AR content data 123 (step S144). The target device data 127 is acquired from an external device together with the AR content data 123 in step S162, for example.

  The control unit 140 is connected to the operation target 5 and shifts to a state where data communication is possible (step S165). The operation target 5 is also connected to the head-mounted display device 100 and shifts to a state where data communication is possible (step S171).

  The control unit 140 authenticates that the device (operation object 5) connected in step S165 is a device corresponding to the target device data 127 stored in step S164 (step S166). Here, in response to a request from the head-mounted display device 100, the operation target 5 transmits authentication data such as a product name and a product model number, and performs authentication with the head-mounted display device 100. (Step S172).

  After the authentication is successful, the control unit 140 transmits data for displaying the marker 510 to the operation target 5 (step S167). For example, this data is acquired from an external device together with the target device data 127 in step S162. The operation target 5 receives and stores the data for displaying the marker 510 transmitted from the head-mounted display device 100 (step S173).

  By the processing of FIG. 21, the operation target 5 acquires and stores data for displaying the new marker 510, and can display the new marker 510 based on this data. The head-mounted display device 100 displays AR content corresponding to the marker 510 displayed by the operation target 5. Since this AR content includes operation displays like the operation displays B31 to B39 illustrated in FIG. 17, by imaging the operation target 5 with the camera 61 while the operation target 5 displays the marker 510, The AR content can be displayed at a position corresponding to the marker 510. The head-mounted display device 100 can detect the mark 511 included in the new marker 510 and display the AR content including the operation display corresponding to the mark 511. Further, when an operation on the mark 511 included in the new marker 510 is detected, a process specified by the operation display displayed corresponding to the mark 511 can be executed.

  In this case, the types of markers 510 that can be displayed on the operation target 5 can be increased. The new marker 510 has an advantage that the marker 510 of various designs can be displayed because there is no restriction such as the common use of the marker 510 and the mark 511 that can be displayed before. Furthermore, in the head-mounted display device 100, the type of the mark 511 that can detect an operation can be changed or added. Furthermore, a function to be executed in response to an operation on the mark 511 can be added. For example, the communication protocol executed between the operation target 5 and the head-mounted display device 100 is changed, a new design operation display that matches the appearance of the operation target 5 is displayed, or a new function is executed. It becomes possible to do. Accordingly, it becomes possible to add the operation target 5 to be used for the head-mounted display device 100, and the function that the head-mounted display device 100 executes in accordance with the change of the operation target 5. The usage can be changed.

  Here, the AR content data 123 and the target device data 127 are not limited to data for setting the functions of the head-mounted display device 100. For example, in addition to the AR content data 123 and the target device data 127, a program corresponding to a function executed by the control unit 140 based on the AR content data 123 may be included. Also, the AR content data 123 and the target device data 127 may be a single application program including a program. In this case, the function itself executed by the control unit 140 can be added or changed. For example, it is assumed that the operating system 150 is an OS for a mobile device (smart phone or tablet computer) such as an Android (registered trademark) OS, an iOS (registered trademark), or a Windows (registered trademark). In this case, by using an application program including the AR content data 123 and the target device data 127, the AR content data 123 and the target device data 127 can be easily downloaded and set as a unit. When the user purchases a new operation object 5 or uses a new marker 510, the head-mounted display device 100 corresponding to the new operation object 5 or the marker 510 is easily set. It becomes possible.

  As described above, the display system 1 according to the embodiment to which the present invention is applied includes the head-mounted display device 100 and the operation device 3 configured separately from the image display unit 20. The head-mounted display device 100 includes a transmissive image display unit 20 that is mounted on the user's head and allows the user to visually recognize the outside scene. The operation device 3 includes a bezel 301, a touch panel 304, a button 305, a crown type operator 307, and a button 309 as an operation unit that receives a user's operation. The operation unit may include a 9-axis sensor 368, or the band unit 300 having the 9-axis sensor 368 may be used as the operation unit. The head-mounted display device 100 includes a control unit 140 that controls the display of the image display unit 20. Based on an operation on the operation device 3, the control unit 140 causes the image display unit 20 to perform an AR display related to an outside scene that passes through the image display unit 20. According to this configuration, when the head-mounted display device 100 performs AR display related to the outside scene, the operation of the operation device 3 separate from the head-mounted display device 100 is reflected, which is intuitive. The AR display can be controlled by an input operation.

For example, in the operation device 3, after a function is assigned to the operation unit of the band unit 300 according to the setting illustrated in FIG. 7A, an operation on the operation device 3 such as touching the operation unit or moving a finger. May be detected by the head-mounted display device 100. Specifically, the control unit 140 may detect an operation on the operation device 3 based on the captured image data of the camera 61.
In addition, the operation device 3 may include a wireless communication unit 361 that transmits data indicating the operation received by the operation unit to the head-mounted display device 100. In this case, the control unit 140 causes the image display unit 20 to perform AR display related to the outside scene that is transmitted through the image display unit 20 in a manner associated with the operation unit based on the data transmitted from the operation device 3.

  Further, the control unit 140 detects the operation of the operation unit based on the data transmitted from the operation device 3, and changes the AR display being displayed on the image display unit 20 in accordance with the detected operation. For this reason, the AR display can be changed by operating the operation device 3 while performing the AR display.

The control unit 140 may be capable of setting the allocation of AR display to the operation unit.
In this case, the control unit 140 may be able to execute processing for updating the allocation of the AR display to the operation unit.

In addition, the operation unit includes a touch panel 304 that detects a contact operation. The touch panel 304 has a plurality of operation areas, and the control unit 140 causes the image display unit 20 to display an AR display assigned to the operation area operated on the touch panel 304. For this reason, by performing a touch operation with the operation device 3, various operations for the AR display can be performed.
The control unit 350 included in the operation device 3 may detect the movement of the operation device 3 as an operation.
The head-mounted display device 100 can be operated by moving the operation device 3.

  Further, the operation device 3 may include the camera 311, and data including a captured image of the camera 311 may be transmitted to the head-mounted display device 100 by the wireless communication unit 361. In this case, the captured image captured by the operation device 3 separate from the head-mounted display device 100 can be displayed by the head-mounted display device 100. In the display system 1, the head-mounted display device 100 can be authenticated by the operation device 3.

11 and 12 are diagrams illustrating examples of functions executed by the head-mounted display device 100 in response to the operation of the operation device 3.
FIG. 11A shows an example in which a selection screen A41 for selecting files and folders is displayed in the visual field VR of the user. When an operation for rotating the bezel 301 is performed in a state where a plurality of selection candidates (folders in this example) are displayed side by side as described above, the candidates are sequentially selected.
FIG. 11B shows a virtual keyboard A43 displayed by operating the operation device 3 or the control device 10. A plurality of virtual keys corresponding to characters and symbols are arranged on the virtual keyboard A43. When an operation for rotating the bezel 301 is performed while the virtual keyboard A43 is displayed, virtual keys are sequentially selected on the virtual keyboard A43.

  As shown in FIGS. 11A and 11B, in the operation of selecting one candidate from a plurality of candidates displayed in the visual field VR of the user, an operation of rotating the bezel 301 is preferable. In this case, in accordance with the rotation direction of the bezel 301, the direction in which candidates are switched in order may be changed. In addition, the operation for confirming the selection can be performed by, for example, a button 305 provided in the operation device 3. The bezel 301 can be operated intuitively and is located on the outer peripheral portion of the flat surface portion 300A of the operation device 3, so that it can be operated by groping. Therefore, the user can operate the bezel 301 while paying attention to the AR display displayed by the image display unit 20. Furthermore, since the selection can be confirmed by a button 305 which is provided at a position apart from the plane portion 300A and is larger than other switch type operation elements, the operation can be performed more easily. Since the position and size of the button 305 are different from those of the button 309 and the crown-type operation element 307, the operation by groping is easy like the bezel 301. Therefore, an intuitive and reliable operation can be performed while viewing the image displayed by the image display unit 20.

  In addition, as shown to FIG. 11 (A) and (B), it is good also as a structure which can be selected with an audio | voice by operation which selects one candidate from the some candidate displayed on a user's visual field VR. In this case, when a unique name, number, code, or the like assigned to the candidate is designated by voice, the operation device 3 detects this voice with the microphone 365 and transmits the detected voice data to the head-mounted display device 100. Then, the control unit 140 may perform the setting.

FIG. 12 schematically shows the configuration of the transitioned screen for an example in which the screen is transitioned according to the operation of the bezel 301.
When the operating system 150 of the head-mounted display device 100 is configured to switch and change a plurality of screens like iOS (registered trademark) and Android (registered trademark), the operation of the bezel 301 is more effectively performed. Can be used. In the example of FIG. 12, four screens of screens 1 to 4 can be sequentially switched and displayed by rotating the bezel 301. Further, the feeding direction when the screens 1 to 4 are sequentially selected may be opposite to the forward direction corresponding to the operation direction of the bezel 301.

In the head mounted display device 100, after the screen 2 is displayed among the screens 1 to 4, the display screen is determined by operating the button 305, and then the screens 2, 2-1, The four screens 2-2 and 2-3 can be changed. Also in this case, the feeding direction when sequentially selecting the four screens may be opposite to the forward direction corresponding to the operation direction of the bezel 301.
In the example illustrated in FIG. 12, the display screen can be quickly changed by rotating the bezel 301 without performing the confirmation operation of the button 305.

Operation | movement of the display system 1 is not limited to said example, For example, the following operations can be performed.
The head-mounted display device 100 may perform enlargement, reduction, rotation, copy, page feed, and the like of an image (including characters) displayed on the image display unit 20 in response to the operation of the operation device 3. In this case, the operation of the operation device 3 is not only the rotation of the bezel 301, the pressing operation of the button 309, the pressing operation of the button 305, and the operation of rotating the crown-shaped operation element 307, but for example, touching the touch panel 304 with a finger. The operation to perform may be included. For example, the display image may be enlarged in response to an operation of turning the bezel 301 to the right, and the display image may be reduced in response to an operation of turning the bezel 301 to the left. Further, the display image is enlarged when a finger contact with the center of the touch panel 304 is detected, and the display image is reduced when a finger contact with the peripheral edge of the touch panel 304 (for example, near the lower left frame) is detected. Also good. Further, when the finger touching the touch panel 304 rotates in a clockwise direction or in a counterclockwise direction while being in contact, the display image may be rotated in accordance with the rotation.

  Further, when the touch panel 304 supports a so-called multi-touch operation, the display image of the head-mounted display device 100 is enlarged or reduced in response to an operation for expanding or reducing the interval between a plurality of operation points on the touch panel 304. Alternatively, a rotating operation may be performed.

  The operation device 3 can also transmit the detection value of the 9-axis sensor 368 to the head-mounted display device 100. Further, the operation device 3 may include a GPS detection unit, and the current position detected by the GPS detection unit may be transmitted to the head-mounted display device 100. In this case, the control unit 140 of the head-mounted display device 100 determines the relative relationship between the operation device 3 and the head-mounted display device 100 based on the detection value received from the operation device 3 and the detection value of the three-axis sensor 113. The position can be detected. Then, in response to the detected change in relative position, display changes such as enlargement, reduction, and rotation of the display image may be performed, and fade-in display and fade-out display of the image may be performed. In this case, the display can be changed by an operation of bringing the operation device 3 close to or away from the head-mounted display device 100. Furthermore, the head-mounted display device 100 can detect an operation of moving the operation device 3 by the operation device 3 transmitting the detection value data of the 9-axis sensor 368 to the head-mounted display device 100.

  As described above, the head-mounted display device 100 includes the head-mounted display device 100 having a display unit that is mounted on the user's head, and the operation device 3 that is configured separately from the display unit. Prepare. The operation device 3 includes a bezel 301, a touch panel 304, a button 305, a crown type operator 307, and a button 309 as an operation unit that receives a user's operation. The operation unit may include a 9-axis sensor 368, or the band unit 300 having the 9-axis sensor 368 may be used as the operation unit. The head-mounted display device 100 includes a control unit 140 that controls the display of the image display unit 20, and the control unit 140 performs a process of changing the screen displayed on the image display unit 20 based on the operation of the operation device 3. Including display control. For this reason, since the operation device 3 separate from the head-mounted display device 100 is operated and the screen is changed, the head-mounted display device 100 can be controlled by a more intuitive input operation.

  In addition, the operation device 3 includes a wireless communication unit 361 that transmits data indicating the operation received by the operation unit to the head-mounted display device 100. The control unit 140 of the head-mounted display device 100 detects the operation of the operation device 3 based on the data transmitted from the operation device 3. For this reason, the operation on the operation device 3 can be reliably detected by the head-mounted display device 100. Further, it is possible to reduce the load on the head-mounted display device 100 related to the detection of the operation.

In addition, the operation device 3 includes a band unit 300 as a mounting unit that is mounted on the user's body, and performs a more intuitive input operation using the operation device 3 that is mounted on the user's body. The head-mounted display device 100 can be controlled.
Moreover, the operation part which the operation device 3 has detects operation with rotation by the bezel 301 and the rotation detection part 367, for example. The control unit 140 of the head-mounted display device 100 displays a screen to be displayed on the image display unit 20 corresponding to the rotation direction of the operation when data indicating an operation involving rotation is transmitted from the operation device 3. For example, the transition is made as shown in FIG. For this reason, a screen can be changed by operation with rotation, and more intuitive operation can be realized. Further, since the direction of operation corresponds to the direction of transition of the screen, for example, a groping operation can be performed.

The control unit 140 of the head-mounted display device 100 performs display control including any of enlargement, reduction, and rotation of the image displayed on the image display unit 20 based on the data transmitted from the operation device 3. Also good.
The control unit 350 included in the operation device 3 may detect the movement of the operation device 3 as an operation.
The image display unit 20 may be a transmissive display unit that allows the user to visually recognize the outside scene. In this case, the user can control the display of the head-mounted display device 100 by operating the operation device 3 while looking at the outside scene. Since it can be operated intuitively without gazing only at the head-mounted display device 100, it is suitable when the head-mounted display device 100 is used while looking at the outside scene.

  The head-mounted display device 100 transmits output data to the operation device 3, and the operation device 3 receives the output data transmitted from the head-mounted display device 100 by the communication unit, and receives the data. Output data can be output.

  When the control unit 140 detects the operation objects 4 and 5 from the captured image of the camera 61, the control unit 140 associates the operation images associated with the operation objects 4 and 5 with the operation objects 4 and 5. Are displayed on the image display unit 20. Thereby, the information regarding the operation objects 4 and 5 that are separate from the head-mounted display device 100 can be provided to the user in an easily understandable manner.

  In addition, when detecting an operation on the operation objects 4 and 5 on which the operation image is displayed, the control unit 140 executes processing corresponding to the operation content and the operation image. As a result, the head-mounted display device 100 executes processing in response to an operation on the operation objects 4 and 5, so that an intuitive operation can be realized using the operation objects 4 and 5.

  Further, since the control unit 140 functions as a detection unit that detects an operation on the operation objects 4 and 5, it is possible to quickly detect an operation on the operation objects 4 and 5. Even if the operation objects 4 and 5 do not have a communication function, the head-mounted display device 100 can execute processing corresponding to the operation of the operation objects 4 and 5. The control unit 140 detects the operation objects 4 and 5 based on the captured image of the camera 61 and detects an operation on the operation objects 4 and 5. Thereby, even if the operation objects 4 and 5 do not perform communication, the head-mounted display device 100 can detect the operation objects 4 and 5 and can detect an operation on the operation objects 4 and 5.

  The head-mounted display device 100 includes a storage unit 120 that stores AR content data 123 for AR display of an operation image corresponding to the operation unit included in the operation target. The control unit 140 can change the AR display data stored in the storage unit 120. Thereby, since the data can be changed for the function of performing the AR display corresponding to the operation target, the data can be changed to correspond to a new operation target or an unknown operation target. Therefore, it is possible to reduce the restrictions on the operation target to be displayed on the AR, and to improve the convenience.

  The AR content data 123 stored in the storage unit 120 includes position attribute data 123b that associates an element that is an operation image with an operation unit included in the operation target. The control unit 140 can change the correspondence between the operation image in the AR content data 123 and the operation unit included in the operation target.

  In addition, the present invention can be executed by the control unit 140 that controls the head-mounted display device 100 that is mounted on the user's head and has the transmissive image display unit 20 that allows the user to see through the outside scene. Can be realized as a simple program. When the CPU of the control unit 140 executes this program, the operation device 3 and the operation objects 4 and 5 that are configured separately from the image display unit 20 can be detected. In addition, the control unit 140 detects an operation unit included in the operation device 3 and the operation objects 4 and 5 and displays an operation image as an AR at a position corresponding to the detected operation unit. Further, the control unit 140 detects an operation related to the operation image, and executes processing corresponding to the operation image from which the operation is detected. As a result, the head-mounted display device 100 displays an operation image corresponding to the operation device 3 and the operation objects 4 and 5 separate from the head-mounted display device 100, and the operation image is displayed. Process is executed according to the operation related to. For this reason, it becomes possible to cause the head-mounted display device 100 to execute processing using the operation device 3 and the operation objects 4 and 5, and an intuitive operation on the head-mounted display device 100 becomes possible. .

In addition, this invention is not restricted to the structure of the said embodiment, In the range which does not deviate from the summary, it can be implemented in a various aspect.
In the present embodiment, the example in which the operation device 3 used as the user interface of the head-mounted display device 100 has the shape of a wristwatch has been described. The present invention is not limited to this, and any shape that is suitable for being worn on the body of a user who uses the head-mounted display device 100 may be used. For example, it can be a ring type, brooch type, pendant or necklace type, or pen type device. Moreover, it is good also as a headphone type | mold, earphone type | mold, or headset type device with which a user's head and ear | edge are mounted | worn. In this case, the function as the operation device of the present invention may be mounted on headphones or earphones that output sound. The function as the operation device of the present invention may be implemented in a headset including a speaker that outputs audio and a microphone that collects audio uttered by the user. In this configuration, the headphones, earphones, and speakers may be bone conduction speakers, and the microphone may be a bone conduction microphone. In this case, the device preferably includes at least an operation element that detects a contact operation or a pressing operation, and includes a wireless communication unit 361, an LED 313, a camera 311, and a 9-axis sensor 368. In these configurations, the same effects as those in the above-described embodiment can be obtained except for operations related to display.

Moreover, although the head mounted display apparatus 100 of the said embodiment was set as the structure fixed to a user's body by the band part 300, the structure of the operation device of this invention is not limited to this. That is, it is not essential that the operation device is fixed by the band unit 300 or the like, as long as it can be attached to the user's body. Therefore, it may be a pendant or necklace shape as described above, or a shape imitating other accessories, may be a hat or helmet shape, or may be other clothing.
Furthermore, the operating device of the present invention is not limited to a mode in which it is directly attached to or fixed to the user's body. For example, an apparatus corresponding to an operation device may be fixed to a wristwatch, an accessory, clothes, a bag, a hat, or a helmet that is attached or fixed to the user's body. The operation device in this case can be configured, for example, by removing the band unit 300 from the operation device 3.

  The display system 1 may be configured to include a plurality of operation devices. For example, the operation device 3 may include a different type of operation device such as a wristwatch-type operation device and a pen-type operation device. In this case, the display screen of the head-mounted display device 100 may be changed based on operations on a plurality of operation devices. Examples of the method in which the head-mounted display device 100 detects operations on a plurality of operation devices include a method of transmitting data indicating operations from the plurality of operation devices.

Further, the operation device 3 transmits data indicating the detection value of the nine-axis sensor 368 that is an inertial sensor to the head-mounted display device 100, so that even if the display system 1 operates using the movement of the operation device 3 as an operation. Good.
For example, using the detection value of the 9-axis sensor 368 and the detection value of the 9-axis sensor 66 included in the head-mounted display device 100, the control unit 140 transmits an external scene that can be viewed through the image display unit 20. You may perform matching with the display area of the image display part 20. FIG. In this case, association can be performed with higher accuracy by using detection values of sensors provided in a plurality of apparatuses. This association can be used when the AR content is displayed on the image display unit 20.

  The operation device 3 may include a sensor that measures the heart rate of the user. Specifically, a sensor or the like that detects light by irradiating light on the user's arm can be used. In this case, the operation device 3 may transmit data relating to the measured heart rate to the head-mounted display device 100, and display data relating to the heart rate on the head-mounted display device 100. For example, display of the result of statistical processing of heart rate, real-time display of heart rate, heart rate and detection value of 9-axis sensor 368, and display data related to user activity and physical condition management Also good. Further, the user's excitement level and fatigue level may be estimated based on the heart rate, and the display color in the image display unit 20 and the content of the image to be displayed may be changed to bring comfort to the user.

  In the above-described embodiment, the configuration in which the operation device 3 is an electronic device that is operated by the power supply unit 360 has been described as an example. However, the present invention is not limited to this. For example, a marker for specifying the position of the bezel 301 in the rotational direction may be provided at a position adjacent to the bezel 301 on the operation device 3 and / or the planar portion 300A. In this case, the device control unit 183 can determine the rotation direction and the rotation amount of the bezel 301 based on the captured image data captured by the camera 61. Even when the operation device 3 includes the power supply unit 360, the head-mounted display device 100 may obtain the rotation direction and the rotation amount of the bezel 301 by the above operation. This operation can be realized, for example, as a power saving operation mode operation that suppresses the consumption of the power supply unit 360. When the remaining capacity of the power supply unit 360 is small, the display system 1 can be shifted to the power saving operation mode under the control of the operation device 3 or the head-mounted display device 100.

  In addition, the operation objects 4 and 5 are configured separately from the image display unit 20 included in the head-mounted display device 100, and are crowns 401 and bezels 402 that are objects for displaying an operation image. It is sufficient that each part has a marker 510 or the like. For this reason, for example, stationery including writing instruments, tools, furniture, computer peripheral devices, various electronic devices such as telephones, and the like may be anything as long as they are separate from the image display unit 20, and worn by the user. The original function is not limited.

  Further, instead of the image display unit 20 of the above-described embodiment, for example, an image display unit of another method such as an image display unit worn like a hat may be adopted, corresponding to the left eye of the user. What is necessary is just to provide the display part which displays an image, and the display part which displays an image corresponding to a user's right eye. The display device of the present invention may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet. In this case, the part for positioning the position with respect to the user's body and the part positioned with respect to the part can be used as the mounting portion.

Furthermore, in the above-described embodiment, the configuration in which the image display unit 20 and the control device 10 are separated and connected via the connection unit 40 has been described as an example. The present invention is not limited to this, and the control device 10 and the image display unit 20 may be configured integrally and mounted on the user's head.
As the control device 10, a notebook computer, a tablet computer, or a desktop computer may be used. The control device 10 may be a game machine, a portable phone, a portable electronic device including a smartphone or a portable media player, other dedicated devices, or the like. Further, the control device 10 may be configured separately from the image display unit 20, and various signals may be transmitted and received between the control device 10 and the image display unit 20 by wireless communication.

  For example, as a configuration for generating image light in the image display unit 20, a configuration including an organic EL (Organic Electro-Luminescence) display and an organic EL control unit may be used. Further, as a configuration for generating image light, LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like can be used.

The optical system that guides the image light to the user's eyes may include an optical member that transmits external light that is incident on the apparatus from the outside and that is incident on the user's eyes together with the image light. Moreover, you may use the optical member which is located ahead of a user's eyes and overlaps a part or all of a user's visual field. Further, a scanning optical system that scans a laser beam or the like to obtain image light may be employed. Further, the optical member is not limited to guiding the image light inside the optical member, and may have only a function of guiding the image light by refracting and / or reflecting it toward the user's eyes.
For example, the present invention can also be applied to a laser retinal projection type head mounted display. That is, the light emitting unit includes a laser light source and an optical system that guides the laser light source to the user's eyes, and the laser light is incident on the user's eyes to scan the retina and form an image on the retina. A configuration that allows the user to visually recognize an image may be employed.
Further, the present invention can be applied to a display device that employs a scanning optical system using a MEMS mirror and uses MEMS display technology. That is, a light emitting unit includes a signal light forming unit, a scanning optical system having a MEMS mirror that scans light emitted from the signal light forming unit, and an optical member on which a virtual image is formed by light scanned by the scanning optical system. You may prepare. In this configuration, the light emitted from the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided through the optical member, and reaches the virtual image forming surface. When the MEMS mirror scans the light, a virtual image is formed on the virtual image forming surface, and the user recognizes the virtual image with the eyes, thereby recognizing the image. The optical component in this case may be one that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above embodiment, and may use a half mirror surface. .

In addition, at least a part of the functional blocks illustrated in FIGS. 3 and 4 may be realized by hardware, or may be configured by cooperation of hardware and software. Further, the program executed by the control unit 140 may be stored in the storage unit 120 or the storage device in the control device 10, or the program stored in the external device is acquired via the communication unit 117 or the interface 125. It is good also as a structure to execute.
Moreover, only the operation part 111 among the structures formed in the control apparatus 10 may be formed as a single user interface (UI). Further, the configuration formed in the control device 10 may be formed in the image display unit 20 in an overlapping manner. For example, a control unit corresponding to the control unit 140 may be formed in both the control device 10 and the image display unit 20. Moreover, it is good also as a structure by which the function which the control part 140 formed in the control apparatus 10 and the CPU formed in the image display part 20 perform separately is divided | segmented.

  In addition, the present invention can be realized in various forms other than the head-mounted display device 100. For example, a control method of the head-mounted display device 100, an information system having the head-mounted display device 100, a control method of the head-mounted display device 100, a computer program for realizing the information system, and the computer program Can be realized in the form of a recording medium on which the computer program is recorded, a server device connected to a communication line such as the Internet for distributing the computer program, a data signal including the computer program and embodied in a carrier wave.

  DESCRIPTION OF SYMBOLS 1 ... Display system, 2 ... Frame, 3 ... Operation device, 4, 5 ... Operation object, 10 ... Control apparatus, 20 ... Image display part (display part), 21 ... Right holding part, 22 ... Right display drive part, DESCRIPTION OF SYMBOLS 23 ... Left holding part, 24 ... Left display drive part, 26 ... Right optical image display part, 28 ... Left optical image display part, 61 ... Camera, 66 ... 9 axis | shaft sensor, 100 ... Head-mounted display apparatus (display apparatus) 117 ... Communication unit, 120 ... Storage unit, 140 ... Control unit, 150 ... Operating system, 160 ... Image processing unit, 170 ... Display control unit, 181 ... Imaging processing unit, 183 ... Device control unit, 185 ... AR display Control unit, 187 ... voice processing unit, 300 ... band unit, 300A ... flat surface part, 301 ... bezel, 303 ... LCD, 304 ... touch panel (touch operation unit), 305 ... button, 307 ... crown type operator, 30 ... button, 310 ... optical part, 311 ... camera (imaging part), 313 ... LED, 331-334, 341-345 ... function display, 350 ... control part, 351 ... operation detection part, 352 ... output control part, 353 ... Imaging control unit, 354 ... communication control unit, 355 ... audio processing unit, 360 ... power supply unit, 361 ... wireless communication unit, 362 ... NFC unit, 365 ... microphone, 366 ... button operation unit, 367 ... rotation detection unit, 368 ... 9-axis sensor, 369 ... vibrator, 370 ... speaker.

Claims (32)

A display device that is mounted on the user's head and has a transmissive display unit that allows the user to see through the outside scene, and an operation object configured separately from the display unit;
The display device includes a control unit that performs AR display related to the outside scene that is transmitted through the display unit, by the display unit, corresponding to the operation target,
A display system characterized by The display device displays a control target corresponding image associated with the operation target by the display unit in association with the operation target;
The display system according to claim 1. The operation object includes an operation unit that receives an operation of the user;
A communication unit that transmits data indicating an operation received by the operation unit to the display device, and an operation device.
The control unit of the display device performs AR display by the display unit based on an operation of the operation device;
The display system according to claim 1, wherein: The control unit of the display device performs an AR display related to the outside scene that is transmitted through the display unit by the display unit in a mode associated with the operation unit based on an operation of the operation device.
The display system according to claim 3. The control unit detects an operation of the operation unit based on data transmitted from the operation device, and changes the AR display being displayed on the display unit in response to the detected operation;
The display system according to claim 3 or 4, characterized in that: The control unit can set the allocation of the AR display to the operation unit;
The display system according to claim 3, wherein: The control unit is capable of executing a process of updating the allocation of the AR display to the operation unit;
The display system according to claim 6. The control unit of the display device obtains a relative position between the operation device and the display device, and performs AR display on the display unit based on the obtained relative position;
The display system according to claim 3, wherein: The operation unit includes a touch operation unit that detects a contact operation,
The touch operation unit has a plurality of operation areas,
The control unit displays the AR display assigned to the operation area operated by the touch operation unit by the display unit.
The display system according to any one of claims 3 to 8, wherein: The operation device includes an imaging unit, and transmits data including a captured image of the imaging unit to the display device by the communication unit;
The display system according to claim 3, wherein: A display device that is mounted on the user's head and has a transmissive display unit that allows the user to see through the outside scene, and an operation object configured separately from the display unit;
The display device includes a control unit that performs display control including a process of transitioning a screen displayed by the display unit based on an operation on the operation target;
A display system characterized by The operation object is an operation device including an operation unit that receives an operation of the user;
The display system according to claim 11. The operation device includes a communication unit that transmits data indicating the operation received by the operation unit to the display device.
The control unit of the display device detects an operation of the operation device based on data transmitted from the operation device;
The display system according to claim 12. The operation unit of the operation device detects an operation involving rotation,
The control unit of the display device transitions a screen displayed by the display unit in response to a rotation direction of the operation when data indicating an operation involving rotation is transmitted from the operation device.
The display system according to claim 12, wherein: The control unit of the display device performs display control including any of enlargement, reduction, or rotation of an image displayed by the display unit based on data transmitted from the operation device;
The display system according to any one of claims 12 to 14, wherein: The display device transmits output data to the operation device,
The operation device receives the output data transmitted by the display device at the communication unit, and outputs the received output data.
The display system according to claim 12, wherein: The operation device includes an imaging unit, and transmits data including a captured image of the imaging unit to the display device by the communication unit;
The display system according to any one of claims 12 to 16, wherein: The control unit of the display device receives data including a captured image from the operation device, and obtains a relative position between the operation device and the display device based on the received data;
The display system according to claim 17. The operation unit of the operation device detects movement of the operation device as an operation;
The display system according to claim 11, wherein: The control unit of the display device obtains a relative position between the operation device and the display device, and controls display of the display unit based on the obtained relative position;
The display system according to claim 11, wherein: The display device includes an imaging unit, and the control unit obtains a relative position between the operation device and the display device based on a captured image of the imaging unit.
The display system according to claim 20. The display unit is a transmissive display unit that allows the user to see through the outside scene;
The display system according to claim 11, wherein: The operating device has a mounting portion to be mounted on the user's body;
The display system according to claim 1, wherein: When the operation target is detected, the control unit included in the display device displays a control target corresponding image associated with the operation target on the display unit corresponding to the operation target. ,
The display system according to claim 1, wherein: The control unit included in the display device performs processing corresponding to an operation content and the control target corresponding image when detecting an operation on the operation target displaying the control target corresponding image.
25. A display system according to claim 24. The display device includes a detection unit that detects an operation on the operation object;
26. A display system according to claim 24 or 25. The display device includes an imaging unit,
The control unit detects the operation target based on a captured image of the imaging unit, and detects an operation on the operation target;
26. A display system according to claim 24 or 25. The display device includes a data storage unit that stores data for AR display for AR display of an operation image corresponding to the operation unit included in the operation target,
The control unit is capable of changing the AR display data stored in the data storage unit,
The display system according to any one of claims 1 to 27. The AR display data stored in the data storage unit includes data associating the operation image with the operation unit included in the operation object,
The control unit can change a correspondence between the operation image and the operation unit included in the operation object in the AR display data stored in the data storage unit,
29. A display system according to claim 28. In a control method for controlling a display device having a display unit mounted on a user's head using an operation object configured as a separate body from the display unit,
The display unit performs AR display related to an outside scene that is transmitted through the display unit, corresponding to the operation object, by the display unit.
A control method of a display device characterized by the above. In a control method for controlling a display device having a display unit mounted on a user's head using an operation object configured as a separate body from the display unit,
Performing display control including a process of transitioning a screen displayed by the display unit based on an operation on the operation target by the display device;
A control method of a display device characterized by the above. A program that can be executed by a control unit that controls a display device that is mounted on a user's head and has a transmissive display unit that allows the user to see through the outside scene.
By the control unit,
An operation object configured as a separate body from the display unit is detected,
Detecting an operation unit included in the operation object,
An operation image is displayed at a position corresponding to the operation portion of the detected operation object,
Detecting an operation related to the operation image;
Executing a process corresponding to the operation image in which the operation is detected;
A program characterized by

Images