KR20190061825A - Tethering type head mounted display and method for controlling the same - Google Patents

Abstract

An HMD and a method of controlling the HMD are disclosed. According to an embodiment of the present invention, there is provided an HMD (Head Mounted Display) connected to a mobile terminal, including a communication unit for communicating with the mobile terminal, a display unit for displaying a virtual space image corresponding to the content stored in the mobile terminal, And a controller connected to the sensing unit and the communication unit for transmitting the first motion value detected by the sensing unit to the connected mobile terminal while the HMD is operating in the moving mode. Also, the controller receives and displays a screen corresponding to the head position of the user calculated based on the first motion value sensed by the HMD and the second motion value sensed by the mobile terminal, from the mobile terminal.

Description

TECHNICAL FIELD [0001] The present invention relates to an HMD (HMD)

The present invention relates to a tethering type HMD connected to a mobile terminal and a control method of the HMD.

The head-mounted display (HMD) is a display device which is worn on the head of a user and can present an image directly in front of the user's eyes. The HMD allows a user to enjoy image contents with a larger image than a TV or a screen . Alternatively, a virtual space screen may be displayed so that the user can enjoy the virtual space experience.

On the other hand, the functions of mobile terminals are becoming diversified these days due to the development of technology. For example, there are data and voice communication, photographing and video shooting through a camera, voice recording, music file playback through a speaker system, and outputting an image or video on a display unit. Some terminals are equipped with an electronic game play function or a multimedia player function. In particular, modern mobile terminals can receive multicast signals that provide visual content such as broadcast and video or television programs. Such a terminal has various functions according to the development of technology. For example, it is implemented in the form of a multimedia device having a combination of functions such as photographing and photographing of a moving picture, reproduction of a music or video file, reception of a game, and broadcasting.

Accordingly, a method of using the excellent mobile terminal function in cooperation with the HMD is emerging. As a part of such a scheme, an HMD (Tethering Type HMD) in which the HMD and the mobile terminal are connected to each other so that the mobile terminal can share the workload of the HMD can be processed.

The tethering type HMD connected to the mobile terminal can reduce the workload of the HMD by interlocking the connected mobile terminal with the HMD. As a result, the tethered HMD does not require high performance compared to the stand-type HMD that the HMD handles for all tasks, which can be produced at a lower cost. Also, various functions using the connected mobile terminal can be provided.

Tethering Type The HMD detects the user's head movement and provides the changed image according to the head movement. At this time, when a virtual reality experience is performed using a tethering type HMD while moving through a moving means, when the head movement is detected, movement of the moving means is detected at the same time so that an unintended image is provided .

It is therefore an object of the present invention to solve the above problems and provide an image display apparatus and a method for realizing the virtual reality experience through the HMD while moving through the moving means, A possible HMD and a control method of the HMD are provided.

According to an aspect of the present invention, there is provided an HMD (Head Mounted Display) connected to a mobile terminal, the HMD comprising: a communication unit for communicating with the mobile terminal; A display unit for displaying a virtual space image corresponding to the content stored in the mobile terminal; A sensing unit for sensing movement of the HMD; And a controller coupled to the communication unit and transmitting the first motion value detected by the sensing unit to the connected mobile terminal while the HMD is operating in the moving mode. The controller receives and displays a screen corresponding to the head position of the user calculated based on the first motion value and the second motion value sensed by the mobile terminal from the mobile terminal.

In one embodiment, the user's head position is tracked while excluding the second motion value from the first motion value during the motion mode.

In one embodiment, the first and second motion values are obtained using an acceleration sensor, a gyro sensor, and a magnetic sensor.

In one embodiment, the second motion value may be a first correction value obtained based on a second value obtained from the acceleration sensor and the gyro sensor of the mobile terminal using the first value obtained from the magnetic sensor of the mobile terminal .

In one embodiment, the head position of the user is estimated from a first motion value obtained from the acceleration sensor and the gyro sensor of the HMD, and a second correction value obtained based on the first correction value.

In one embodiment, the control unit controls the communication unit to limit transmission of the first motion value when the motion mode is released.

In one embodiment, the control unit performs the movement mode in response to reception of a set signal from the mobile terminal through the communication unit.

In one embodiment, the setting signal is generated through menu setting of the mobile terminal and is received from the mobile terminal when it is detected that the position of the mobile terminal is fixed.

In one embodiment, when the movement mode is started, first initial axis values corresponding to an initial position of the HMD and second initial axis values corresponding to an initial position of the mobile terminal are stored.

In one embodiment, when the occurrence of a predetermined event is detected, the controller determines whether to change the initial positions by comparing the degree of change of each of the current axis values from the stored initial axis values, and, based on the determination, And transmits a radio signal for re-tracking the mobile terminal to the mobile terminal.

In one embodiment, the event is detected when the degree of motion of at least one of the mobile terminal and the HMD is out of the reference range, the detachment of the HMD is detected, or there is a reconnection of the mobile terminal and the HMD .

In one embodiment, during the moving mode, whether a change in the initial positions corresponding to the stored initial axis values is re-searched for every predetermined time interval.

In one embodiment, the movement mode is started when movement of a moving means aboard a user wearing the HMD after connection between the HMD and the mobile terminal is sensed.

In one embodiment, if the movement of the moving means is not detected for a predetermined time, the moving mode is released, and when the movement of the moving means is sensed, the movement mode is restarted.

In one embodiment, the control unit controls the current screen direction to be maintained based on the sensor correction value based on the second motion value, when a change in the moving direction of the moving unit is detected and there is no head movement of the user .

In one embodiment, when the change in the moving direction of the moving means is detected and the head movement of the user is sensed, the controller subtracts the sensor correction value based on the second motion value from the first motion value, And the direction is changed.

A method of controlling an HMD (Head Mounted Display) according to an embodiment of the present invention is a method of controlling an HMD (Head Mounted Display) connected to a mobile terminal, the method comprising: displaying a virtual space image corresponding to a content stored in the mobile terminal step; Transmitting a first motion value of the HMD to the connected mobile terminal while the HMD is operating in a moving mode; And receiving and displaying a screen corresponding to the head position of the user calculated based on the first motion value and the second motion value sensed by the mobile terminal from the mobile terminal.

As described above, according to the HMD and the control method of the HMD according to the embodiment of the present invention, when the virtual space image is viewed using the HMD in the moving means such as the moving vehicle, the malfunction in which the head tracking is recognized in the moving direction of the moving means The problem is solved. In addition, it provides user convenience by eliminating the sensor value which is not related to the virtual space image by the HMD without the additional input or recognition of the user.

FIG. 1 and FIG. 2 are conceptual diagrams showing a state in which an HMD and a mobile terminal are connected to each other according to the present invention.
3A, 3B and 3C are illustrative drawings for explaining the configuration of the HMD related to the present invention.
FIGS. 4A, 4B, and 4C are explanatory diagrams illustrating a configuration of a mobile terminal connected to an HMD associated with the present invention.
FIGS. 5A, 5B, and 5C are views for explaining an example in which an image of a virtual space displayed on a display unit is controlled according to a movement of a user wearing the HMD related to the present invention.
Figs. 6 and 7 are diagrams for explaining a method of tracking the head position of the HMD in the moving means in the HMD related to the present invention. Fig.
8A, 8B and 8C are diagrams for explaining the expected screen direction displayed on the HMD according to the change of the moving direction of the moving means in the HMD related to the present invention.
9 is a representative flowchart for explaining a control method of the HMD related to the present invention.
10A and 10B are views for explaining a method of setting a movement mode of the HMD in a mobile terminal connected to the HMD related to the present invention.
11 and 12 are flowcharts for explaining another control method of the HMD related to the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like or similar elements are denoted by the same or similar reference numerals, and redundant description thereof will be omitted. The suffix " module " and " part " for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role. In the following description of the embodiments of the present invention, a detailed description of related arts will be omitted when it is determined that the gist of the embodiments disclosed herein may be blurred. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. , ≪ / RTI > equivalents, and alternatives.

The mobile terminal described in this specification includes a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a slate PC A tablet PC, an ultrabook, a wearable device (e.g., a smart watch, a glass glass, a head mounted display (HMD), etc.) .

FIG. 1 and FIG. 2 illustrate an example in which a tethering HMD 200 related to the present invention is connected to a mobile terminal 100.

As shown in FIGS. 1 and 2, the HMD 200 according to the embodiment of the present invention can be connected to the mobile terminal 200. Here, the mobile terminal 200 may be various devices. For example, the mobile terminal 200 may be a smart phone, a tablet PC, or the like. The HMD 200 may receive information or sensed signals input through the mobile terminal 200 or may share various information and data stored in the mobile terminal 200.

An image of a virtual space displayed on the display unit of the HMD 200 may be generated in the HMD 200 or may be generated in the mobile terminal 200 connected to the HMD 200. [ For example, when the HMD 200 generates an image of the virtual space, the HMD 200 performs image processing and rendering processing for processing the image of the virtual space, and the image processing and rendering processing result And the generated image information can be output through the display unit. Meanwhile, when the mobile terminal 200 generates an image of the virtual space, the mobile terminal 200 performs the image processing and rendering processing, and transmits the image information generated as a result of the image processing and rendering processing to the HMD 200). Then, the HMD 200 can output image information received from the mobile terminal 200.

Meanwhile, the HMD 200 according to the embodiment of the present invention may receive a control signal for controlling the function of the HMD 200 from the mobile terminal 200. For example, the HMD 200 may receive a result of sensing movement of the mobile terminal 200 from the mobile terminal 200, and may control the function of the HMD 200 based on the detection result. Or this control signal may be that the HMD 200 itself is sensed. That is, the HMD 200 senses the head movement of the wearer wearing the HMD 200 using the sensors provided in the HMD 200, and controls the function of the HMD 200 based on the detection result It is possible.

The control of the function of the HMD 200 according to the movement of the mobile terminal 200 or the movement of the HMD 200 is controlled according to the movement of the mobile terminal 200 or the movement of the HMD 200 It may mean that image information is displayed. In other words, the HMD 200 according to the embodiment of the present invention may be configured to display, in a virtual space displayed on the display unit of the HMD 200, a direction corresponding to the motion of the mobile terminal 200 or the motion of the HMD 200 The image of the virtual space of the display unit 251 may be displayed on the display unit 251. The HMD 200 can simulate the movement of the user in the virtual space or display the virtual space image in the other direction according to the movement of the head of the user.

Meanwhile, the control signal input from the mobile terminal 200 may include not only the result of sensing the movement of the mobile terminal 200 according to the sensed values of the gyro sensor or the acceleration sensor provided in the mobile terminal 200, Or may be a touch input input through a touch sensor provided in the mobile terminal 200. [ For example, the control signal may include at least one touch or touch and drag input applied on the display unit 251 of the mobile terminal 200, or a touch on one point on the display unit 251 for a predetermined time or longer And may be a hold input that continuously applies an input.

Meanwhile, the mobile terminal 200 may share various information with the HMD 200. Accordingly, a variety of information related to the mobile terminal 200 can be displayed on the display unit of the HMD 200. The user can also view the content through the HMD 200, The detected events can be confirmed.

In addition, various information related to the controller device 200 may be provided to the HMD 200 according to a function that the mobile terminal 200 can provide. Therefore, when the mobile terminal 200 is connected to the HMD 200 and functions as a controller device as shown in FIGS. 1 and 2, functions available through the mobile terminal 200, that is, e-mail a message function such as an e-mail function, a call function, a social network service (SMS) function, a short messaging service (SMS), or a multimedia messaging service (MMS), and various applications installed in the mobile terminal 200 Information can be displayed through the HMD 200. [0050]

Accordingly, the user can notify the mobile terminal 200 of the event, that is, the reception of a call or the reception of a message, or the news of the SNS community, or various status information related to the mobile terminal 200 through the HMD 200 Can be confirmed through the HMD 200.

Next, FIGS. 3A and 3B are block diagrams for explaining the HMD 200 related to the present invention.

3A, the HMD 200 according to the embodiment of the present invention includes a wireless communication unit 210, a sensing unit 240, an output unit 250, an interface unit 260, a memory 270, a controller 280 And a power supply unit 290 and the like. 3A are not essential for implementing the HMD 200 according to the embodiment of the present invention, so that the HMD 200 described in this specification is more or less than the above listed components Components.

More specifically, among the above components, the wireless communication unit 210 may be connected between the HMD 200 and various peripheral devices, for example, between the mobile terminal 200 and the HMD 200, Lt; RTI ID = 0.0 > wireless < / RTI > It may also include one or more modules that connect the HMD 200 to one or more networks.

Meanwhile, the sensing unit 240 may include at least one sensor for sensing a head movement of a user wearing the HMD 200. For example, the sensing unit 240 may include an acceleration sensor 241, a magnetic sensor 242, and a gyro sensor 243. Here, the acceleration sensor 241 and the gyro sensor 243 can sense the acceleration and the angular velocity according to the movement of the head of the user. Also, the magnetic sensor 243 can sense x, y, z axis values corresponding to the user's head or the current position of the HMD 200.

The sensing unit 240 may further include an eye tracking sensor for tracking the user's eyes and detecting where the user's eyes are staying. For example, the eye tracking sensor can detect the direction of the user's gaze toward a specific region on the display unit 251 by detecting an area on the display unit 251 corresponding to the position of the user's eyes.

The output unit 250 may include at least one of a display unit 251, an audio output unit 252, and a haptrip module 253 for generating an output related to visual, auditory or tactile sense. The display unit 251 may be installed at a position corresponding to both sides of the user when the user wears the HMD 200 so as to provide a larger size image to the user. The sound output unit 252 may be formed in the form of a headphone that can be brought into close contact with both ears of the user when the user wears the HMD 200 so that the sound signal related to the content to be reproduced can be transmitted . In addition, the haptic module 153 may generate a vibration related to the currently playing content to the user, if necessary, so that the user can view the content more realistically.

Meanwhile, the interface unit 260 serves as a channel with various kinds of external devices connected to the HMD 200. The interface unit 260 may include at least one of various ports such as a wired / wireless headset port, an external charger port, and a wired / wireless data port. For example, when the HMD 200 and the mobile terminal 200 are connected by wire, the interface unit 260 may be a channel through which various data and information are exchanged between the HMD 200 and the mobile terminal 200 Can be performed.

In addition, the memory 270 stores data supporting various functions of the HMD 200. The memory 270 may store a plurality of application programs (application programs or applications) driven by the HMD 200, data for operation of the HMD 200, and commands. At least some of these applications may be downloaded from an external server via wireless communication. At least a part of these application programs may exist on the HMD 200 from the time of shipment for the basic functions of the HMD 200 (for example, the reproduction of the contents and the output of the video signal and the sound signal of the reproduced contents) . The application program may be stored in the memory 270 and may be installed on the HMD 200 and may be driven by the control unit 280 to perform the operation (or function) of the HMD 200.

The control unit 280 of the HMD 200 typically controls the overall operation of the HMD 200 in addition to the operations associated with the application program. The control unit 280 may process or process signals, data, information or the like inputted or outputted through the above-mentioned components or may drive an application program stored in the memory 270 to provide or process appropriate information or functions to the user.

In addition, the controller 280 may control at least some of the components illustrated in FIG. 3A to drive an application program stored in the memory 270. FIG. In addition, the control unit 280 may operate at least two or more of the components included in the HMD 200 in combination with each other for driving the application program.

Under the control of the controller 280, the power supply unit 290 receives power from the external power source and supplies power to the respective components included in the HMD 200. The power supply unit 290 includes a battery, which may be an internal battery or a replaceable battery.

In addition, 'user' used in the embodiments described below may mean a person using an HMD or a device using an HMD (for example, a robot having artificial intelligence). In addition, in the embodiments described below, a technique related to control of a virtual space image rendered from a mobile terminal will be described.

3B, the HMD 200 is configured to be worn on the head (or head, face, head) of a user (human body) and includes a frame portion (case, housing, cover, etc.) can do. The frame part may be formed of a flexible material to facilitate wearing. In this figure, the frame portion includes a first frame 201 and a second frame 202 of different materials.

The first frame 201 serves to provide a space in which at least one of the components described in FIG. 3A can be disposed, and the second frame 202 serves to provide a space in which the first frame 201 (Or fixed) to be mountable on the head of the user (human body).

The frame portion may be referred to as a body (or an HMD body) or a body (or an HMD body). Here, the HMD body (or the HMD body) may be understood as a concept of referring to the HMD 200 as at least one aggregate.

If the frame portion including the first frame 201 and the second frame 202 is regarded as one HMD body, the body of the HMD related to the present invention can be formed in various forms. Specifically, the main body may include a plurality of surfaces having predetermined angles. The plurality of surfaces refer to the surfaces located outside the body of the HMD 200. In this regard, the plurality of surfaces may refer to the surface (outer surface, outer surface, etc.) of the HMD 200. Each of the plurality of surfaces may be flat or curved.

The main body (frame portion) is supported on the head portion, and a space for mounting various components is provided. As shown in the figure, electronic components such as a camera 221, a display unit 251, a user input unit 223, a control unit, a sensing unit 240, and an interface unit 260 may be mounted on the first frame 201 have.

The second frame 202 may be provided with an electronic component such as an acoustic output unit 252 or the like. However, the present invention is not limited to this, and the first frame 201 and the second frame 202 may have other components required for the HMD to be variously arranged according to the user's selection. That is, the HMD 200 described herein may have more or fewer components than those listed above.

The control unit 280 of the HMD controls various electronic components included in the HMD 200.

The camera 221 may be provided on the display unit 251 of the main body. For example, the camera 221 may be disposed on one side (e.g., the front side) of the HMD 200 main body. The camera 221 is disposed adjacent to at least one of the left eye and the right eye, and is configured to capture (receive, input) a forward image. Since the camera 221 is disposed adjacent to the eye and facing forward, the camera 221 can acquire a scene viewed by the user as an image. In the figure, one camera 221 is provided, but the present invention is not limited thereto. The plurality of cameras 221 may be provided to acquire stereoscopic images.

The display unit 251 is mounted on the frame unit and outputs screen information (e.g., image, image, moving picture, etc.) in front of the user's eyes. When the user wears the HMD 200, the display unit 251 may be arranged to correspond to at least one of the left and right eyes so that screen information can be displayed before the user's eyes. That is, the display unit 251 may be formed to cover at least one of the left and right eyes of the user (or to face at least one of the left and right eyes of the user).

As an example, the display portion 251 of the electronic device associated with the present invention may be located inside the electronic device body. Specifically, the display unit 251 is disposed inside the electronic device, and may be disposed at a position facing the user's eyes when the user wears the electronic device on the user's head.

Referring to FIG. 3C, the HMD 200 may include a lens unit 203 so that the user can view the screen information output from the display unit 251. That is, the HMD 200 related to the present invention is configured such that all the screen information (or light) output from the display unit 251 can be transmitted through the lens unit 203 to the user's eyeball (or field of view) .

For example, the lens unit 203 may be arranged to correspond to at least one of the eyes of the user (i.e., the left eye and the right eye). In addition, the lens section 203 can be arranged to lie between the user ' s eye and the display section 251 when the user wears the electronic device on the head.

Since the viewing angle may vary depending on the distance between the user's eyeball and the display unit, the lens unit 203 may be formed to be variable in position by user control. The lens portion 203 may be formed through a concave lens or a convex lens, or a combination thereof.

Also, the display unit 251 can project an image using the prism to the user's eyes. Further, the prism may be formed to be transmissive so that the user can view the projected image and the general view of the front (the range that the user views through the eyes) together.

As described above, the image output through the display unit 251 can be overlapped with the general view. The HMD 200 can provide an Augmented Reality (AR) in which a virtual image is superimposed on a real image or a background and displayed as a single image, using the characteristics of the display.

That is, the display unit 251 is formed so that external light can not pass through to realize a virtual reality (VR), or external light passes through the display unit 251 to realize Augmented Reality (AR) .

4A is a block diagram illustrating a configuration of a mobile terminal 100 that can be connected to the HMD 200 related to the present invention in a tethering manner. FIGS. 4B and 4C are conceptual diagrams showing the HMD 200 related to the present invention and the mobile terminal 100 that can be connected in a tethering manner, from the direction of the other.

The mobile terminal 100 includes a wireless communication unit 110, an input unit 120, a sensing unit 140, an output unit 150, an interface unit 160, a memory 170, a control unit 180, and a power supply unit 190 ), And the like. The components shown in FIG. 4A are not essential for implementing a terminal device, so that the terminal described in this specification can have more or fewer components than those listed above. In addition, the terminal or mobile terminal described in this specification may be referred to as a 'source device' or a 'VR source device' in that it provides contents corresponding to a virtual space image to be displayed in the HMD 200 described above.

More specifically, the wireless communication unit 110 among the above-mentioned components can perform wireless communication between the terminal 100 and the wireless communication system, between the terminal 100 and another terminal 100, or between the terminal 100 and an external server Lt; RTI ID = 0.0 > and / or < / RTI > In addition, the wireless communication unit 110 may include one or more modules for connecting the terminal 100 to one or more networks.

The wireless communication unit 110 may include at least one of a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short distance communication module 114, and a location information module 115 .

The input unit 120 includes a camera 121 or an image input unit for inputting a video signal, a microphone 122 for inputting an audio signal, an audio input unit, a user input unit 123 for receiving information from a user A touch key, a mechanical key, and the like). The voice data or image data collected by the input unit 120 may be analyzed and processed by a user's control command.

The sensing unit 140 may include one or more sensors for sensing at least one of the information in the terminal device, the surrounding environment information surrounding the terminal device, and the user information. For example, the sensing unit 140 may include a proximity sensor 141, an illumination sensor 142, a touch sensor, an acceleration sensor 143, a magnetic sensor A G-sensor 144, a gyro sensor 145, a motion sensor, an RGB sensor, an infrared (IR) sensor, a finger scan sensor An ultrasonic sensor, an optical sensor (see for example camera 121), a microphone (see microphone 122), a battery gauge, an environmental sensor (for example, a barometer, (E.g., a thermometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, a gas detection sensor, etc.), a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the terminal device disclosed in the present specification can combine and utilize information sensed by at least two of the sensors.

The output unit 150 includes at least one of a display unit 151, an acoustic output unit 152, a haptic tip module 153, and a light output unit 154 to generate an output related to visual, auditory, can do. The display unit 151 may have a mutual layer structure with the touch sensor or may be integrally formed to realize a touch screen. The touch screen functions as a user input unit 123 that provides an input interface between the terminal 100 and a user and can provide an output interface between the terminal 100 and a user.

The interface unit 160 serves as a path for communication with various types of external devices connected to the terminal 100. The interface unit 160 is connected to a device having a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, And may include at least one of a port, an audio I / O port, a video I / O port, and an earphone port. In the terminal 100, corresponding to the connection of the external device to the interface unit 160, it is possible to perform appropriate control related to the connected external device.

In addition, the memory 170 stores data supporting various functions of the terminal 100. The memory 170 may store a plurality of application programs or applications that are driven by the terminal 100, data for operation of the terminal 100, and commands. At least some of these applications may be downloaded from an external server via wireless communication. Also, at least some of these application programs may reside on the terminal 100 from the time of departure for the basic functions (e.g., call incoming, outgoing, message receiving, originating functions) of the terminal 100. Meanwhile, the application program may be stored in the memory 270, installed on the terminal 100, and operated by the control unit 180 to perform the operation (or the function) of the terminal apparatus.

The control unit 180 typically controls the overall operation of the terminal 100, in addition to the operations associated with the application program. The control unit 180 may process or process signals, data, information, and the like input or output through the above-mentioned components, or may drive an application program stored in the memory 170 to provide or process appropriate information or functions to the user.

In addition, the controller 180 may control at least some of the components illustrated in FIG. 4A to drive an application program stored in the memory 170. FIG. In addition, the controller 180 can operate at least two of the components included in the terminal 100 in combination with each other for driving the application program.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power to the components included in the terminal 100. The power supply unit 190 includes a battery, which may be an internal battery or a replaceable battery.

At least some of the components may operate in cooperation with each other to implement a method of operation, control, or control of a terminal device according to various embodiments described below. In addition, the operation, control, or control method of the terminal device may be implemented on the terminal device by driving at least one application program stored in the memory 170.

First, referring to the wireless communication unit 110, the broadcast receiving module 111 of the wireless communication unit 110 receives broadcast signals and / or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. More than one such broadcast receiving module may be provided to the terminal 100 for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

The mobile communication module 112 may be a mobile communication module or a mobile communication module such as a mobile communication module or a mobile communication module that uses technology standards or a communication method (e.g., Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution And an external terminal, or a server on a mobile communication network established according to a long term evolution (AR), a long term evolution (AR), or the like.

The wireless signal may include various types of data depending on a voice call signal, a video call signal or a text / multimedia message transmission / reception.

The wireless Internet module 113 is a module for wireless Internet access, and may be built in or externally attached to the terminal 100. The wireless Internet module 113 is configured to transmit and receive a wireless signal in a communication network according to wireless Internet technologies.

Wireless Internet technologies include, for example, wireless LAN (WLAN), wireless fidelity (Wi-Fi), wireless fidelity (Wi-Fi) Direct, DLNA (Digital Living Network Alliance), WiBro Interoperability for Microwave Access, High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE) and Long Term Evolution-Advanced (LTE-A) 213 transmit and receive data according to at least one wireless Internet technology in a range including internet technologies not listed above.

The wireless Internet module 213 for performing a wireless Internet connection through the mobile communication network can be used for wireless Internet access by WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE and LTE- May be understood as a kind of the mobile communication module 112.

The short-range communication module 114 is for short-range communication, and includes Bluetooth ™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB) (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology. The short range communication module 114 may be connected to the wireless communication system between the terminal 100 and the wireless communication system through the wireless area network or between the terminal 100 and another terminal 100, (100, or an external server) is located. The short-range wireless communication network may be a short-range wireless personal area network.

Herein, the other terminal 100 is a wearable device (e.g., a smartwatch, a smart smart, etc.) capable of interchanging data with the terminal 100 according to the present invention glass, HMD (head mounted display)). The short-range communication module 114 may detect (or recognize) a wearable device capable of communicating with the terminal 100, around the terminal 100. If the detected wearable device is a device authenticated to communicate with the terminal 100 according to the present invention, the control unit 180 may transmit at least a part of data processed in the terminal 100 to the short- To the wearable device. Accordingly, the user of the wearable device can use the data processed by the terminal 100 through the wearable device. For example, the user can make a phone call through the wearable device when a phone call is received in the terminal 100, or transmit the received message via the wearable device when a message is received in the terminal 100 It is possible to confirm.

The position information module 115 is a module for obtaining the position (or current position) of the terminal device, and a representative example thereof is a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, when the terminal device utilizes the GPS module, the terminal device can acquire the position of the terminal device using a signal transmitted from the GPS satellite. As another example, when the terminal device utilizes the Wi-Fi module, the terminal device can acquire the position of the terminal device based on information of a wireless access point (AP) transmitting or receiving the wireless signal with the Wi-Fi module. Optionally, the location information module 115 may perform any of the other modules of the wireless communication unit 110 to obtain replacement or additionally data regarding the location of the terminal device. The position information module 115 is a module used to obtain the position (or the current position) of the terminal device, and is not limited to the module that directly calculates or acquires the position of the terminal device.

The input unit 120 is for inputting image information (or signal), audio information (or signal), data, or information input from a user. For inputting image information, A plurality of cameras 121 may be provided. The camera 121 processes image frames such as still images or moving images obtained by the image sensor in the video communication mode or the photographing mode. The processed image frame may be displayed on the display unit 151 or stored in the memory 170. [ A plurality of cameras 121 provided in the terminal 100 may be arranged to have a matrix structure and a plurality of cameras 121 having various angles or foci may be provided to the terminal 100 through the camera 121 having a matrix structure, Can be input. In addition, the plurality of cameras 121 may be arranged in a stereo structure to acquire a left image and a right image for realizing a stereoscopic image.

The microphone 122 processes the external acoustic signal into electrical voice data. The processed voice data can be utilized variously according to a function (or an application program being executed) being performed in the terminal 100. Meanwhile, the microphone 122 may be implemented with various noise reduction algorithms for eliminating noise generated in receiving an external sound signal.

The user input unit 123 is for receiving information from a user. When information is inputted through the user input unit 123, the controller 180 can control the operation of the terminal 100 to correspond to the input information. The user input unit 123 may include a mechanical input unit (or a mechanical key such as a button located on the front / rear or side of the terminal 100, a dome switch, a jog wheel, Switches, etc.) and touch-based input means. For example, the touch-type input means may comprise a virtual key, a soft key or a visual key displayed on the touch screen through software processing, The virtual key or the visual key can be displayed on the touch screen with various forms. For example, the virtual key or the visual key can be displayed on the touch screen, ), An icon, a video, or a combination thereof.

Meanwhile, the sensing unit 140 senses at least one of the information in the terminal device, the surrounding environment information surrounding the terminal device, and the user information, and generates a corresponding sensing signal. The control unit 180 may control the driving or operation of the terminal 100 or may perform data processing, function or operation related to the application program installed in the terminal 100 based on the sensing signal. Representative sensors among various sensors that may be included in the sensing unit 140 will be described in more detail.

First, the proximity sensor 141 refers to a sensor that detects the presence of an object approaching a predetermined detection surface, or the presence of an object in the vicinity of the detection surface, without mechanical contact by using electromagnetic force or infrared rays. The proximity sensor 141 may be disposed in the inner area of the terminal device wrapped by the touch screen or proximity sensor 141 near the touch screen.

Examples of the proximity sensor 141 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. In the case where the touch screen is electrostatic, the proximity sensor 141 can be configured to detect the proximity of the object with a change of the electric field along the proximity of the object having conductivity. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.

On the other hand, for convenience of explanation, the act of recognizing that the object is located on the touch screen in proximity with no object touching the touch screen is referred to as " proximity touch & The act of actually touching an object on the screen is called a " contact touch. &Quot; The position at which the object is closely touched on the touch screen means a position where the object corresponds to the touch screen vertically when the object is touched. The proximity sensor 141 can detect a proximity touch and a proximity touch pattern (e.g., a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, have. Meanwhile, the control unit 180 processes data (or information) corresponding to the proximity touch operation and the proximity touch pattern sensed through the proximity sensor 141 as described above, and further provides visual information corresponding to the processed data It can be output on the touch screen. Furthermore, the control unit 180 may control the terminal 100 so that different operations or data (or information) are processed according to whether the touch to the same point on the touch screen is a proximity touch or a contact touch.

The touch sensor uses a touch (or touch input) applied to the touch screen (or the display unit 151) by using at least one of various touch methods such as a resistance film type, a capacitive type, an infrared type, an ultrasonic type, Detection.

For example, the touch sensor may be configured to convert a change in a pressure applied to a specific portion of the touch screen or a capacitance generated in a specific portion to an electrical input signal. The touch sensor may be configured to detect a position, an area, a pressure at the time of touch, a capacitance at the time of touch, and the like where a touch object touching the touch screen is touched on the touch sensor. Here, the touch object may be a finger, a touch pen, a stylus pen, a pointer, or the like as an object to which a touch is applied to the touch sensor.

Thus, when there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and transmits the corresponding data to the control unit 280. Thus, the control unit 180 can know which area of the display unit 151 is touched or the like. Here, the touch controller may be a separate component from the control unit 180, and may be the control unit 180 itself.

On the other hand, the control unit 180 may perform different controls or perform the same control according to the type of the touch object touching the touch screen (or a touch key provided on the touch screen). Whether to perform different controls or to perform the same control depending on the type of the touch object can be determined according to the current state of the terminal 100 or an application program being executed.

On the other hand, the touch sensors and the proximity sensors discussed above can be used independently or in combination to provide a short touch (touch), a long touch, a multi touch, a drag touch ), Flick touch, pinch-in touch, pinch-out touch, swipe touch, hovering touch, and the like. Touch can be sensed.

The ultrasonic sensor can recognize the position information of the object to be sensed by using ultrasonic waves. Meanwhile, the controller 180 can calculate the position of the wave generating source through the information sensed by the optical sensor and the plurality of ultrasonic sensors. The position of the wave source can be calculated using the fact that the light is much faster than the ultrasonic wave, that is, the time when the light reaches the optical sensor is much faster than the time the ultrasonic wave reaches the ultrasonic sensor. More specifically, the position of the wave generating source can be calculated using the time difference with the time when the ultrasonic wave reaches the reference signal.

The camera 121 includes at least one of a camera sensor (for example, a CCD, a CMOS, etc.), a photo sensor (or an image sensor), and a laser sensor.

The camera 121 and the laser sensor may be combined with each other to sense a touch of the sensing object with respect to the three-dimensional stereoscopic image. The photosensor can be laminated to the display element, which is adapted to scan the movement of the object to be detected proximate to the touch screen. More specifically, the photosensor mounts photo diodes and TRs (Transistors) in a row / column and scans the contents loaded on the photosensor using an electrical signal that varies according to the amount of light applied to the photo diode. That is, the photo sensor performs coordinate calculation of the object to be sensed according to the amount of change of light, and position information of the object to be sensed can be obtained through the calculation.

The display unit 151 displays (outputs) information to be processed by the terminal 100. For example, the display unit 151 may display execution screen information of an application program driven by the terminal 100, or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information.

In addition, the display unit 151 may be configured as a stereoscopic display unit for displaying a stereoscopic image.

In the stereoscopic display unit, a three-dimensional display system such as a stereoscopic system (glasses system), an autostereoscopic system (no-glasses system), and a projection system (holographic system) can be applied.

The sound output unit 152 may output audio data received from the wireless communication unit 110 or stored in the memory 170 in a call signal reception mode, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, The sound output unit 152 also outputs sound signals related to functions (e.g., call signal reception sound, message reception sound, and the like) performed in the terminal 100. [ The audio output unit 152 may include a receiver, a speaker, a buzzer, and the like.

The haptic module 153 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 153 may be vibration. The intensity and pattern of the vibration generated in the haptic module 153 can be controlled by the user's selection or the setting of the control unit. For example, the haptic module 153 may synthesize and output different vibrations or sequentially output the vibrations.

In addition to vibration, the haptic module 153 may be configured to perform various functions such as a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or suction force of the air through the injection port or the suction port, a touch on the skin surface, And various tactile effects such as an effect of reproducing a cold sensation using an endothermic or exothermic element can be generated.

The haptic module 153 can transmit the tactile effect through the direct contact, and the tactile effect can be felt by the user through the muscles of the finger or arm. At least two haptic modules 153 may be provided according to the configuration of the terminal 100.

The light output unit 154 outputs a signal for notifying the occurrence of an event using the light of the light source of the terminal 100. Examples of events that occur in the terminal 100 may include message reception, call signal reception, missed call, alarm, schedule notification, email reception, information reception through an application, and the like.

The signal output by the light output unit 154 is realized as the terminal device emits light of a single color or a plurality of colors to the front or rear surface. The signal output may be terminated by the terminal device detecting the event confirmation of the user.

The interface unit 160 serves as a path for communication with all external devices connected to the terminal 100. In this specification, the interface unit 160 The HMD 200 and And serves as a channel for tethering connections. Receives data from an external device, supplies power to each component in the terminal 100, or allows data in the terminal 100 to be transmitted to an external device. For example, a port for connecting a device equipped with a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, an audio I / O port, a video I / O port, an earphone port, and the like may be included in the interface unit 160.

The identification module is a chip for storing various information for authenticating the usage right of the terminal 100 and includes a user identification module (UIM), a subscriber identity module (SIM) a universal subscriber identity module (USIM), and the like. Devices with identification modules (hereinafter referred to as "identification devices") can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 100 through the interface unit 160. [

When the terminal 100 is connected to an external cradle, the interface unit 160 may be a path through which power from the cradle is supplied to the terminal 100, or various commands A signal may be transmitted to the terminal 100. Various command signals input from the cradle or the power source can be operated with a signal for recognizing that the terminal 100 is correctly mounted on the cradle.

The memory 170 may store a program for the operation of the controller 180 and temporarily store input / output data (e.g., a phone book, a message, a still image, a moving picture, etc.). The memory 170 may store data related to vibration and sound of various patterns outputted when a touch is input on the touch screen.

The memory 170 may be a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), a multimedia card micro type ), Card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read memory, a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and / or an optical disk. The terminal 100 may operate in association with a web storage that performs the storage function of the memory 170 on the Internet.

Meanwhile, as described above, the control unit 180 controls the operations related to the application program and the general operation of the terminal 100. [ For example, when the state of the terminal device satisfies a set condition, the control unit 180 can execute or release a lock state for restricting the input of a user's control command to applications.

In addition, the control unit 180 performs control and processing related to voice communication, data communication, video call, or the like, or performs pattern recognition processing to recognize handwriting input or drawing input performed on the touch screen as characters and images, respectively . Further, the controller 180 may control any one or a plurality of the above-described components in order to implement various embodiments described below on the terminal 100 according to the present invention.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power necessary for operation of the respective components. The power supply unit 190 includes a battery, the battery may be an internal battery configured to be chargeable, and may be detachably coupled to the terminal body for charging or the like.

Also, the power supply unit 190 may include a connection port, and the connection port may be configured as an example of an interface 260 through which an external charger for supplying power for charging the battery is electrically connected.

As another example, the power supply unit 190 may be configured to charge the battery in a wireless manner without using the connection port. In this case, the power supply unit 190 may use at least one of an inductive coupling method based on a magnetic induction phenomenon from an external wireless power transmission apparatus and a magnetic resonance coupling method based on an electromagnetic resonance phenomenon Power can be delivered.

In the following, various embodiments may be embodied in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

5A, 5B and 5C are views for explaining an example in which an image of a virtual space displayed on a display unit is controlled according to a movement of a user wearing the HMD according to the present invention.

Referring to FIG. 5A, FIG. 5A shows a head portion of a user wearing the HMD 200. FIG. When the user wears the HMD 200, the control unit 280 of the HMD 200 can display the image of the virtual space corresponding to the content selected by the user through the display unit 251. [ In this case, the control unit 280 may display on the display unit 251 a screen corresponding to the front side 300 of the current HMD 200 among the images in the virtual space.

The controller 280 may detect a roll, a yaw, and a pitch according to a movement of a user wearing the HMD 200. Also, the control unit 280 can change the image of the virtual space corresponding to the viewing angle of the user according to the detected roll, yaw, and pitch values.

For example, when the user lifts the head, the screen corresponding to the front face 300 may be changed to a screen corresponding to the upward gaze angle in accordance with the change of the pitch value. That is, the image of the virtual space displayed on the display unit 251 of the HMD 200 according to the changed viewing angle can be changed to a screen of another area (upper area).

As another example, the viewing angle of the image 300 in the virtual space displayed on the display unit 251 of the HMD 200 may be changed according to the change of the yaw value. That is, the image of the virtual space displayed on the display unit 251 of the HMD 200 according to the changed viewing angle may be changed to a screen of another area (e.g., another area in the left direction or the right direction).

The control unit 280 of the HMD 200 controls the operation of the HMD 200 according to the motion of the head of the user sensed through the HMD 200, Can be displayed. Accordingly, the wearer wears the HMD 200, and then the user can watch the image of the other area in the virtual space by turning the head or lifting the head.

Next, FIGS. 5B and 5C are views showing a virtual space image of the virtual reality contents displayed in the HMD 200 related to the present invention.

The HMD 200 according to the present invention can provide the virtual reality contents to the user through the display unit 251. [ These virtual reality contents represent reality that is different from actual reality. That is, as shown in FIG. 5B, a user wearing the HMD 100 provides a three-dimensional virtual space 500 so as to be mistaken as if he or she is in a specific area or another space while preventing the actual reality from being seen. As a result, the user can feel as if he is in a virtual space. This virtual space can be actually achieved by providing a three-dimensional spatial image 500 through a left-eye image and a right-eye image respectively corresponding to a user's binocular parallax.

Also, the virtual space 500 (or the virtual space image) related to the present invention means a 3D image formed in all directions (for example, 360 degrees) based on the user wearing the HMD 200 as described above can do. In addition, the virtual space 500 refers to a rendered three-dimensional space image (or stereoscopic image, 3D image) so as to recognize that a user wearing the HMD 200 exists in a specific space (or virtual space) can do. In addition, the virtual space contents are contents formed by such 3D images or stereoscopic space images, such as a web browser, a video playback, a schedule management, a call, a game, music, a document work, a message, e-book, traffic information, and a program that performs the function of updating the application may be formed as 3D images or stereoscopic images.

Hereinafter, such a virtual space content can be named 'content' for convenience of explanation. Accordingly, contents received in the mobile terminal 100 connected to the HMD 200 according to the present invention mean virtual space contents received to form a virtual space 500 on the display unit 251 of the HMD 200 can do.

5B, the display unit 251 of the HMD 200 displays one area (for example, the first area A) of the virtual space 500 corresponding to the content, Can be output. To this end, the controller 280 of the HMD 200 may transmit a request for transmission of the virtual space contents to the mobile terminal 100.

Here, the first area may be a default area that is output as a default when the virtual space is outputted through the display unit 251 of the HMD 200, Or may be a specific area set by the user. In addition, the specific object may be an object set by the user (or set by default) among at least one graphic object included in the virtual space.

When the virtual space is previously output from the HMD 200 and then output to the display unit 251 of the HMD 200, the first area is the virtual space previously output from the HMD 200, It may be one area displayed at the end of the display.

One area A of the virtual space output to the display unit 251 of the HMD 200 may be changed based on the movement of the user wearing the HMD 200 or the movement of the HMD 200 accordingly have. For this purpose, the sensing unit 240 of the HMD 200 can detect and track such movements (including the degree of motion and direction) of the user's head gesture wearing the HMD 200. The motion may be performed in a state in which one area A of the virtual space contents is output to the display unit 251 of the HMD 200. [ It should be understood that the movement includes movement, rotation, or combination of the heads of the user wearing the HMD 200, movement of the HMD 200, rotation, or a combination thereof.

The sensing unit 240 or the HMD control unit 280 controls the control unit 280 of the HMD 200 or the control unit 180 of the mobile terminal 100 . Then, the controller 280 (or the controller 180) controls the display unit (not shown) of the HMD 200 to output the displayed area A and the area B 251).

5C, when the HMD main body 200 is moved in accordance with the head gesture of the user while the one area A of the virtual space is being output to the display unit 251 The control unit 180 of the mobile terminal 100 displays the area A and the area B of the virtual space 500 on the display unit 2.51 of the HMD as shown in FIG. ). ≪ / RTI >

With this configuration, the present invention can provide a user interface that allows a user who uses the HMD 200 to feel as if it is actually moving in a virtual space.

On the other hand, in the present specification, it is assumed that the virtual space experience through the HMD 200 is performed in the moving means. Here, the moving means may include all kinds of transportable vehicles such as all vehicles such as a car, a taxi, and a truck, a bicycle, a ship, a train, an airplane, a cable car, and the like.

In the case of experiencing the virtual space using the HMD 200 during the movement by the moving means, as described above with reference to FIGS. 5A to 5C, in addition to the screen change due to the intentional head movement of the user, Unintentional head movements may be detected. For example, in the case of performing a virtual space experience through the HMD 200 while moving on a vehicle, if the vehicle performs a curve drive to the right side, the user wearing the HMD 200 moves the head to the right And the right area of the virtual space is recognized. That is, there may occur a malfunction problem in which the HMD 200 is changed to a screen different from the actual screen expected by a user who wears the HMD 200 while moving. Accordingly, the present invention is implemented to display a virtual space suitable for an actual screen that a user expects while moving through a moving means.

Figs. 6 and 7 are diagrams for explaining a method of tracking the head position of the HMD in the moving means in the HMD related to the present invention. Fig.

Referring to FIG. 6, the HMD 200 according to the present invention is a block diagram for explaining a position correction method for preventing a malfunction when a virtual space image is provided during movement by a moving means. The HMD 200 and the VR source device 100 are connected in a tethering manner so that the HMD 200 can receive signals from the VR source device 100 or transmit signals to the VR source device 100 have.

The HMD 200 includes an acceleration sensor 241 and a gyroscope 243 or ii) a three-axis magnetic sensor 242 or iii) an acceleration sensor 241, a magnetic sensor 242, And the gyro sensor 243 to sense the motion value of the HMD 200 and transmit the sensing value to the control unit 280. [ Hereinafter, the motion value of the HMD 200 sensed in this way can be named as " first motion value ".

Meanwhile, in the present invention, while the movement by the moving means is sensed, the provided sensing value of the controller 280 is provided to the connected VR source device 100 (50). This is for controlling the virtual space image according to the head movement of the user and the motion value of the HMD 200.

Then, the VR source device 100 receiving the motion value of the HMD 200 also senses its motion value. To this end, at least some of the acceleration sensor 141, the magnetic sensor 142, and the gyro sensor 143 provided in the VR source device 100 may be utilized. Further, correction of the VR source device 100 may be performed through a plurality of combinations. Hereinafter, the motion value of the VR source device 100 sensed in this manner can be named " second motion value ".

The VR source device 100 can control the virtual space image to be output to the HMD 200 by tracking the corrected actual head position by subtracting the second motion value from the first motion value. That is, the image of the corrected head position is provided to the HMD 200 (70). According to this, even if the first motion value includes the change value of the movement direction and the movement direction by the movement means, the virtual space image is not controlled unintentionally by excluding it through the second motion value. Accordingly, the user receives only the expected screen corresponding to the actual head movement regardless of the movement of the moving means.

7 shows a specific process of correcting the head position of the user.

In FIG. 7, 'A' is a VR source device for rendering a content corresponding to a virtual space image and a virtual space image according to a head movement of a user, 'B' is a virtual space image, And an HMD that outputs a screen controlled according to the head movement.

(X, y, and z axis values of the acceleration sensor and the gyro sensor are obtained (701), and the x, y, and z axis values of the acceleration sensor and the gyro sensor Value ') is obtained (702). Then, the motion values of A and B obtained above are aligned with each other based on the x, y and z axes of the respective magnetic sensors provided in A and B (calibration). When the directions of the axis values of the acceleration sensors A and B and the magnetic path sensors A and B are adjusted to correspond to each other based on the magnetic sensor, a similar data value pattern is now displayed in the moving means in which the motion values of A and B move.

Here, the movement of the moving means is a motion irrespective of the intention of the user wearing B However, even after the calibration, the motion values of A and B are reflected in the movement of the moving means, and even if there is no head movement of the user, a malfunction occurs in which the screen is controlled according to the moving direction of the moving means.

In order to exclude the unintentional head movement of the user, the motion values of A and B are compared, and the process of determining the difference as the actual head motion of the user is performed. More specifically, the head movement of the actual user is calculated by subtracting the motion value of A from the motion value of B (703). Then, by providing the head position correction value to the control unit (704), the image is rendered using the head position correction value so that only the screen change corresponding to the user's intentionally moving head direction is generated.

By such head position correction, for example, the problem that the virtual space image is misunderstood in the direction in which the user is not moving, that is, in the traveling direction of the moving means, is misunderstood and the screen control malfunctions is solved.

Next, FIGS. 8A, 8B and 8C show views for more specifically explaining the change of the expected screen direction displayed on the HMD according to the change of the moving direction of the moving means.

8A shows an example of a moving path and a moving direction of the moving means carried by a user wearing the HMD 200. The moving means moves in the right direction at the position A, ) Direction.

Meanwhile, the mobile terminal 100 connected to the HMD 200 in a tethering form provides a virtual space image corresponding to the motion value in the HMD 200 to the HMD 200. To this end, the acceleration sensor and the gyro sensor of the HMD 200 sense the rotational angular velocity and the acceleration, respectively, and the magnetic sensor senses the surrounding magnetic field and transmits the respective axis values to the control unit 280. Then, the control unit 280 compares the sensor values of the acceleration sensor and the gyro sensor in the different directions on the basis of the x, y, and z axis values of the magnetic sensor.

In addition, the control unit 280 calculates the position movement value of the HMD 200, that is, the first motion value, by performing fusion processing on these sensor values. Specifically, the controller 280 acquires rotational angular velocity values for two axes (y, z axes) horizontal to the ground, among the sensor values obtained through the three-axis gyro sensor 243, through the acceleration sensor 241 Axis gyro sensor 243 and obtains a rotational angular velocity value for one axis (x-axis) perpendicular to the ground of the sensor values obtained through the three-axis gyro sensor 243 from the sensor values acquired through the magnetic sensor 242 It is possible to calculate the first motion value by correcting it with the sensor value.

This first motion value is changed based on the sensing values continuously generated through the acceleration sensor 241, the gyro sensor 243, and the magnetic sensor 242 in accordance with the movement of the head of the user wearing the HMD 200 do. The first motion value thus calculated is transmitted to the controller 180 of the tethered mobile terminal 100. The control unit 180 of the mobile terminal 100 may correct the first motion value using the second motion values sensed through its sensors (gyro sensor, acceleration sensor, and magnetic sensor) Thereby calculating the head position of the actual user and rendering the virtual space image corresponding to the corrected position.

FIG. 8B shows a virtual space image expected when there is no head movement of a user while the moving means is moving rightward along the moving path at point A in FIG. 8A. If the right screen 801 is provided in both eyes, even though there is no actual head movement of the user, movement of the moving means is reflected in the conventional method. In the present invention, the current virtual space image corresponding to the corrected position, (810). ≪ / RTI > Accordingly, the expected screen of the user and the actual screen coincide with each other.

FIG. 8C shows a virtual space image expected when the user's head movement is directed outward while the moving means moves to the left along the movement path at point B in FIG. 8A. In the past, the movement of the movement means is reflected in the head movement of the user, and a left screen + a left screen (i.e., a rear screen 802 rotated more than the actual screen) is provided. However, in the present invention, it can be confirmed that the virtual space image corresponding to the corrected position, i.e., the left image 811, is provided in both eyes through the HMD 200. [ Accordingly, the expected screen of the user and the actual screen coincide with each other.

9 is a representative flowchart for explaining a control method of the HMD related to the present invention.

Referring to FIG. 9, a first sensor value corresponding to head movement is detected (901) in the HMD 200 connected to the VR source device 100 in a tethering manner. Here, the first sensor value means that the x, y, and z values measured by the acceleration sensor and the gyro sensor of the HMD 200 are aligned based on the x, y, and z axis values of the magnetic sensor. That is, the first sensor value becomes the above-described first motion value. The detailed description is given above, so it will not be described here.

Next, the controller 280 of the HMD 200 receives the first sensor value and determines whether it is in the moving mode by the current moving means (902). Whether the mode is the moving mode by the moving means can be determined through various methods such as setting through the main body or the terminal 100, the boarding detection signal to the moving means, and the movement detection of the moving means. Also, the moving mode may be automatically released while the moving means is not moving even while boarding the moving means.

If the motion mode is determined, the VR source device 100 receives the first sensor value from the HMD 200 (903). Also, the VR source device 100 acquires the second sensor value using the acceleration sensor and the gyro sensor, and acquires the third sensor value using the magnetic sensor (904).

Then, a first correction is performed using the obtained second sensor value and the third sensor value (905). Specifically, the first correction may refer to a value in which the sensor values of the acceleration sensor and the gyro sensor are aligned with respect to the three-axis values of the magnetic sensor. Alternatively, the first correction may refer to a second motion value in the VR source device 100. Alternatively, the first correction may mean that the three-axis values of the magnetic sensor of the VR source device 100 are recognized and applied to the second sensor value in consideration of the three-axis values by the magnetic sensor of the HMD 200. [ Thus, the value obtained by the first correction becomes the first motion value.

The second correction is performed using the first correction value and the first sensor value received from the HMD 200 (906). Specifically, the first correction value is subtracted (or subtracted) from the first sensor value received from the HMD 200. As a result, the finally corrected position value can be calculated.

Then, the VR source device 100 estimates the corrected position value as a head position, and performs rendering (907). The rendered image or rendering data itself is transferred from the VR source device 100 to the HMD 200. [ The rendered image corresponding to the actual head position is output through the display unit 251 of the HMD 200 (908)

On the other hand, if the moving mode is not the moving mode, the head position is estimated and tracked using only the first sensor value received from the HMD 200 without using the motion value and the position correction value on the VR source device 100 side, And outputs the corresponding virtual space image to the HMD 200 (907, 908).

Although not shown, in one embodiment, the detection interval of the sensor value or the calculation interval of the position correction value may be appropriately adjusted in accordance with the moving direction of the moving means as well as the moving speed of the moving means .

At least one of the display unit 251 of the HMD 200 or the display unit 151 of the VR source device 100 is displayed while the HMD 200 is operating in the moving mode, .

In yet another example, in order to improve the accuracy of the head position estimation, the peripheral image captured by the camera 221 mounted on the HMD 200 is subjected to image processing in addition to the first, second, and third sensor values described above, It is possible to detect a change in the head posture of the user wearing the portable terminal 200. Such head attitude change can be obtained by integrating and calculating the angular velocity obtained from the gyro sensor and calculating the movement amount of the head from this. The thus obtained value may be used as the first sensor value.

As another example, there is a method of generating a specific input signal through the input unit 223 of the HMD 200 or the like to have a fixed head position. For example, the moving unit may perform a predetermined input through the input unit 223 while the moving unit is moving so that the virtual space image is fixed to the current viewing area. However, in this case, even if the user moves the head, it does not move to another area of the virtual space.

In addition, in one embodiment, when the moving mode by the moving means is released, the control unit 280 of the HMD 200 may stop the first sensor value (or the first motion value) The operation of the communication unit 210 can be restricted so as not to be transmitted.

10A and 10B illustrate an example of a method of setting a movement mode of the HMD in a mobile terminal connected to the HMD related to the present invention.

The setting of the movement mode of the HMD 200 may be performed through the setting screen of the mobile terminal, as shown in FIG. 10A. The control unit 280 of the HMD 200 may perform the movement mode in response to the reception of the predetermined setting signal from the mobile terminal 100 connected through the communication unit 210. [ For example, as shown in FIG. 10A, when 'setting' is selected in the pop-up message 1010 for setting the mobile mode, the mobile terminal 100, that is, the VR source device A guidance message 1020 is displayed on the display unit 151 to fix the mobile terminal 100 so as not to move. At this time, the guide image 1002 may be output to the display unit 151 together.

In addition, the setting signal for entering the moving mode may be transmitted to the HMD 200 at a time point when the position of the VR source device 100 is detected according to the guide message. Accordingly, sensor values are initialized through the gyro sensor, the acceleration sensor, and the magnetic sensor in the VR source device 100 and the HMD 200, respectively.

Meanwhile, although not shown, the movement mode may be automatically performed when the movement of the movement means is sensed after the HMD 200 and the VR source device 100 are connected in a tethering manner. In this case, if the movement of the movement means is no longer detected for a predetermined time, the movement mode is automatically released, and if the movement of the movement means is sensed, the movement mode can be started again. However, also in this case, the first, second, and third sensor values can be continuously acquired and accumulated.

11 and 12 are flowcharts for explaining another control method of the HMD related to the present invention. FIG. 11 is a flowchart illustrating an example in which a change in an initial position of a terminal connected to an HMD is detected according to the occurrence of an event and is reflected in sensing correction. 12 is a flowchart showing an example of performing head position correction according to whether or not a movement direction change of the moving means is sensed.

11, after the HMD 200 and the terminal 100, that is, the VR source device are tethered, the axis values corresponding to the initial positions are initialized (1111). Here, the axis values corresponding to the initial positions may be directions corresponding to the x, y, and z axis values of the VR source device 100 and the magnetic sensor of the HMD 200. [ Alternatively, the axis values corresponding to the initial position may only refer to directions corresponding to the x, y, and z axis values by the magnetic sensor corresponding to the fixed position of the VR source device at the time of entering the movement mode. Alternatively, the axis values corresponding to the initial position may refer to all of the initial axis values by the gyro sensor, the acceleration sensor, and the magnetic sensor corresponding to the fixed position of the VR source device.

When the movement mode of the HMD 200 is started, the first initial axis values corresponding to the initial position of the HMD 200 and the second initial axis values corresponding to the initial position of the VR source device 100 are stored in the predetermined memory area . Accordingly, the initial position of the HMD 200 and the VR source device 100 can be recognized. Since the HMD 200 and the VR source device 100 are simultaneously changed to the same pattern even if the initial position is changed according to the movement of the moving means, It can be determined that the fixed position of the VR source device 100 is released.

While the HMD 200 is operating in the moving mode, the HMD 200 may sense occurrence of a predetermined event (1112). Here, an event refers to an event in which the degree of motion of at least one of the mobile terminal (or the VR source device) and the HMD is out of the reference range, the detachment of the HMD is detected, This may mean that there is a reconnection of the HMD.

When the occurrence of such an event is detected, the change of the values of the axis values of the VR source device, for example, the x-axis, y-axis, and z-axis through the acceleration sensor 141 and the gyro sensor 243 included in the VR source device (Step 1113). Specifically, the axis values of the VR source device may be sensed through the sensing unit 140 of the terminal 100 or may be received and confirmed by the HMD 200. Alternatively, when the occurrence of an event is detected, the HMD 200 may transmit a control signal requesting the terminal 100 to sense a change in the axis values of the VR source device through the communication unit 210.

Here, the change of the axis values is excluded when the axis values of the VR source device change due to the movement of the moving means. That is, the change of the axis values means that the values of the x-axis, y-axis, and z-axis initialized as the fixed position of the VR source device 100 sensed at the time when the HMD 200 operates in the moving mode are changed . For example, there may be a case where the VR source device 100 is placed in the user's pocket, but is quickly moved in the moving means according to the inertial force while being removed from the pocket.

When a change in the axis values of the VR source device is detected, the initial value correction of the sensing is performed based on the changed axis values (1114). That is, the control unit 280 of the HMD 200 recognizes that the initial position of the VR source device has been changed, and acquires the values of the new x-axis, y-axis, and z-axis through the magnetic sensor so as to recognize the changed position A control signal can be provided.

Specifically, when the occurrence of the predetermined event is detected, the control unit 280 of the HMD 200 determines whether to change the initial positions by comparing the degree of change of each of the current axis values from the stored initial axis values, To the VR source device 100, a wireless signal for re-tracking the user's head position.

In another example, unlike the example described in FIG. 11, while the HMD 200 is operating in the moving mode, it may re-search whether the initial positions corresponding to the initial axis values stored at predetermined time intervals have changed.

Next, FIG. 12 shows an example of performing head position correction according to whether or not a movement direction change of the moving means is sensed. In this embodiment, it is presupposed that it is not necessary to perform the position correction when there is almost no change in the moving direction even when the moving means is moving depending on the type of the moving means.

Referring to FIG. 12, the step of detecting the connection between the HMD 200 and the VR source device 100 is started (1211).

Thereafter, a step of detecting a change in the moving direction of the moving means on which the user wearing the HMD 200 is sensed is performed (1212). The change of the moving direction can be judged by a predetermined reference time t unit. The change in the moving direction can be excluded when the moving means moves but maintains the straight direction, or when the first moving direction and the moving angle are maintained.

When a change in the movement direction is detected, the head movement of the user wearing the HMD 200 can be detected (1213). Such head movement may be detected through the external camera 221 or the like in addition to the sensing unit 240 of the HMD 200. [

As a result of the detection, if there is a head movement, by changing the screen direction by the difference of the sensor correction values (1214), another area of the virtual space is displayed.

On the other hand, if there is no head movement, the sensor correction value is excluded from the head movement value sensed by the HMD 200 (1215), and the current screen orientation is maintained (1216).

Meanwhile, the sensing means for maintaining the current screen orientation or changing the current screen orientation by the actual head movement may be performed by combining the sensors provided in the moving means. For example, it is possible to exclude movement by the moving means according to the direction change pattern detected through the magnetic sensor of the moving means itself. In this case, it is not necessary to perform the first correction and the second correction on the side of the VR source device 200. [ That is, the corrected first sensor value, the second sensor value, and the third sensor value by the moving means can be processed by fusion processing to calculate the corrected head position. At this time, the process of calibrating the fixed position of the VR source device 200 is also unnecessary.

As described above, according to the HMD and the control method of the HMD according to the embodiment of the present invention, when the virtual space image is viewed using the HMD in the moving means such as the moving vehicle, This erroneous recognition problem is solved. In addition, it provides user convenience by eliminating the sensor value which is not related to the virtual space image by the HMD without the additional input or recognition of the user.

The present invention described above can be embodied as computer-readable codes on a medium on which a program is recorded. The computer readable medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of the computer readable medium include a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, , And may also be implemented in the form of a carrier wave (e.g., transmission over the Internet). In addition, the computer may include a control unit 280 of the terminal. Accordingly, the above description should not be construed in a limiting sense in all respects and should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention.

Claims (17)

1. An HMD (Head Mounted Display) connected to a mobile terminal,
A communication unit for performing communication with the mobile terminal;
A display unit for displaying a virtual space image corresponding to the content stored in the mobile terminal;
A sensing unit for sensing movement of the HMD;
And a controller coupled to the communication unit and transmitting the first motion value detected by the sensing unit to the connected mobile terminal while the HMD is operating in the moving mode,
Wherein,
Wherein the HMD receives and displays a screen corresponding to the head position of the user calculated based on the first motion value and the second motion value sensed by the mobile terminal from the mobile terminal. The method according to claim 1,
Wherein the user's head position is tracked while excluding the second motion value from the first motion value during the motion mode. 3. The method of claim 2,
Wherein the first and second motion values are acquired using an acceleration sensor, a gyro sensor, and a magnetic sensor. 3. The method of claim 2,
Wherein the second motion value is a first correction value obtained based on a second value obtained from the acceleration sensor and the gyro sensor of the mobile terminal using the first value obtained from the magnetic sensor of the mobile terminal HMD. 5. The method of claim 4,
Wherein the head position of the user is estimated from a first motion value obtained from the acceleration sensor and the gyro sensor of the HMD and a second correction value obtained based on the first correction value. The method according to claim 1,
Wherein,
And controls the communication unit to limit transmission of the first motion value when the movement mode is released. The method according to claim 1,
Wherein,
And performs the movement mode in response to reception of a set signal from the mobile terminal through the communication unit. 8. The method of claim 7,
Wherein the setting signal is generated through menu setting of the mobile terminal and is received from the mobile terminal when it is detected that the position of the mobile terminal is fixed. The method according to claim 1,
Wherein the first initial axis values corresponding to the initial position of the HMD and the second initial axis values corresponding to the initial position of the mobile terminal are stored when the movement mode is started. 10. The method of claim 9,
Wherein,
Determining whether to change the initial positions by comparing the degree of change of each of the current axis values from the stored initial axis values when the occurrence of the predetermined event is detected, and transmitting a radio signal for re- To the mobile terminal. 11. The method of claim 10,
Wherein the event is detected when the degree of motion of at least one of the mobile terminal and the HMD is out of the reference range, the detachment of the HMD is detected, or there is a reconnection of the mobile terminal and the HMD. 10. The method of claim 9,
Wherein during the moving mode, whether or not the initial positions corresponding to the stored initial axis values change is re-searched at predetermined time intervals. The method according to claim 1,
Wherein the movement mode is started when a movement of a moving means aboard a user wearing the HMD after connection between the HMD and the mobile terminal is sensed. 14. The method of claim 13,
Wherein the movement mode is released if movement of the movement means is not detected for a predetermined time, and the movement mode is restarted when movement of the movement means is detected. 14. The method of claim 13,
Wherein,
And controls the current screen direction to be maintained based on the sensor correction value based on the second motion value when a change in the moving direction of the moving means is detected and there is no head movement of the user. 15. The method of claim 14,
Wherein,
And controls to change the current screen direction by subtracting the sensor correction value based on the second motion value from the first motion value when a change in the moving direction of the moving means is detected and a head movement of the user is detected The HMD. A method of controlling an HMD (Head Mounted Display) connected to a mobile terminal,
Displaying a virtual space image corresponding to the content stored in the mobile terminal;
Transmitting a first motion value of the HMD to the connected mobile terminal while the HMD is operating in a moving mode; And
And receiving and displaying a screen corresponding to the head position of the user calculated based on the first motion value and the second motion value sensed by the mobile terminal from the mobile terminal, Way.

Images