HK1234171B - Method and device for display device viewer gaze attraction - Google Patents

Description

Method and device for attracting the viewer's gaze of a display device

Background Art

Content creation and/or delivery equipment can utilize sight line (gaze) tracking system to track the position of user's sight line.In some cases, content creators such as film producers, advertisers, or game developers may wish to guide users to look at specific positions or features displayed via display devices. The sight line tracking system can be used to estimate the current position of the user's sight line. If such current position does not correspond to the desired position or feature, the desired position or feature can be visually enhanced in a significant manner to capture the user's attention and sight line. For example, the desired position and feature can be highlighted with bright contrasting colors, the desired position and feature can be visually jumped or flickered, and/or the desired position and feature can be changed in appearance in other ways to attract the user's attention.

However, enhancing the desired location or feature in this way may distract the user and may interrupt an otherwise pleasant user experience. Additionally, where the desired feature is an advertisement, the user may not want to be intentionally diverted from the user experience to view the advertisement.

Summary of the Invention

Various examples of attracting a viewer's gaze are disclosed herein. In one disclosed method, a display device is controlled to display a target object, and gaze tracking data from a gaze tracking system is used to monitor the viewer's gaze position on the display. A guidance element is displayed to move along a calculated dynamic path that passes through a predetermined area adjacent to the viewer's gaze position and leads to the target object.

The gaze tracking data is used to determine whether the viewer's gaze position is within a predetermined deviation threshold of the guidance element. If the viewer's gaze position is within the predetermined deviation threshold of the guidance element, the display is controlled to continue displaying the guidance element moving along the calculated dynamic guidance path toward the target object. However, if the viewer's gaze position deviates from the guidance element by at least the predetermined deviation threshold, the display is controlled to discontinue displaying the guidance element moving along the calculated dynamic guidance path toward the target object.

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

1 is a schematic diagram of an example computing device that may be utilized with a display to attract a viewer's attention according to examples of the present disclosure.

2 is a schematic diagram of a viewer in a room interacting with a computing device and a display that attracts the viewer's attention, according to an example of the present disclosure.

FIG. 3 is a schematic diagram of a movie displayed on a display device that may attract a viewer's attention according to an example of the present disclosure.

FIG4 is a schematic diagram of a game displayed on a display device that may attract the viewer's attention according to an example of the present disclosure.

FIG5 is a schematic diagram of a web page displayed on a display device that can attract a viewer's attention according to an example of the present disclosure.

FIG6 is a schematic diagram of an example head-mounted display device.

7A and 7B are flowcharts of a method for attracting the sight of a viewer of a display device according to an example of the present disclosure.

8 is a simplified schematic diagram of an example of a computing device.

DETAILED DESCRIPTION

FIG1 shows a schematic diagram of an example implementation of a computing device for attracting the sight of a viewer of a display device. As described in more detail below, the computing device uses sight tracking data from a sight tracking system to monitor the viewer's sight position. The sight attraction program controls the display device to display a guiding element moving along a calculated dynamic path, which passes through a predetermined area adjacent to the viewer's sight position and leads to a target object. If the viewer's sight does not leave the guiding element, the guiding element continues to move toward the target object along the calculated dynamic guiding path. If the viewer's sight deviates from the guiding element by at least a predetermined deviation threshold, the guiding element is no longer displayed moving toward the target object along the calculated dynamic guiding path.

In various examples, the computing device can be physically separate from or integrated into a display device with which a viewer can interact. FIG1 schematically illustrates an example of a computing device 10 that is physically separate from a display device 14. In this example, computing device 10 can include or be integrated into a separate device, such as a set-top box, a game console, a webcam, a head-mounted computing device or other wearable computing device, a keyboard, a dedicated peripheral, or other similar device that does not include an integrated display.

Computing device 10 can be operatively connected to display device 14 using a wired connection, or can be wirelessly connected via WiFi, Bluetooth, or any other suitable wireless communication protocol. For example, computing device 10 can be communicatively coupled to network 16. Network 16 can take the form of a local area network (LAN), a wide area network (WAN), a wired network, a wireless network, a personal area network, or a combination thereof, and can include the Internet. Additional details about the components and computing aspects of computing device 10 are described in more detail below with reference to FIG8.

1 also shows an example of a computing device 12 integrated into a head-mounted display (HMD) device 18. The HMD device 18 can create and display a virtual reality environment or a mixed reality environment to a first viewer 22. In these examples, the HMD device 18 can include a display program 26 that can generate the virtual environment or mixed reality environment for display via the HMD device. The virtual environment can include one or more visual elements in the form of virtual images (e.g., three-dimensional (3D) holographic objects and two-dimensional (2D) virtual images) generated and displayed via the HMD device 18. In the mixed reality environment, the HMD device 18 can enable the viewer to view such holographic objects and virtual images within the physical environment surrounding the viewer.

As described in more detail below, in some examples, HMD device 18 may include a transparent, translucent, or non-transparent display supported in front of one or both eyes of a viewer. HMD device 18 may include various sensors and related systems that receive physical environment data from the physical environment. For example, HMD device 18 may include a depth sensor system 30 that includes one or more depth cameras that generate depth image data.

In some examples, HMD device 18 may include an optical sensor system 32 that utilizes at least one outward-facing sensor (e.g., an RGB camera or other optical sensor). The outward-facing sensor can capture two-dimensional information from the physical environment. HMD device 18 may also include a position sensor system 34 that includes one or more accelerometers, gyroscopes, head tracking systems, and/or other sensors for determining the user's position or orientation.

The HMD device 18 may also include a transducer system 38, which includes one or more actuators that convert electrical signals into another form of energy. In some examples, the transducer system 38 may include one or more speakers for providing audio feedback to the viewer. In other examples, the transducer system 38 may include one or more tactile transducers for generating tactile feedback (e.g., vibration) and providing it to the viewer. The HMD device 18 may also include a microphone system 42 and one or more microphones for receiving audio input from the physical environment.

Additionally, the example shown in FIG1 shows the computing device 12 integrated into the HMD device 18. It should be understood that in other examples, the computing device can be a separate component from the HMD device 18. Many types and configurations of HMD devices 18 having various form factors can be used and are within the scope of the present disclosure. A more detailed description of an example HMD device is provided below with reference to FIG6.

It should also be understood that computing device 12 may include or be integrated into any other suitable type or form of display device, such as a tablet computer, laptop computer, smartphone, or other mobile computing device, desktop computing device, stand-alone monitor, wall-mounted display, interactive whiteboard, or other similar device with an integrated display. Such a device may also include an eye tracking system, as described in more detail below.

Both computing devices 10 and 12 may include an eye-attracting program 46 that may be stored in mass storage 40. Eye-attracting program 46 may be loaded into memory 48 and executed by processor 52 to perform one or more of the methods and processes described in greater detail below.

Computing device 10 and computing device 12 may receive gaze tracking data 50 from gaze tracking system 54. In various examples, gaze tracking system 54 may be located in display device 14, in HMD device 18, or in a common housing with any other suitable type or form of display device (including, but not limited to, those example devices with integrated displays discussed above). In other examples, gaze tracking system 54 and computing device 10 may be integrated into a common housing that does not include an integrated display, such as a head-mounted or other wearable device, or any other suitable type or form of computing device that does not include an integrated display (including, but not limited to, those example devices without integrated displays discussed above).

1 , the example display device 14 may include a display system 58 for presenting one or more visual elements to a second viewer 62. As described in greater detail below, a gaze attraction program 46 may utilize gaze tracking data 50 from a gaze tracking system 54 to attract the viewer's gaze via guidance elements displayed by the display device 14, HMD 18, or other display device.

Referring now to Figures 2-5 , a description of an example use case will now be provided. Figure 2 is a schematic diagram of several viewers in a room 200 interacting with a computing device and a display device that uses gaze tracking data from a gaze tracking system to attract the viewers' gaze. In one example, viewer Alex 202 is viewing a movie 206 displayed on a wall-mounted display 210. In this example, the wall-mounted display 210 is communicatively coupled to a set-top box 214, which includes the gaze tracking system 54 and a computing device including the gaze attracting program 46.

Referring now to FIG3 , in one example, the producers of movie 206 may desire to draw a viewer's attention to coffee shop 302 depicted in a scene from the movie. To attract the viewer's attention, gaze attractor program 46 may be configured to control display device 210 to display a guiding element. In this example, the guiding element includes bird 306, which may be a computer-generated image, added to the movie scene. Gaze attractor program 46 may use gaze tracking data from gaze tracking system 54 to monitor the gaze location of viewer Alex 202 on wall-mounted display 210. For example, and as shown in FIG3 , gaze tracking system 54 may use gaze tracking data 50 to determine that viewer Alex 202 is currently gazing at gaze location 308.

Bird 306 can be displayed as moving along a calculated motion path 310 leading to coffee house 302. Furthermore, and to attract the attention of viewer Alex 202, calculated motion path 310 can pass through a predetermined area 314 proximate to viewer Alex's 202 gaze location 308. Additionally, and to minimize interruptions or distractions to viewer Alex's 202 viewing experience, motion path 310 can be calculated in a manner that makes the movement of bird 306 appear natural and realistic. Advantageously, utilizing such a motion path allows viewer Alex 202 to continue watching and enjoying movie 206 without feeling that his attention is being manipulated or intentionally diverted.

In some examples, computer-generated bird 306 can be displayed as moving along calculated dynamic path 310 according to computer-generated image movement rules that govern the movement of computer-generated images rendered in real time in movie 206. It should also be understood that this example of a guidance element in the form of bird 306 is provided for illustrative purposes, and that many other types, forms, and examples of guidance elements may be utilized and are within the scope of this disclosure. For example, and with respect to movie 206, other computer-generated guidance elements that may be utilized include, but are not limited to, floating leaves, people, cars, or any other suitable guidance elements.

In the example shown in FIG3 , the predetermined area 314 proximate to the gaze location 308 of viewer Alex 202 is a circle having a radius R that is concentric with the gaze location. The radius R can be determined in any suitable manner and can be based on one or more of the following criteria, such as the distance from the wall-mounted display 210 to viewer Alex 202, the size of the wall-mounted display 210, the size of one or more elements displayed on the wall-mounted display, the accuracy of the gaze tracking system 54, or any other suitable criteria. In various examples, the radius R can have a length of approximately 0.5 mm, 1.0 mm, 5.0 mm, 10.0 mm, 50.0 mm, 100.0 mm, or any other suitable distance. It should be understood that any other suitable shape and/or configuration of the predetermined area proximate to the viewer's gaze location can also be used and is within the scope of the present disclosure.

3 , as bird 306 travels along calculated dynamic path 310, gaze attraction program 46 can determine whether viewer Alex 202's gaze follows the bird's flight. In one example, after bird 306 passes gaze location 308, gaze attraction program 46 can determine whether viewer Alex 202's updated gaze location 308' is within a predetermined deviation threshold of bird 306.

3 , updated gaze position 308′ of viewer Alex 202 may be determined to be within a predetermined deviation threshold of bird 306 when the updated gaze position overlaps at least a portion of bird 306. In another example, updated gaze position 308′ of viewer Alex 202 may be determined to be within a predetermined deviation threshold of bird 306 when the gaze position is within a predetermined distance of the bird (without necessarily overlapping the bird).

3 , the predetermined deviation threshold 318 may include a circle having a radius T that is concentric with the updated gaze position 308 ′. As discussed above with respect to radius R, the radius T may be determined in any suitable manner and may be based on one or more of the following criteria, such as the distance from the wall-mounted display 210 to the viewer Alex 202, the size of the wall-mounted display, the size of the bird 306 and/or one or more other elements displayed on the wall-mounted display, the accuracy of the gaze tracking system 54, or any other suitable criteria. In various examples, the radius T may have a length of approximately 0.5 mm, 1.0 mm, 5.0 mm, 10.0 mm, 50.0 mm, 100.0 mm, or any other suitable length. It should also be understood that any other suitable shape and/or configuration of the predetermined area proximate the viewer's gaze position may also be used and is within the scope of the present disclosure.

If the updated gaze position 308' is within the predetermined deviation threshold of the bird, the gaze attraction program 46 can control the wall-mounted display 210 to continue to display the bird 306 moving along the calculated dynamic path 310 toward the coffee house 302. Advantageously and in this manner, the gaze of the viewer Alex 202 can be directed toward the coffee house 302, thereby increasing Alex's perception of the coffee house. Additionally, and as described in more detail below, in some examples, the gaze attraction program 46 can also determine that the gaze position of the viewer Alex overlaps with the coffee house 302, and in response, can allocate advertising consumption charges to the coffee house.

In another example, gaze attraction program 46 may determine that the gaze location of viewer Alex 202 has deviated from bird 306 by at least a predetermined deviation threshold 318. Alternatively, and with respect to the example of FIG3 , gaze attraction program 46 may determine that the gaze location of viewer Alex 202 is outside the circle indicated at 318. For example, gaze attraction program 46 may determine that viewer Alex 202 has returned his gaze to gaze location 308. In this example, gaze attraction program 46 may control wall-mounted display 210 to stop displaying bird 306 moving along calculated dynamic path 310 toward coffee house 302.

In another example, the gaze attraction program 46 can monitor and track the gaze trajectory 312 of the gaze position of the viewer Alex. In this example, when the gaze trajectory 312 is within the path deviation threshold 316 of the calculated dynamic path 310, it can be determined that the gaze position of the viewer Alex 202 is within the predetermined deviation threshold of the bird 306. For example, after the bird 306 passes through the predetermined area 314 near the gaze position 308 of Alex, the viewer Alex 202 can initially use his gaze to follow the flight of the bird so that his gaze trajectory 312 is within the path deviation threshold 316.

As shown in FIG3 , in this example, path deviation threshold 316 is the distance between gaze track 312 and calculated dynamic path 310 at a given moment. In various examples, path deviation threshold 316 can have a length of approximately 0.5 mm, 1.0 mm, 5.0 mm, 10.0 mm, 50.0 mm, 100.0 mm, or any other suitable length. It should also be understood that any other suitable comparison between gaze track 312 and calculated dynamic path 310 can be used to determine whether viewer Alex's gaze is following bird 306, and any other suitable comparison is within the scope of the present disclosure.

As discussed above, if the gaze attraction program 46 determines that the gaze trajectory 312 is within the path deviation threshold 316, the program can control the wall-mounted display 210 to continue displaying the bird 306 moving along the calculated dynamic path 310 toward the coffee house 302. In another example, the gaze attraction program 46 can determine that the gaze trajectory 312 of the viewer Alex 202 deviates from the calculated dynamic path 310 by at least the path deviation threshold 316. For example, the gaze attraction program 46 can determine that the viewer Alex 202 shifts his gaze to the gaze location 308" indicated by the gaze trajectory 312. In this example, the gaze attraction program 46 can control the wall-mounted display 210 to stop displaying the bird 306 moving along the calculated dynamic path 310 toward the coffee house 302.

In one example, the eye-attracting program 46 can stop displaying the bird 306 moving along the calculated dynamic path 310 by shifting the bird's movement to move along an alternative path 322 that does not lead to the coffee shop 302. In this way, the eye-attracting program 46 can avoid displaying all guidance elements moving toward the target object, even when the viewer is not looking at the guidance elements. In other examples, the eye-attracting program 46 can stop displaying the bird 306 moving along the calculated dynamic path by no longer displaying the guidance elements. For example, the eye-attracting program 46 can cause the guidance elements to disappear from the display.

In another example, the calculated dynamic path 310 may be programmatically adjusted based on a change in the gaze position of the viewer Alex 202. For example, when the viewer Alex 202 changes his gaze to the updated gaze position 308″ at the bystander 320, the gaze attraction program 46 may programmatically adjust the calculated dynamic path to pass through the interior of the predetermined area adjacent to the updated gaze position 308″ and then continue to the coffee house 302.

In another example, and referring now to Figures 1 and 2, viewer Mary 230 may be playing computer game 234 on her tablet computer 238. Tablet computer 238 may include display 242, gaze tracking system 54, and gaze attraction program 46. Referring now to Figure 4, in one example, computer game 234 includes baseball game 400, which includes a player character in the form of batter 404, which can be controlled by viewer Mary 230, and non-player characters in the form of pitcher 408 and outfielders 412 and 414, which move according to the non-player character movement rules of the baseball game. Baseball game 400 may also include at least one object that moves according to the object movement rules of the baseball game. In this example, the object may include baseball 420.

In this example, the guidance elements may include a pitcher 408, an outfielder 412, and/or a baseball 420. For example, the eye-attraction program 46 may control the display 242 of the tablet computer 238 to display the outfielder 412 moving along a calculated dynamic path 430 according to the non-player character movement rules. The calculated dynamic path 430 leads to an advertisement 434 located on the outfield wall and encouraging the viewer Mary 230 to "eat at Cafe A." The eye-attention location 450 of the viewer Mary 230 on the display may be determined and monitored as described above. The movement of the outfielder 412 along the calculated dynamic path 430 may also be controlled as described above.

In some examples, Cafe A may pay a promotional fee to have its advertisement 434 displayed during baseball game 400. In one example, if eye-attraction program 46 determines that viewer Mary 230's eye position 450 overlaps with advertisement 434, the advertising fee may be allocated to Cafe A. Advantageously, in this manner, advertisers pay impression-based advertising fees that are directly related to viewers' actual impressions of the advertiser's advertisement.

In another example, the promotional fee paid by Cafe A for its advertisement 434 may be based at least in part on the time period during which a viewer gazed at the advertisement. For example, if viewer Mary 230's gaze position 450 overlapped with advertisement 434 for less than a predetermined time period, a first advertising consumption fee may be assigned to Cafe A. If viewer Mary 230's gaze position 450 overlapped with advertisement 434 for at least a predetermined time period, a second advertising consumption fee that is higher than the first advertising consumption fee may be assigned to Cafe A. In some examples, the predetermined time period may be 0.5 seconds, 1.0 seconds, 2.0 seconds, 5.0 seconds, or any other suitable time period. Similarly, any suitable amount may be utilized for the first and second advertising consumption fees.

5 , in some examples, a target object may include an advertisement displayed on a web page. For example, viewer Mary 230 may be viewing a web page on her tablet computer 238 including a touch-sensitive display 242. A guidance element in the form of a striped comet 504 may be displayed moving along a calculated dynamic path 506 that passes within a predetermined area 508 proximate to viewer Mary's 230 viewing location 510. The calculated dynamic path may lead to a target object in the form of a selectable advertisement 512 from Cafe A that announces a "Buy One Pizza, Get One Free" promotion.

In one example, viewer Mary 230 may provide viewer input associated with selectable advertisement 512 by touching the screen of display 242. The location of her touch selection may be interpreted by the touch-sensitive screen as being located at touch location point 514 that overlaps a portion of advertisement 512. Thus, this viewer input selecting advertisement 512 may trigger an advertising consumption charge that may be allocated to Cafe A.

However, in some examples, viewer Mary 230 may not have intended to select advertisement 512. For example, viewer Mary 230 may have larger-than-average fingers and may have intended to select "click here" selectable button 520. The touch detection system of touch-sensitive display 242 may have incorrectly interpreted her intended touch location as being at point 514. To address this possibility, when viewer Mary 230's touch input is received and associated with advertisement 512, gaze attraction program 46 may determine whether viewer Mary's 230 gaze location overlaps with the advertisement.

In one example, if viewer Mary 230's gaze position 524 overlaps with advertisement 512 when viewer input is received, advertising consumption charges are allocated to Coffee Shop A. On the other hand, if viewer Mary 230's gaze position 530 does not overlap with advertisement 512 when viewer input is received, advertising consumption charges are canceled. Advantageously, in this manner, inadvertent or unintentional selections of selectable advertisements or other components on a webpage can be identified, and corresponding erroneous advertising consumption charges and/or other unintentional operations can be avoided.

2 , viewer Wally 250 wears an HMD device in the form of a pair of glasses 254. Viewer Wally 250, via the HMD glasses 254, participates in a mixed reality experience that includes a holographic wizard 260 and a guiding element in the form of a floating ball 264, both displayed by the glasses.

In one example, the developer of a mixed reality experience may wish to draw the attention of viewer Wally to a holographic coffee shop advertisement 270 displayed in room 20. As described above, the gaze attractor 46 of the HMD glasses 254 may be configured to display a guiding element. In this example, the guiding element includes a floating ball 264. The gaze attractor 46 may use gaze tracking data from the gaze tracking system 54 of the HMD glasses 254 to monitor the gaze position of viewer Wally 250.

The floating ball 264 can be displayed as moving along a calculated dynamic path 274 toward the coffee shop advertisement 270. As described above, the calculated dynamic path 274 can pass through a predetermined area 278 adjacent to the gaze position 282 of the viewer Wally 250. As the floating ball moves along the calculated dynamic path 274, the gaze attraction program 46 can determine whether the gaze position of the viewer Wally 250 is within a predetermined deviation threshold from the ball. In one example, after the floating ball 264 passes the gaze position 282, the gaze attraction program 46 determines that the updated gaze position 282′ of the viewer Wally 250 is at the ball and within the predetermined deviation threshold. Consequently, the gaze attraction program 46 can control the HMD glasses 254 to continue displaying the floating ball 264 moving along the calculated dynamic path 274 toward the coffee shop advertisement 270.

In other examples and as described above, gaze attraction program 46 may also determine that another gaze location 282" of viewer Wally 250 overlaps with coffee house advertisement 270 and, in response, may assign advertising consumption charges to the coffee house.

In another example, the gaze attraction program 46 may determine that the gaze position of the viewer Wally 250 has deviated from the floating ball 264 by at least a predetermined deviation threshold. In this example, the gaze attraction program 46 may control the HMD glasses 254 to stop displaying the floating ball 264 moving along the calculated dynamic path 274 toward the coffee shop advertisement 270. For example, the HMD glasses 254 may offset the floating ball 264 to move along an alternative path 286 that does not overlap with the advertisement 270. In other examples, the HMD glasses 254 may stop displaying the floating ball 264.

Referring now to FIG6 , an example of an HMD device 600 in the form of a pair of wearable glasses with a transparent display is provided. It should be understood that in other examples, the HMD device 600 can take other suitable forms, wherein a transparent, translucent, and/or non-transparent display is supported in front of one or both eyes of a user. It should be understood that the HMD device shown in FIG1 and FIG2 can take the form of the HMD device 600 described in detail below, or any other suitable HMD device.

The HMD device 600 includes a display system 602 and a see-through or transparent display 604 that enables images (e.g., holographic objects) to be delivered to the eyes of a wearer of the HMD device. The transparent display 604 can be configured to visually augment the appearance of the real world, the physical environment, to the wearer viewing the physical environment through the transparent display. For example, the appearance of the physical environment can be augmented by graphical content (e.g., one or more pixels, each having a corresponding color and brightness) presented via the transparent display 604 to create an augmented reality environment.

The transparent display 604 can also be configured to enable the wearer of the HMD device to view physical, real-world objects in the physical environment by displaying one or more partially transparent pixels representing virtual objects. As shown in FIG6 , in one example, the transparent display 604 can include an image-producing element (e.g., a see-through organic light-emitting diode (OLED) display) located within the lens 606. As another example, the transparent display 604 can include a light modulator on the edge of the lens 606. In this example, the lens 606 can act as a light guide for transmitting light from the light modulator to the wearer's eye. Such a light guide can enable the wearer to perceive 3D holographic images located within the physical environment the wearer is viewing, while also allowing the wearer to view physical objects in the physical environment, thereby creating an augmented reality environment.

HMD device 600 may also include various sensors and related systems. For example, HMD device 600 may include gaze tracking system 608, which includes one or more image sensors configured to obtain image data in the form of gaze tracking data from the wearer's eyes. If the wearer has consented to the acquisition and use of this data, gaze tracking system 608 may use this information to track the position and movement of the wearer's eyes.

In one example, the gaze tracking system 608 includes a gaze detection subsystem configured to detect the direction of each eye of the wearer. The gaze detection subsystem can be configured to determine the gaze direction of each of the wearer's eyes in any manner. For example, the gaze detection subsystem can include one or more light sources (e.g., infrared light sources) configured to cause a flickering light to reflect from the cornea of each of the wearer's eyes. Then, one or more image sensors can be configured to capture images of the wearer's eyes.

The optical axis of each eye can be determined using images of glints and pupils determined from image data collected from the image sensor. This information can then be used by the gaze tracking system 608 to determine the direction of the wearer's gaze. The gaze tracking system 608 can additionally or alternatively determine which physical or virtual object the wearer is looking at, and where on the physical or virtual object the wearer is looking. This gaze tracking data can then be provided to the HMD device 600.

It should also be understood that the gaze tracking system 608 can have any suitable number and arrangement of light sources and image sensors. For example, and referring to FIG6 , the gaze tracking system 608 of the HMD device 600 can utilize at least one inward-facing sensor 610 .

The HMD device 600 may also include a sensor system that receives physical environment data from the physical environment. For example, the HMD device 600 may also include a head tracking system 612 that utilizes one or more gesture sensors (e.g., gesture sensor 614 on the HMD device 600) to capture head gesture data and thereby support position tracking, direction/location and orientation sensing, and/or motion detection of the wearer's head.

In one example, the head tracking system 612 may include an inertial measurement unit (IMU) configured as a three-axis or three-degree-of-freedom position sensor system. For example, the example position sensor system may include three gyroscopes to indicate or measure changes in the orientation of the HMD device 600 in 3D space about three orthogonal axes (e.g., x, y, and z, or roll, pitch, and yaw). In some examples, the orientation derived from the sensor signals of the IMU can be used to display one or more virtual objects with body-locked positions via the transparent display 604, where the position of each virtual object appears to be fixed relative to the wearer of the see-through display, and the position of each virtual object appears to be movable relative to real objects in the physical environment.

In another example, the head tracking system 612 may include an IMU configured as a six-axis or six-degree-of-freedom position sensor system. For example, this example position sensor system may include three accelerometers and three gyroscopes to indicate or measure changes in the position of the HMD device 600 along three orthogonal axes, as well as changes in the orientation of the device about the three orthogonal axes.

The head tracking system 612 may also support other suitable positioning technologies, such as GPS or other global positioning systems. Furthermore, while specific examples of position sensor systems have been described, it should be understood that any other suitable position sensor system may be used. For example, head pose and/or motion data may be determined based on sensor information from any combination of sensors mounted on and/or external to the wearer, including but not limited to any number of gyroscopes, accelerometers, inertial measurement units, GPS devices, barometers, magnetometers, cameras (e.g., visible light cameras, infrared cameras, time-of-flight depth cameras, structured light depth cameras, etc.), communication devices (e.g., WiFi antennas/interfaces), etc.

In some examples, the HMD device 600 may also include an optical sensor system that captures image data using one or more outward-facing sensors (e.g., optical sensor 616 on the HMD device 600). The outward-facing sensors can detect motion within their field of view, such as gesture-based input or other motion performed by the wearer or by people or physical objects within the field of view. The outward-facing sensors can also capture 2D image information and depth information from the physical environment and physical objects within the environment. For example, the outward-facing sensors can include depth cameras, visible light cameras, infrared cameras, and/or position tracking cameras.

The optical sensor system can include a depth tracking system that generates depth tracking data via one or more depth cameras. In one example, each depth camera can include a left camera and a right camera of a stereo vision system. Time-resolved images from one or more of these depth cameras can be registered with each other and/or with images from another optical sensor (e.g., a visible spectrum camera) and can be combined to produce depth-resolved video.

In other examples, a structured light depth camera can be configured to project structured infrared illumination and image the illumination reflected from a scene, where the illumination is projected onto the scene. A depth map of the scene can be constructed based on the spaces between adjacent features in various areas of the imaged scene. In other examples, the depth camera can take the form of a time-of-flight depth camera configured to project pulsed infrared illumination onto a scene and detect the illumination reflected from the scene. For example, the illumination can be provided by infrared light source 618. It should be understood that any other suitable depth camera can be used within the scope of the present disclosure.

The outward-facing sensors can capture images of the physical environment in which the wearer of the HMD device is located. With respect to the HMD device 600, in one example, the augmented reality display program can include a 3D modeling system that models the environment surrounding the wearer of the HMD device, the 3D modeling system using such captured images to generate a virtual environment. In some examples, the optical sensor 616 can cooperate with the IMU to determine the position and orientation of the HMD device 600 in six degrees of freedom. Such position and orientation information can be used to display one or more virtual objects with world-locked positions via the transparent display 604, wherein the position of each virtual object appears fixed relative to real-world objects that can be seen through the transparent display, and the position of each virtual object appears movable relative to the wearer of the see-through display.

The HMD device 600 also includes a microphone system that includes one or more microphones that capture audio data, such as microphone 620. In other examples, the audio can be presented to the wearer via one or more speakers (e.g., speaker 622 on the HMD device 600).

The HMD device 600 may also include a controller, such as controller 624. The controller 624 may include a logic subsystem and a storage subsystem, as discussed in more detail below with respect to FIG8 , that communicate with various sensors and systems of the HMD device 600. In one example, the storage subsystem may include instructions that may be executed by the logic subsystem to receive signal inputs from sensors, determine a posture of the HMD device 600, and adjust display properties of content displayed via the transparent display 604.

7A and 7B illustrate a flow chart of a method 700 for attracting the eye of a viewer of a display device according to an implementation of the present disclosure. The following description of method 700 is provided with respect to the software components and hardware components described above and shown in FIGs. 1-6. It should be understood that method 700 may also be performed in other contexts using other suitable hardware components and software components.

7A , at 704, method 700 may include controlling a display device to display a target object. At 708, method 700 may include monitoring a viewer's gaze position using gaze tracking data from a gaze tracking system. At 712, method 700 may include controlling a display device to display a guidance element moving along a calculated dynamic path that passes through a predetermined area adjacent to the viewer's gaze position and leads to the target object.

At 716, method 700 may include using the gaze tracking data to determine whether the viewer's gaze position is within a predetermined deviation threshold of the guidance element. At 720, method 700 may include controlling the display device to continue displaying the guidance element moving along the calculated dynamic guidance path toward the target object if the viewer's gaze position is within the predetermined deviation threshold of the guidance element. At 724, method 700 may include controlling the display device to discontinue displaying the guidance element moving along the calculated dynamic guidance path toward the target object if the viewer's gaze position deviates from the guidance element by at least the predetermined deviation threshold.

At 728, not continuing to display the guidance element moving along the calculated dynamic guidance path toward the target object may include causing the guidance element to deviate from the calculated dynamic path. At 732, not continuing to display the guidance element moving along the calculated dynamic guidance path toward the target object may include stopping displaying the guidance element. At 736, the predetermined deviation threshold includes a distance to the guidance element. At 738, the predetermined deviation threshold includes a path deviation threshold from the calculated dynamic path, and method 700 may include determining whether the viewer's gaze trajectory deviates from the calculated dynamic path by at least the path deviation threshold. If the viewer's gaze trajectory deviates from the calculated dynamic path by at least the path deviation threshold, the display may be controlled to not continue to display the guidance element moving along the calculated dynamic guidance path toward the target object, as discussed above.

7B , at 740, when the method 700 is executed while the viewer is playing a computer game having at least one non-player character that moves according to non-player character movement rules and at least one object that moves according to object movement rules, controlling the display device to display the guidance element may further include causing the guidance element to move along a calculated dynamic path according to the non-player character movement rules or the object movement rules. At 744, when the method is executed while the user is watching a movie, controlling the display device to display the guidance element may further include causing the guidance element to move along a calculated dynamic path according to movement rules for computer-generated images rendered in real time in the movie.

At 748, and if the target object includes an advertisement from an advertiser displayed on the display device, method 700 may further include allocating an advertising consumption fee to the advertiser if the viewer's gaze position overlaps with the advertisement. At 752, and if the advertising consumption fee is a first advertising consumption fee, method 700 may further include allocating a second advertising consumption fee that is greater than the first advertising consumption fee if the viewer's gaze position overlaps with the advertiser for at least a predetermined period of time.

At 756, in the event that the target object includes an advertisement from an advertiser displayed on a webpage, method 700 may include: receiving a viewer input associated with the advertisement, wherein the viewer input triggers an advertisement consumption charge. At 760, if the viewer's gaze position overlaps with the advertisement on the webpage when the viewer input is received, method 700 may include allocating the advertisement consumption charge to the advertiser. At 764, if the viewer's gaze does not overlap with the advertisement on the webpage when the viewer input is received, method 700 may include canceling the advertisement consumption charge. At 768, the display device may include a wearable display device that includes an eye tracking system.

It should be understood that method 700 is provided as an example and is not meant to be limiting. Therefore, it should be understood that method 700 may include additional and/or alternative steps to those steps shown in Figures 7A and 7B. Furthermore, it should be understood that method 700 may be performed in any suitable order. Furthermore, it should be understood that one or more steps may be omitted from method 700 without departing from the scope of this disclosure.

FIG8 schematically illustrates a non-limiting example of a computing system 800 that can perform one or more methods and processes described above. Computing device 10 and computing device 12 can take the form of one or more aspects of computing system 800 or include one or more aspects of computing system 800. Computing system 800 is shown in simplified form. It should be understood that any computer architecture can be used virtually without departing from the scope of this disclosure. In different examples, computing system 800 can take the following forms, including: mainframe computers, server computers, desktop computers, tablet computers, home entertainment computers, network computing equipment, tablet computers, notebook computers, smart phones or other mobile computing devices, mobile communication devices, gaming devices, etc.

As shown in FIG8 , computing system 800 includes a logic subsystem 804 and a storage subsystem 808. Computing system 800 may optionally include a sensor subsystem 812, a display subsystem 816, a communication subsystem 820, an input subsystem 822, and/or other subsystems and components not shown in FIG8 . Computing system 800 may also include computer-readable media, wherein the computer-readable media includes computer-readable storage media and computer-readable communication media. Computing system 800 may also optionally include other user input devices, such as a keyboard, a mouse, a game controller, and/or a touch screen. In addition, in some embodiments, the methods and processes described herein may be implemented as computer applications, computer services, computer APIs, computer libraries, and/or other computer program products in a computing system including one or more computers.

The logic subsystem 804 may include one or more physical devices configured to execute one or more instructions. For example, the logic subsystem 804 may be configured to execute one or more instructions as part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical structures. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more devices, or otherwise achieve a desired result.

The logic subsystem 804 may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processors of the logic subsystem may be single-core or multi-core, and the programs executed thereon may be configured for parallel processing or distributed processing. The logic subsystem may optionally include individual components distributed across two or more devices, which may be remotely located and/or configured for coordinated processing. One or more aspects of the logic subsystem may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

The storage subsystem 808 may include one or more physical, persistent devices configured to store data and/or instructions that can be executed by the logic subsystem 804 to implement the methods and processes described herein. When implementing such methods and processes, the state of the storage subsystem 808 may be transformed (e.g., to store different data).

The storage subsystem 808 may include removable media and/or built-in devices. The storage subsystem 808 may include optical storage devices (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory devices (e.g., RAM, EPROM, EEPROM, etc.), and/or magnetic storage devices (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), etc. The storage subsystem 808 may include devices having one or more of the following characteristics: volatility, non-volatility, dynamic, static, read/write, read-only, random access, sequential access, location addressable, file addressable, and content addressable.

In some examples, aspects of the logic subsystem 804 and the storage subsystem 808 can be integrated into one or more common devices, wherein the functionality described herein can be at least partially implemented by the one or more common devices. Such hardware logic components may include, for example, field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC/ASICs), program and application specific standard products (PSSP/ASSPs), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

8 also illustrates aspects of the storage subsystem 808 in the form of a removable computer-readable storage medium 824 that can be used to store data and/or instructions executable to implement the methods and processes described herein. The removable computer-readable storage medium 824 can take the form of a CD, DVD, HD-DVD, Blu-ray disc, EEPROM, and/or floppy disk, among others.

It should be understood that the storage subsystem 808 includes one or more physical, persistent devices. In contrast, in some implementations, aspects of the instructions described herein may be propagated in a transient manner via pure signals (e.g., electromagnetic signals, optical signals, etc.) that are not retained by a physical device for at least a limited duration. Furthermore, data and/or other forms of information related to the present disclosure may be propagated via pure signals via computer-readable communication media.

When included, the sensor subsystem 812 may include one or more sensors configured to sense various physical phenomena as described above (e.g., visible light, infrared light, sound, acceleration, orientation, position, etc.). The sensor subsystem 812 may be configured to provide sensor data to the logic subsystem, for example. Such data may include gaze tracking information, image information, ambient light information, depth information, audio information, position information, motion information, user position information, and/or any other suitable sensor data that may be used to perform the methods and processes described above.

When included, the display subsystem 816 can be used to present a visual representation of the data stored by the storage subsystem 808. As the methods and processes discussed above change the data stored by the storage subsystem 808 and thus transform the state of the storage subsystem, the state of the display subsystem 816 can also be transformed to visually represent the changes in the underlying data. The display subsystem 816 can include one or more display devices utilizing virtually any type of technology. Such a display device can be combined with the logic subsystem 804 and/or the storage subsystem 808 in a shared housing, or such a display device can be a peripheral display device.

When included, the communication subsystem 820 can be configured to communicatively couple the computing system 800 with one or more networks and/or one or more other computing devices. The communication subsystem 820 may include wired and/or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem 820 can be configured to communicate via a wireless telephone network, a wireless local area network, a wired local area network, a wireless wide area network, a wired wide area network, or the like. In some embodiments, the communication subsystem can allow the computing system 800 to send messages to and/or receive messages from other devices via a network such as the Internet.

When included, the input subsystem 822 may include or interface with one or more sensors or user input devices, such as a game controller, a gesture input detection device, a speech recognizer, an inertial measurement unit, a keyboard, a mouse, or a touch screen. In some embodiments, the input subsystem 822 may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the transduction and/or processing of input actions may be processed on-board and/or off-board. Example NUI components may include microphones for voice and/or speech recognition; infrared, color, stereo and/or depth cameras for machine vision and/or gesture recognition; head trackers, eye trackers, accelerometers, and/or gyroscopes for motion detection and/or intent recognition; and electric field sensing components for assessing brain activity.

The term "program" may be used to describe aspects of computing devices 10 and 12 that are implemented to perform one or more specific functions. In some cases, such a program may be instantiated via logic subsystem 804 executing instructions stored by storage subsystem 808. It should be understood that different programs may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same program may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The term "program" is intended to include both individual and grouped executable files, data files, libraries, drivers, scripts, database records, and the like.

It should be understood that the configuration and/or method described in this article are exemplary in nature, and these specific embodiments or examples should not be understood as limiting, because multiple variations are possible. The specific routine or method described in this article can represent one or more of any number of processing strategies. Therefore, the various behaviors shown and/or described can be performed in the order shown and/or described, performed in other orders, performed in parallel, or omitted in some cases. Equally, the order of the above process can be changed.

The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems, and configurations, and other features, functions, acts, and/or properties disclosed herein, and any and all equivalents thereof.

Claims (10)

1. A method for attracting the gaze of a viewer of a display device, the method comprising: Control the display device to display the target object; The viewer's gaze position is monitored using gaze tracking data from the gaze tracking system; The display device is controlled to show guide elements moving along a calculated dynamic path that passes through a predetermined area near the viewer's line of sight and leads to the target object; The eye-tracking data is used to determine whether the viewer's gaze position is within a predetermined deviation threshold of the guiding element; If the viewer's gaze position is within a predetermined deviation threshold of the guide element, the display device is controlled to continue displaying the guide element moving towards the target object along the calculated dynamic path; and If the viewer's gaze position deviates from the guide element by at least a predetermined deviation threshold, the display device is controlled to stop displaying the guide element moving toward the target object along the calculated dynamic path. 2. The method according to claim 1, wherein controlling the display device not to continue displaying the guide element along the calculated dynamic path further comprises: causing the guide element to deviate from the calculated dynamic path or stopping the display of the guide element. 3. The method according to claim 1, wherein the predetermined deviation threshold includes the distance to the guiding element. 4. The method according to claim 1, wherein using the gaze tracking data to determine whether the viewer's gaze position is within a predetermined deviation threshold of the guiding element further comprises: determining whether the viewer's gaze trajectory is within a path deviation threshold of the calculated dynamic path. 5. The method of claim 1, wherein the method is performed when the viewer plays a computer game having at least one non-player character moving according to non-player character movement rules and at least one object moving according to object movement rules, and controlling the display device to display the guide element further includes causing the guide element to move along the calculated dynamic path according to the non-player character movement rules or the object movement rules. 6. A computing device for attracting the gaze of a viewer of a display device, the computing device comprising: A gaze attraction program executed by the processor of the computing device, the gaze attraction program being configured to: Control the display device to display the target object; The viewer's gaze position is monitored using gaze tracking data from the gaze tracking system; The display device is controlled to show guide elements moving along a calculated dynamic path that passes through a predetermined area near the viewer's line of sight and leads to the target object; The eye-tracking data is used to determine whether the viewer's eye trajectory is within the path deviation threshold of the calculated dynamic path; If the viewer's gaze position is within the path deviation threshold of the calculated dynamic path, the display device is controlled to continue displaying the guide element moving towards the target object along the calculated dynamic path; and If the viewer's gaze position deviates from the calculated dynamic path by at least the path deviation threshold, the display device is controlled to stop the guide element from moving along the calculated dynamic path. 7. The computing device of claim 6, wherein the target object includes an advertisement from an advertiser displayed on the display device, and the gaze-attracting procedure is further configured to allocate an advertising consumption fee to the advertiser if the viewer's gaze position overlaps with the advertisement. 8. The computing device of claim 7, wherein the advertising consumption fee is a first advertising consumption fee, and the gaze attraction procedure is further configured to: allocate a second advertising consumption fee greater than the first advertising consumption fee if the viewer's gaze position overlaps with the advertisement for at least a predetermined time period. 9. The computing device of claim 6, wherein the target object includes an advertisement from an advertiser displayed on a webpage, and the eye-attracting procedure is further configured to: Receive viewer input associated with the advertisement displayed on the webpage, the viewer input triggering an ad consumption charge; If the viewer's gaze position overlaps with the advertisement on the webpage when the viewer's input is received, then the advertising fee is allocated to the advertiser; and If the viewer's gaze position does not overlap with the advertisement on the webpage when the viewer's input is received, the advertisement consumption fee is cancelled. 10. The computing device of claim 6, wherein the computing device is integrated into a wearable display device including the eye-tracking system.