HK1182485B - Making static printed content dynamic with virtual data - Google Patents

Description

Using dummy data to render static print content dynamic

Technical Field

The present invention relates to rendering static print content dynamic with dummy data.

Background

Static printed material can be considered a form of read-only memory that requires no power and stores its data in a form visible to the human eye. Kraft paper documents survive more than a thousand years to date. In addition, the data is typically presented in a format that is comfortably readable by the human eye, which presents the printed text against a contrasting physical background of white or another contrasting color of paper. The physical nature of printed material allows users to physically sift through their data for "content of interest," for example, by paging through pages of a magazine and viewing pictorial or attractive headings in the magazine. One can keep his position in the physical book and flip back a stack of pages (the user's brain knows that this is near the correct chapter) to re-read a chapter and look back at the position the fingers hold. Of course, physical books, periodicals, and papers also have their disadvantages due to the permanent placement of information on the pages.

Disclosure of Invention

Mixed reality is a technology that allows virtual images to be mixed with real-world views. A user may wear a see-through, head-mounted, mixed reality display device to view a mixed image of real and virtual objects displayed in the user's field of view. Such Head Mounted Display (HMD) devices may update and, in some cases, restore data embodied in static printed material. In other words, other embodiments of a physical book, magazine, or static printed material are a form of dynamic memory in the sense that: the presence of printed paper, printed cards, or other printed media may vary.

The present technology provides an embodiment of a method for rendering static print content dynamic using a see-through, near-eye, mixed reality display device. The method comprises the following steps: the method includes identifying a print content item in a field of view of a see-through, near-eye, mixed reality display device, and identifying a user selection of the print content selection within the print content item based on a physical action user input. A user-requested task selected for the print content is determined based on the physical action user input and the task is executed. Virtual data related to the print content selection is displayed according to the requested job.

The present technology provides an embodiment of a system for rendering static printed material into a dynamic see-through, near-eye, mixed reality display device system. The system includes a respective see-through display for each eye positioned by the support structure. One example of a support structure is a frame. At least one outward facing camera for capturing image data in a field of view of the respective see-through display is positioned on the support structure. One or more software controlled processors are communicatively coupled to the at least one outward facing camera to receive image data and to at least one image generation unit optically coupled to the respective see-through display.

The one or more software controlled processors identify a user selection of a print content selection based on the physical action user input and the image data. For example, a page of a book may be in view of at least one outward facing camera. Physical action user input is an action performed by a user using a body part and captured by a Natural User Interface (NUI). The physical action provides data or commands that indicate the operation of the application. Some examples of physical actions are eye gaze and gestures.

One or more software controlled processors are communicatively coupled to a search engine having access to a data store including content, layout and virtual data for a work and print content items embodying the work. The one or more software controlled processors identify, based on formulating one or more queries based on the image data, a print content item that includes the print content selection and a media independent version of a work that includes the print content selection. The one or more queries are sent to the search engine.

The one or more software controlled processors cause the at least one communicatively coupled image generation unit to display, through each optically coupled respective see-through display, virtual data associated with the print content selection or the media independent version of the print content selection.

The technology provides embodiments of one or more processor-readable storage devices having instructions encoded thereon that cause one or more processors to perform a method for improving readability of static printed material with virtual data using a see-through, near-eye, mixed reality display device system. The method includes identifying printed material in a field of view of a see-through, near-eye, mixed reality display device system based on image data captured by one or more outward-facing cameras of the display device system, and determining whether the printed material located in the field of view satisfies readability criteria. In response to the readability criteria not being met, displaying a virtual version of the printed material in the field of view that meets the readability criteria. Further, if the printed material is still within the field of view of the one or more outward facing cameras, taking an action in response to a physical action user input to one or both of the virtual version of the printed material or the printed material.

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

Drawings

FIG. 1A is a block diagram depicting example components of one embodiment of a see-through, mixed reality display device system.

Fig. 1B is a block diagram depicting example components of another embodiment of a see-through, mixed reality display device system.

Fig. 1C is a block diagram depicting example components of another embodiment of a see-through, mixed reality display device system using a mobile device as a processing unit.

Fig. 2A is a side view of a temple of a frame in an embodiment of a transparent, mixed reality display device embodied as eyeglasses that provides support for hardware and software components.

Fig. 2B is a top view of an embodiment of a display optical system of a see-through, near-eye, mixed reality device.

FIG. 3 is a block diagram of a system from a software perspective for providing a mixed reality user interface through a see-through, mixed reality display device system in which software for rendering printed material dynamic may operate.

Fig. 4A illustrates an example of a print content selection metadata record.

Fig. 4B shows an example of print-medium dependent and medium-independent cross-reference content data storage.

FIG. 5 is a flow diagram of an embodiment of a method for rendering static print content dynamic with virtual data.

FIG. 6 is a flow diagram of an embodiment of a process for identifying print content items in a field of view of a see-through, mixed reality display device.

FIG. 7A is a flow diagram of an example of an implementation of a process for identifying at least one physical action for a user's eye selection of print content selection.

FIG. 7B is a flow diagram of another implementation example of a process for identifying at least one physical action of a user's eye selection print content selection.

FIG. 7C is a flow diagram of an example of an implementation of a process for identifying at least one physical action of a gesture to select print content selection.

FIG. 8A is a flow diagram of an example of an implementation of a process for generating a user-defined gesture and associating the gesture with a task.

Fig. 8B is a flowchart of an implementation example of a process for determining placement of dummy data relative to printed material.

FIG. 9A is a flow diagram of an embodiment of a method of performing the task of up-to-date content selection.

FIG. 9B is a flow diagram of an embodiment of a method of performing a search based on at least one keyword associated with print content selection.

FIG. 9C is a flow diagram of an embodiment of a method of performing a search based on at least one keyword associated with a tag selected for print content, the tag being within a sub-division that includes other tags.

FIG. 10A illustrates an example of static printed material that is out-of-date and includes invisible indicia.

Fig. 10B shows an example of inserting virtual data to make the content the latest and interactive virtual content and displaying the search result.

FIG. 11A is a flow diagram of an embodiment of a method for annotating static print content with virtual data.

FIG. 11B is a flow diagram of an embodiment of a method for displaying stored annotations entered for one version of a work for another printed version of the work.

Fig. 12A illustrates an example of a gesture to specify a selection of handwritten content.

FIG. 12B shows an example of a virtual typing input device displayed for a user.

FIG. 12C shows an example of a finger gesture used to select printed text to annotate.

FIG. 12D illustrates an example of virtual data displaying the formatted text version of the handwritten content selection of FIG. 12A as an annotation.

FIG. 13A is a flow diagram of an embodiment of a method for providing a virtual version of print content at a comfortable reading position.

FIG. 13B is a flow diagram of an embodiment of a method for providing a virtual version of printed content for improved visibility of the content.

Fig. 14 illustrates an example of providing dummy data for improved visibility and a multi-user perspective shared view.

FIG. 15 is a flow diagram of an embodiment of a method for providing a virtual version of print content linked to a user field of view.

FIG. 16 is a block diagram of one embodiment of a computing system that may be used to implement a network accessible computing system.

FIG. 17 is a block diagram of an exemplary mobile device that may operate in embodiments of the present technology.

Detailed Description

The present technology provides embodiments for virtual data to be viewed and displayed by a see-through, near-eye, mixed reality display device system to make static print content dynamic. The see-through display device system identifies a real book, magazine, newspaper, or other real printed material in the user's field of view. Books, magazines, newspapers, cards, or individual sheets of paper are examples of printed content items that can be identified by object recognition software from image data captured by a front-facing camera positioned on a display device system for capturing objects in the field of view of the display device that approximates the field of view of a user looking through the display device.

In some cases, the eye gaze data identifies where in the field of view the user is focusing on, and can therefore identify which portion of the printed content item the user is looking at. A continuation of focus on a portion of printed material may identify the portion as a print content selection. Gaze persistence is an example of a physical action of a user using a body part. Gestures performed by a user body part (such as a hand or finger) and captured in the image data are also examples of physical motion user input. Blinks or blinking sequences of the eye may also be gestures. A pointing or specific movement gesture of a hand, finger, or other body part may also indicate a print content selection, such as a word, sentence, paragraph, or photograph. User-generated voice commands (such as voice commands) may also be considered examples of physical actions indicative of user input. Sound-based actions are often accompanied by other physical actions such as gestures and eye gaze.

Once the user selects a picture or text, different tasks or applications may be performed for the content selection, such as augmenting with interactive games and holograms, replacing with updated content, and annotating with three-dimensional, two-dimensional, or both virtual data. Usability may also be improved by generating and displaying a virtual version of at least a portion of the print content item.

Fig. 1A is a block diagram depicting example components of an embodiment of a see-through, augmented or mixed reality display device system. System 8 includes a see-through display device that is a near-eye, head-mounted display device 2 that communicates with processing unit 4, either through line 6 in this example, or wirelessly in other examples. In this embodiment, the frame 115 of the head mounted display device 2 is in the shape of glasses, the frame 115 having a display optical system 14 for each eye, wherein the image data is projected into the user's eyes to generate a display of the image data, while the user also views through the display optical system 14 to obtain an actual direct view of the real world.

The term "actual direct view" is used to refer to the ability to see the real-world object directly with the human eye, rather than seeing the created image representation of the object. For example, looking through glasses at a room will allow a user to get an actual direct view of the room, whereas viewing a video of the room on a television set is not an actual direct view of the room. Each display optical system 14 may also be referred to as a see-through display, and the two display optical systems 14 together may also be referred to as a see-through display.

The frame 115 provides a support structure for holding the various components of the system in place and a conduit for electrical connections. In this embodiment, frame 115 provides a convenient eyeglass frame as a support for the various elements of the system discussed further below. In this embodiment, the frame 115 includes a nose bridge portion 104, the nose bridge portion 104 having a microphone 110 for recording sound and transmitting audio data. A temple or side arm 102 of the frame is positioned over each ear of the user. In this example, the right temple 102r includes a control circuit 136 for the display device 2.

As shown in fig. 2A and 2B, an image generating unit 120 is further included on each temple 102 in this embodiment. Also, not shown in this view but shown in fig. 2A and 2B are outward facing cameras 113, said cameras 113 being used to record digital images and video and to communicate visual recordings to control circuitry 136, control circuitry 136 may in turn send captured image data to processing unit 4, processing unit 4 may also send this data to one or more computer systems 12 via network 50.

The processing unit 4 may take various embodiments. In some embodiments, the processing unit 4 is a separate unit that may be worn on the user's body (e.g., wrist), or may be a separate device such as the illustrated mobile device 4 shown in fig. 1C. Processing unit 4 may communicate with one or more computing systems 12, whether located nearby or at a remote location, by wire or wirelessly (e.g., WiFi, bluetooth, infrared, RFID transmission, Wireless Universal Serial Bus (WUSB), cellular, 3G, 4G, or other wireless communication device) over a communication network 50. In other embodiments, the functionality of processing unit 4 may be integrated in the software and hardware components of display device 2 of FIG. 1B.

Remote, network-accessible computer system 12 may be leveraged to handle power and remote data access. An application may be executed on computing system 12, where the application interacts with display system 8 or performs processing for display system 8, or the application may be executed on one or more processors in a see-through, mixed reality display system 8. FIG. 16 shows an example of hardware components of computing system 12.

FIG. 1B is a block diagram depicting example components of another embodiment of a see-through, augmented or mixed reality display device system 8 that can communicate with other devices over a communication network 50. In this embodiment, the control circuitry 136 of the display device 2 communicates wirelessly with one or more computer systems 12 over the communications network 50 via a wireless transceiver (see 137 in FIG. 2A).

Fig. 1C is a block diagram of another embodiment of a see-through, mixed reality display device system using a mobile device as processing unit 4. An example of hardware and software components of mobile device 4 (such as contained in a smartphone or tablet computing device) is depicted in fig. 17. The display 7 of the mobile device 4 may also display data (e.g. menus) for executing applications, and the display 7 may be touch sensitive to accept user input. Some other examples of mobile devices 4 are smart phones, laptop or notebook computers, and netbook computers.

Fig. 2A is a side view of the temple 102 of the frame 115 in an embodiment of the see-through, mixed reality display device 2 embodied as eyeglasses that provide support for hardware and software components. A camera 113 facing the physical environment is located in front of the frame 115, which camera is capable of capturing video and still images of the real world to map real objects in the field of view of the see-through display and thus the user. The camera is also referred to as an outward facing camera, meaning facing outward from the user's head. Each forward facing camera 113 is calibrated with respect to a reference point of its respective display optical system 14 such that the field of view of the display optical system 14 may be determined from image data captured by the respective camera 113. One example of such a reference point is the optical axis (see 142 in fig. 2B) of its respective display optical system 14. The image data is typically color image data.

In many embodiments, the two cameras 113 provide overlapping image data from which depth information for objects in the scene can be determined based on stereo vision. In some examples, the camera may also be a depth sensitive camera that transmits and detects infrared light from which depth data may be determined. The process identifies and maps the user's real world field of view. Some examples of depth sensing technologies that may be included on head mounted display device 2 are, but are not limited to, SONAR, LIDAR, structured light, and/or time-of-flight.

Control circuitry 136 provides various electronics that support other components of head mounted display device 2. In this example, the right temple 102r includes control circuitry 136 for the display device 2 that includes a processing unit 210, a memory 244 accessible to the processing unit 210 for storing processor readable instructions and data, a wireless interface 137 communicatively coupled to the processing unit 210, and a power supply 239 that provides power to the components of the control circuitry 136 and other components of the display 2 (e.g., the camera 113, microphone 110, and sensor units discussed below). Processing unit 210 may include one or more processors, including a Central Processing Unit (CPU) and a Graphics Processing Unit (GPU).

The earpiece 130, the inertial sensor 132, one or more position or proximity sensors 144 (some examples of which are GPS transceivers, Infrared (IR) transceivers, or radio frequency transceivers for processing RFID data) are located inside the temple 102 or mounted to the temple 102. The optional electrical pulse sensor 128 detects commands via eye movement. In one embodiment, inertial sensors 132 include a three axis magnetometer 132A, three axis gyroscope 132B, and three axis accelerometer 132C. Inertial sensors are used to sense the position, orientation, and sudden acceleration of head mounted display device 2. From these movements, the head position can also be determined. In this embodiment, each of the devices that use analog signals in their operation (such as sensor devices 144, 128, 130, and 132 and microphone 110 and IR illuminator 134A discussed below) includes control circuitry that interfaces with digital processing unit 210 and memory 244 and generates and converts analog signals for its respective device.

An image source or image generation unit 120 that produces visible light representing an image is mounted on the temple 102 or within the temple 102. In one embodiment, the image source includes a microdisplay 120 for projecting an image of one or more virtual objects and a coupled optical lens system 122 for directing the image from microdisplay 120 to a reflective surface or element 124. Microdisplay 120 can be implemented in a variety of technologies including projection technology, micro-Organic Light Emitting Diode (OLED) technology, or reflective technologies such as Digital Light Processing (DLP), Liquid Crystal On Silicon (LCOS), and from high-pass, IncDisplay technology. The reflective surface 124 directs light from the microdisplay 120 to the light guide optical element 112, and the light guide optical element 112 directs light representing an image to the user's eye. The image data of the virtual object may be registered with the real object, which means that the virtual object tracks its position to the position of the real object seen through the see-through display device 2 when the real object is in the field of view of the see-through display 14.

In some embodiments, one or more print content selections tracked for enhancement may be printed with one or more markers to improve detection of the content selections. The tag may also include metadata describing the selection of the content. For example, a photograph in a magazine may be printed with an IR retro-reflective marker or RFID tag that includes an identifier of the person in the photograph, as well as the location, date, and time of day that the photograph was taken. Additionally, identifiers for one or more printed or electronic versions of the work that have been printed with the identifiers may be included. The IR or RFID unit 144 may detect the tag and send the data it contains to the control circuitry 136.

Fig. 2B is a top view of an embodiment of a side of a see-through, near-eye, mixed reality display device including display optical system 14. A portion of the frame 115 of the near-eye display device 2 will surround the display optics 14 for providing support and making electrical connections. To illustrate the various components of display optical system 14 (in this case, right eye system 14r) in head mounted display device 2, a portion of frame 115 surrounding the display optical system is not depicted.

In the illustrated embodiment, the display optical system 14 is an integrated eye tracking and display system. The system includes a light guide optical element 112, an opacity filter 114, and optional see-through lenses 116 and 118. An opacity filter 114 for enhancing the contrast of the virtual image is positioned behind and aligned with optional see-through lens 116, a light guide optical element 112 for projecting image data from microdisplay 120 is positioned behind and aligned with opacity filter 114, and an optional see-through lens 118 is positioned behind and aligned with light guide optical element 112. More details of light guide optical element 112 and opacity filter 114 are provided below.

Light guide optical element 112 transmits light from microdisplay 120 to an eye 140 of a user wearing head mounted display device 2. The light guide optical element 112 also allows light from in front of the head mounted display device 2 to be transmitted through the light guide optical element 112 to the eye 140 as indicated by arrow 142 representing the optical axis of the display optical system 14r, thereby allowing the user to have an actual direct view of the space in front of the head mounted display device 2 in addition to receiving the virtual image from the microdisplay 120. Thus, the walls of the light guide optical element 112 are see-through. The light guide optical element 112 includes a first reflective surface 124 (e.g., a mirror or other surface). Light from microdisplay 120 passes through lens 122 and is incident on reflective surface 124. Reflective surface 124 reflects incident light from microdisplay 120 so that the light is trapped within a waveguide, which in this embodiment is a planar waveguide. Representative reflective elements 126 represent one or more optical elements such as mirrors, gratings, and other optical elements that direct visible light representing an image from a planar waveguide to a user's eye 140.

The infrared illumination and reflection also traverse the planar waveguide 112 for the eye tracking system 134 to track the position of the user's eye. The position of the user's eyes and image data of the eyes are generally available for applications such as gaze detection, blink command detection, and the collection of biometric information indicating the personal presence state (state of the user). The eye tracking system 134 includes an eye tracking illumination source 134A, which in this example is located between the lens 118 and the temple 102, and an eye tracking IR sensor 134B. In one embodiment, the eye-tracking illumination source 134A may include one or more Infrared (IR) emitters, such as infrared Light Emitting Diodes (LEDs) or lasers (e.g., VCSELs), that emit at about a predetermined IR wavelength or a range of wavelengths. In some embodiments, eye tracking sensor 134B may be an IR camera or an IR Position Sensitive Detector (PSD) for tracking the location of glints.

The use of a planar waveguide as the light guide optical element 112 in this embodiment allows flexible placement of the entry and exit optical couplings into and out of the optical path of the waveguide for the image generation unit 120, illumination source 134A, and IR sensor 134B. In this embodiment, wavelength-selective filter 123 passes visible spectrum light from reflective surface 124 and directs infrared wavelength illumination from eye-tracking illumination source 134A into planar waveguide 112, and wavelength-selective filter 125 passes visible illumination from microdisplay 120 and IR illumination from source 134A in an optical path that proceeds in the direction of nose-bridge 104. Reflective element 126 in this example also represents one or more optical elements that implement bi-directional Infrared (IR) filtering that directs IR illumination toward eye 140 (preferably centered on optical axis 142) and receives IR reflections from user eye 140. In addition to the gratings and the like mentioned above, one or more hot mirrors (hot mirrors) may be used to achieve infrared filtering. In this example, the IR sensor 134B is also optically coupled to a wavelength-selective filter 125, the wavelength-selective filter 125 directing only infrared radiation from the waveguide (including infrared reflections of the user's eye 140, preferably including reflections captured around the optical axis 142) out of the waveguide 112 and into the IR sensor 134B.

In other embodiments, the eye tracking unit optics are not integrated with the display optics. For more examples of Eye Tracking systems for HMD devices, see U.S. patent 7,401,920 entitled "head mounted Eye Tracking and Display System" issued to Kranz et al on 22/7/2008; see U.S. patent application No. 13/245,739 to Lewis et al entitled "Gaze Detection in a See-Through, Near-Eye, Mixed Reality Display" filed on 30/8.2011; and see Bohn, U.S. patent application No. 13/245,700 entitled "Integrated Eye Tracking and Display System," filed on 26/9/2011, all of which are incorporated herein by reference.

Another embodiment for tracking the direction of the eye is based on charge tracking. This scheme is based on the following observations: the retina carries a measurable positive charge and the cornea has a negative charge. In some embodiments, the sensor 128 is mounted near the user's ear (near the earpiece 130) to detect the potential of the eye as it rotates and effectively read out what the eye is doing in real time. (see "control your mobile music with eyeball activated headphones |) on 19/2/2010 http:// www.wirefresh.com/control-your-mobile-music-with-eye ball-act variant-headphones, which is incorporated herein by reference). Blinks may be tracked as commands. Other embodiments for tracking eye movement (such as blinking) may also be used, based on pattern and motion recognition in image data from the small eye tracking camera 134B loaded on the inside of the glasses. The eye tracking camera 134B sends the buffer of image data to the memory 244 under the control of the control circuit 136.

Opacity filter 114, aligned with light guide optical element 112, selectively blocks natural light from passing through light guide optical element 112 for enhancing the contrast of the virtual image. When the system renders a scene for an augmented reality display, the system notes which real-world objects are in front of which virtual objects, and vice versa. If a virtual object is in front of a real world object, then opacity is turned on for the coverage area of the virtual object. If the virtual object is (virtually) behind a real-world object, the opacity and any color of the display region are turned off so that the user will only see the real-world object for the corresponding region of real light. The opacity filter helps make the image of the virtual object appear more realistic and represent the full range of colors and intensities. In this embodiment, the electrical control circuitry (not shown) of the opacity filter receives instructions from the control circuitry 136 via electrical connections routed through the frame.

Also, fig. 2A, 2B show only half of the head mounted display device 2. A complete head-mounted display device may include another set of selectable see-through lenses 116 and 118, another opacity filter 114, another light guide optical element 112, another microdisplay 120, another lens system 122, a physical environment-facing camera 113 (also referred to as an outward-facing or forward-facing camera 113), an eye tracking component 134, earphones 130, and sensors 128 (if present). Additional details of the head mounted display 2 are shown in U.S. patent application No. 12/905952 entitled "Fusing Virtual Content Into Real Content," filed on 10/15/2010, which is hereby incorporated by reference in its entirety.

FIG. 3 illustrates a computing environment embodiment from a software perspective that may be implemented by display device system 8, a remote computing system 12 in communication with the display device system, or both. Network connectivity allows for full utilization of available computing resources. The computing environment 54 may be implemented using one or more computer systems. As shown in the embodiment of FIG. 3, the software components of the computing environment 54 include an image and audio processing engine 191 in communication with an operating system 190. The image and audio processing engine 191 includes an object recognition engine 192, a gesture recognition engine 193, a sound recognition engine 194, a virtual data engine 195, and optionally eye tracking software 196 (if eye tracking is used), all in communication with each other. The image and audio processing engine 191 processes video, image and audio data received from a capture device, such as the outward facing camera 113. To assist in detecting and/or tracking objects, the object recognition engine 192 of the image and audio processing engine 191 may access one or more databases of the structure data 200 over one or more communication networks 50.

The virtual data engine 195 processes the virtual objects and registers the position and orientation of the virtual objects with respect to one or more coordinate systems. In addition, the virtual data engine 195 performs translation, rotation, scaling, and perspective operations using standard image processing methods to make virtual objects appear realistic. The virtual object position may be registered with or dependent on the position of the corresponding real object. The virtual data engine 195 determines the position of the image data of the virtual object in the display coordinates of each display optical system 14. Virtual data engine 195 may also determine the location of virtual objects in various maps of the real-world environment stored in memory units of display device system 8 or computing system 12. One map may be the field of view of the display device relative to one or more reference points used to approximate the position of the user's eyes. For example, the optical axis of the see-through display optical system 14 may be used as such a reference point. In other examples, the real world environment map may be independent of the display device, e.g., a 3D map or model of a location (e.g., a store, coffee shop, museum).

The one or more processors of the computing system 12 or the display device 8, or both, also execute the object recognition engine 192 to identify real objects in the image data captured by the environment-facing camera 113. As in other image processing applications, a person may be a type of object. For example, object recognition engine 192 may implement pattern recognition based on structure data 200 to detect particular objects, including people. The object recognition engine 192 may also include facial recognition software that is used to detect the face of a particular person.

The structure data 200 may include structure information about the targets and/or objects to be tracked. For example, a skeletal model of a human may be stored to help identify body parts. In another example, the structural data 200 may include structural information about one or more inanimate objects to help identify the one or more inanimate objects. The structure data 200 may store the structure information as image data or use the image data as a reference for pattern recognition. The image data may also be used for facial recognition. Because printed material typically includes text, the structure data 200 may include one or more image data stores including numbers, symbols (e.g., mathematical symbols), and images of letters and characters from the alphabet used by different languages. Further, the structure data 200 may include handwriting samples of the user for recognition. Based on the image data, the dynamic print materials application 202 can convert the image data into a computer-standardized data format for text with a smaller memory footprint. Some examples of computer-standardized text data formats are the universal code for Universal Character Set (UCS) based Unicode (Unicode) and American Standard Code for Information Interchange (ASCII) formats. The text data may then be searched against a database for an identification of the content comprising the text or for related information relating to the content of the text.

After object recognition engine 192 detects one or more objects, image and audio processing engine 191 may report the identity and corresponding location and/or orientation of each object detected to operating system 190, which operating system 190 communicates to applications such as dynamic printed material application 202.

The voice recognition engine 194 processes audio received via the microphone 110.

In an embodiment of the display device system 8, the outward facing camera 113 incorporates a gesture recognition engine 193 to implement a Natural User Interface (NUI). Blink commands or gaze duration data identified by the eye tracking software 196 are also examples of physical action user input. The voice commands may also supplement other physical actions recognized, such as gestures and eye gaze.

The gesture recognition engine 193 can identify actions performed by the user that indicate controls or commands to the executing application. This action may be performed by a body part of the user (e.g. typically a hand or finger in a reading application), but the eye blink sequence of the eyes may also be a gesture. In one embodiment, the gesture recognition engine 193 includes a set of gesture filters, each comprising information about a gesture that can be performed by at least a portion of the skeletal model. The gesture recognition engine 193 compares a skeletal model derived from the captured image data and movements associated therewith to gesture filters in a gesture library to identify when a user (represented by the skeletal model) has performed one or more gestures. In some examples, a camera (particularly a depth camera) in a separate real-world environment from display device 2 in communication with display device system 8 or computing system 12 may detect gestures and forward notifications to systems 8, 12. In other examples, the gesture may be performed by a body part (such as a user's hand or one or more fingers) in the view of the camera 113.

In some examples, matching image data to an image model of a user's hand or finger during a gesture training session, rather than skeletal tracking, may be used to recognize gestures.

More information on the detection and Tracking of objects can be found in U.S. patent application 12/641,788 entitled "motion detection Using Depth Images" filed 12, 18, 2009, and U.S. patent application 12/475,308 entitled "Device for Identifying and Tracking Multiple human devices over Time", both of which are incorporated herein by reference in their entirety. More information about the Gesture Recognition engine 193 may be found in U.S. patent application 12/422,661 entitled "Gesture recognizer System Architecture," filed on 13.4.2009, which is incorporated herein by reference in its entirety. U.S. patent application No. 12/391,150 entitled "Standard getcures" entitled "Standard Gestures," filed on 23.2.2009 for more information on recognition Gestures; and us patent application 12/474,655 entitled "capture Tool" filed on 29.5.2009, both of which are incorporated by reference in their entirety.

The computing environment 54 also stores data in an image and audio data buffer 199. The buffer provides: a memory for receiving image data captured from the outward facing camera 113, image data from the eye tracking camera of the eye tracking component (if used), a buffer for holding image data of virtual objects to be displayed by the image generation unit 120, and a buffer for audio data such as voice commands from the user via the microphone 110 and instructions to be sent to the user via the headset 130.

The device data 198 may include: a unique identifier for the computer system 8, a network address (e.g., IP address), a model number, configuration parameters (such as installed devices), an identification of the operating system, and what applications are available in the display device system 8 and are executing in the display system 8, and so on. Particularly for the see-through, mixed reality display device system 8, the device data may also include data from or determined from the sensors, such as the orientation sensor 132, the temperature sensor 138, the microphone 110, the electrical pulse sensor 128 (if present), and the location and proximity transceiver 144.

In this embodiment, the display device system 8 and other processor-based systems 161 used by the user execute a client-side version of the push service application 204_NThe push service application 204_NCommunicates with the information push service engine 204 over the communications network 50. The information push service engine 204 in this embodiment is cloud-based. The cloud-based engine is executed on one or more networked computer systems and passesThe one or more networked computer systems store one or more software applications of data. The engine is not tied to a particular location. Some examples of cloud-based software are social networking sites and web-based email sites, such asAndthe user may register an account with the information push service engine 204, which grants the information push service permission to monitor the following data: applications the user is executing and data generated and received by the user, as well as user profile data 197, and device data 198 for tracking the user's location and device capabilities. Based on user profile data aggregated from the user's system 8, 161, data received and transmitted by executing applications on the system 8, 161 used by the user, and device data 198₁、198_NThe information push service 204 may determine the user's physical context, social context, personal context, or a combination of contexts, among other sensor data stored therein.

Local copy 197 of user profile data₁、197_NA portion of the same user profile data 197 may be stored and its local copy may be periodically updated over the communication network 50 with the user profile data stored by the computer system 12 in the accessible database 197. Some examples of user profile data 197 are: the user's expressed preferences, the user's list of friends, the user's preferred activities, the user's favorites (examples of the user's favorites include the color of the favorite, favorite food, favorite book, favorite author, etc.), the user's list of reminders, the user's social group, the user's current location, and other user-created content, such as the user's photos, images, and recorded videos. In one embodiment, the user-specific information may be obtained from one or more data sources or applications, such as the information push service 204, the user's social networking sites, contacts, or addressesAn album, scheduling data from a calendar application, email data, instant messaging data, a user profile or other source on the internet, and data directly entered by the user. As discussed below, the presence status may be derived from the eye data and may be updated and stored in the user profile data 197, either locally or by the remote push service application 204. In this embodiment, a network-accessible presence status look-up table 179 links the identified eye data with the presence status as a reference for deriving the presence status.

The trust level may be determined by user profile data 197 that identifies people known to the user as, for example, social networking friends and family members sharing the same gaming service, which may be subdivided into different groups based on trust level. Further, the user may use the client-side push service application 204_NTrust ratings are explicitly identified in their user profile data 197. In one embodiment, the cloud-based information push service engine 204 aggregates user profile data 197 from data stored on different user computer systems 8, 161 of the user_NThe data of (1).

Each version of the push service application 204 also stores a tracking history of the user in the user profile data 197. Some examples of events, people, and things tracked in the tracking history are: visited places, transactions, purchased content and real-world objects, and detected people who interacted with the user. If electronically identified friends (e.g., social networking friends) are also registered with the push service application 204, or they make information available or publicly available to the user through other applications 166, the push service application 204 can also use this data to track the user's content and social context.

As discussed further below, dynamic printed material application 202 may access one or more search engines 203 to access information identifying print content selections and print content items including the print content selections and associated virtual data 205. Can be searched for identification and related virtual dataIs shown as a publisher database 207 and an indexed print content related resource 209 for internet searches. For example, a general search engine may be accessed (such asOr) And a search engine for a library of congress, university or publisher database available to the public or available based on subscriptions (as identifiable in user profile data). The publisher may have pointers to the virtual content in its database 207 because the publisher may have a business model for encouraging the development of virtual content for its printed material. Furthermore, entities unrelated to the publisher or persons wishing to maintain their own data resources may wish to make the virtual content available through their own website (which is an internet-indexed resource). By searching for information derived from the print content selection and the image data of the print content item containing the print content selection, the data fields in the metadata 201 for print content selection can be populated with values. FIG. 4A, discussed below, provides an example of a metadata record for print content selection.

One advantage of this technique is that: the ability to update previously published material that was printed without any planning for virtual augmentation. As discussed below, a user may be requested to view print version identification data regarding print content, such as a title page of a book or newspaper or a table of contents of a magazine. Other examples of version identification data are standardized identifiers, one example of which is the International Standard Book Number (ISBN) of a book. The ISBN number on a book identifies data such as the language group, publisher, title and version or variant of the book. For a journal, the International Standard Serial Number (ISSN) identifies the title of the journal, while the identification of successive publications and their individual articles (SICI) is the standard used to identify a particular volume, article, or other identifiable portion of the journal. For example, ISSN may identify periodicals, such as the magazine "head mounted display," while SCSI identifies articles through bibliographic items (bibliographic items), some examples of which are titles, volumes and numbers, release dates, start and end pages, and content formats, such as TX for printed text. Other content formats may indicate web publishing and audiovisual formats. Image data from the outward facing camera or text converted from the viewed image data of the identification data is sent to one or more search engines 203.

The following discusses a method for improving the readability of printed material by creating a virtual version of the printed material that satisfies readability criteria 206, which readability criteria 206 may be stored as rules for execution by the rules engine of dynamic printed material application 202. Some examples of readability criteria are comfortable reading position criteria and visibility criteria. Some examples of comfortable reading position criteria are angular position criteria from a respective reference point of a see-through display for each eye included in a see-through, mixed reality display device system, a depth distance from the see-through display for each eye, and an orientation of content data in a field of view of the display device. One example of a reference point is the optical axis 142 of the display, which is generally located near the center of the display, and is approximately aligned with the pupil of the user when the user is looking straight ahead at an object. If the printed material is too offset to one side, a neck sprain may result. The depth distance criteria may indicate that the reading material is too close or too far apart. The orientation of the content data may indicate that the text or picture is upside down or to one side, which is not ideal for reading. An example of a visibility criterion may be the size of text or drawing content in the field of view. If too small or too large, the content size may be adjusted to an appropriate level.

The criteria may be based on the user's actual vision (if the prescription is uploaded), the typical vision for the user's age, or based on the average vision characteristics of a human.

Once the printed version of the work the user is looking at is identified and the print content selections are located therein, dynamic printed material application 202 may query one or more search engines 203 to search for virtual content 205 related to the print content selections based on the print content items comprising the print content selections. In some embodiments, the virtual content is associated with a work or a version of the work that includes a selection of content, regardless of the medium in which the content is expressed. For example, paper or other printable material is an example of a medium. Another medium for expressing a work is an electronic display or an audio recording.

In some cases, the virtual data 205 is data specifically generated for appearing in relation to content selection when arranged on a particular printed version (e.g., on a particular page of a book or other detail of printed material). For example, a publisher may create virtual content for updating a recent version of a textbook in which an explanation will be displayed above a page with outdated information indicating that there are nine (9) planets and listing those planets. The latest explanation may be an image that is specially formatted to cover the entire page and explains that there are now only eight (8) planets instead of nine (9) planets and why plutonia does not qualify as a planet. In another example, a publisher with a book layout stored in its database may provide interactive games and other content for the book at predetermined locations on the pages and for particular pages.

In other examples, the virtual content 205 is bound to a media-independent work or work version. For example, a professor may store her notes that she made at different points in her printed version of a textbook to be available to any version of the textbook that is media independent. In other words, the content of the textbook is a work. The current, previous and future versions of the textbook are versions of the work. The dynamic printed material application 202 links each note to a subdivision of the work in its media-independent organization. For example, a note may be linked to a phrase in a particular paragraph that may be identified by executing software instructions for text matching. Paragraphs are a media-independent subdivision, while pages depend on a particular printed or electronic layout. The paperback copy of the textbook with the smaller printed font is a different version of the printed work than the hardcover copy of the textbook with the larger printed font, but they contain identical versions of the textbook content. A professor may allow her virtual notes to be available for storage or streaming to her students at her discretion by granting permission and access to her class students or past students.

Fig. 4A shows an example of a print content selection metadata record including print content selection descriptive data 210, a print content item version identifier 212, print content selection location data 214, a work version identifier 216, and work version location data 218 (if applicable), a work identifier 220, and work location data 222. The work identifier 220 identifies the creative work regardless of the particular format or media. The work location data 222 identifies one or more locations of print content selection by one or more media-independent subdivisions, such as paragraphs, verses, and the like. A work version identifier 216 may be included to describe different versions or versions (e.g., translations) of the work 210, also independent of a particular format or medium. The work version location 218 may also be defined in terms of one or more media-independent subdivisions. Print content item version identifier 212 identifies a particular print layout for a particular print layout. The print version identifier 212 is bound to a medium that is paper or other material that is physically printed. The print content selection location data 214 may be in accordance with a particular static print layout location, such as a page or a location on a page.

For example, the poem "Beowulf" is a work. The original, old english form of the poem is a version of the work as if it were a version with some words replaced with modern english vocabulary. Another example of a version is a french translation. Another example would be the original old english poem footnoted with comments. The printed version identifier 212 may identify a printed version of the poem on one or more pieces of kraft paper stored in the library. This printed version will also have the original old english version of the work identifier and the beohve's work identifier associated with it. The different printed content item version identifier 212 identifies a selection of english documents that have been printed a version of beohwiv that has been footnoted with a comment beginning at page 37 thereof. This different printed version has a different printed content item version identifier 212 and work version identifier than the original old english form of the poem, but the same work identifier. For the content in the selected version of the poem selected by the user, the position data of the print content selection is as per page 37. In this case, too, the work version location data 218 and the work location data 222 include the same verse.

FIG. 4B illustrates an example of printed media-dependent and media-independent content data stores, shown here as a cross-reference database. These databases 211, 213, 215 provide access to specific layouts including content selections. The layout may be media independent or media dependent. In this example, any of the print content item version identifier 212, the work version identifier 216, or the work identifier 220 may be used to cross-reference or index to any of the media independent works 211 and work version database 213 and the media independent or layout specific print content item database 215. Layout or location data for a work, any version of the work, and the individual print content item versions of the work are also cross-referenced. Also, some examples of media-independent segment identifiers may be paragraphs, verses, and so forth that provide media-independent organization to a work or version of a work. Segment 80 of the work may be cross-referenced to page 16 in one print content item version and to page 24 in a larger print font version of the work in another print content item version. Via the print content item version identifier 212, the developer can link to the print layout of the print version (e.g., a particular version) in the print content item database 215. The print layout includes such things as: page number, margin width, header and footer content, font size, graphics and photo location and their size on the page, and other such layout specific information.

Publishers may provide access to the data store of their copyrighted works for identification purposes and as a reference to the layout, version or printed version of the work for the developer of the virtual content. By having access to the layout of the work, the particular work version, and the particular print content item version, a developer is able to create virtual content 205 for the media independent and media dependent versions of the work. As shown, databases 211, 213, 215 and virtual content 205 may be cross-referenced to each other.

For works without copyrights, it is possible to locate in libraries (especially those with large collections, such as congress libraries, other national libraries, universities, and large public libraries, and book editing sites, such as GoogleAnd a site maintained by a university) to obtain a layout to which the reference location data 214, 218, 222 is referenced.

Embodiments of a method for this technique and example implementations of some of the steps of the method are presented in the following figures. For illustrative purposes, the following method embodiments are described in the context of the above-described system embodiments. However, the method embodiments are not limited to operation in the system embodiments described above, but may be implemented in other system embodiments.

FIG. 5 is a flow diagram of an embodiment of a method for rendering static print content dynamic with virtual data. The object recognition engine 192 may recognize an object in the field of view of the see-through, mixed reality display device as an item of printed material (e.g., a book or a periodical or just a page of paper) and be notified of the object recognition by the operating system 190 to the dynamic printed material application 202. At step 302, the dynamic print materials application 202 identifies print content items, and at step 304, identifies user selections of print content selections within the print content items based on physical action user input.

At step 306, the dynamic printed materials application 202 determines a task selected for the print content based on the physical action user input and executes the task in step 308. In step 310, virtual data related to print content selection is displayed according to the job.

Some examples of tasks are: interactive tasks for displaying and updating interactive virtual content (e.g., games) in response to user input, allowing a user to select print content and send it to another user via a messaging application, such as email, instant messaging, or Short Message Service (SMS)Tools, annotation applications, language translation applications, search tasks, update their applications, define applications, create "follow me" applications where a user can manipulate a virtual version of printed content in his or her field of view without looking away from the actual printed content, readability applications to improve visibility of the content and comfort during reading, and revision and refresh applications that generate unmarked versions of printed content. For example, marked versions of printed content may include underlining, scribbling, and notes at margins that make the content almost unreadable. For example, a user with only a condensed version of a work may define a task to fill in deleted content. Another example is a restore application, where the user identifies lost pages and displays them. As mentioned above, the user may also define the tasks.

FIG. 6 is a flow diagram of an embodiment of a process for identifying print content items in a field of view of a see-through, mixed reality display device. In step 312, the dynamic printed materials application 202 electronically outputs instructions requesting the user to place one or more versioned segments of the printed content item into view of the display device 2. Some examples of version identification sections are the ISBN, ISSN and SICI numbers discussed above, the cover of a newspaper or magazine, title pages, home pages, table of contents, and copyright pages. Copyright pages of books typically provide ISBN number, title, print date, edition, author, publisher, and information about previous copyrights in a standard format. In step 314, the one or more version identification portions may be identified in the image data, e.g., a copyright page based template or a standard digital format template for ISBN, ISSN, and SICI and data extracted and placed in a predetermined search field such as text, author, publisher, etc. In other examples, text on a page may be identified based on the alphabet structure data 200 and converted to a computer standard text data format for a query.

In step 316, a query is formulated based on the one or more version identification segments and the query is sent to a search engine for a print content item version identifier in step 318. The dynamic print materials application 202 receives the print content item version identifier in step 319. Optionally, in step 320, in response to verifying the identity of the print content item, the dynamic print materials application 202 receives the media independent work identifier and any applicable media independent work version identifiers. Dynamic application 202 may also receive a work identifier and a work version identifier by using print content item version identifier 212 as an index into publisher database 207 or internet indexed resource 209.

FIG. 7A is a flow diagram of an example of an implementation of a process for identifying at least one physical action for a user's eye selection of print content selection. Eye tracking software 196 typically identifies the position of the eye within the eye socket based on the pupil position, but the iris position may also be the basis. In step 322, dynamic printed material application 202 determines that the duration of the user's gaze at the printed content object has exceeded the time window, and in step 324, causes image generation unit 120 to display a visual enhancement outlining the printed content object. In step 326, in response to identifying the physical action user input confirmation, the dynamic print materials application 202 identifies the print content object as a print content selection by the user. Some examples of physical actions user input confirmation are actions such as blinking, gestures or voice commands indicating "yes" or "select" or a request for a task. The user may indicate a command other than confirmation by a physical action on the visual enhancement (e.g., outline), such as changing the shape of the visual enhancement to include more or less content or a gesture indicating "no" or "cancel," blinking or a voice command.

FIG. 7B is a flow diagram of another implementation example of a process for identifying at least one physical action of a user's eye selection print content selection. In step 322, dynamic printed material application 202 identifies a selection action of the user's eyes during the user's gaze at the printed content object and, in step 334, causes image generation unit 120 to display a visual enhancement outlining the printed content object. In step 336, in response to identifying the physical action user input confirmation, the dynamic printed materials application 202 identifies the print content object as a user's print content selection.

FIG. 7C is a flowchart of an embodiment of an implementation example of a process for identifying at least one physical action of a gesture selection print content selection. In step 342, the dynamic printed material application 202 receives notification that a start gesture of a finger on a portion (e.g., a page) of the printed content material has been detected, and in step 344 causes the image generation unit 120 to display a visual enhancement outlining the movement of the finger on the portion of the printed content material. In step 346, the dynamic printed material application 202 receives a notification that a stopping gesture of a finger on the printed content material has been detected. Because the finger is typically on some portion of the page or sheet or card that the user is reading, the start and stop gestures clearly distinguish when the user is making a request from when the user simply moves the finger position. Other process examples may not require starting and stopping gestures, but instead differentiate movement from gestures based on monitoring user finger behavior over time. In step 348, in response to identifying the physical action user input confirmation, the dynamic print materials application 202 identifies the print content object as a print content selection by the user.

FIG. 8A is a flow diagram of an example of an implementation of a process for generating a user-defined gesture and associating the gesture with a task. Dynamic printed material application 202 in step 352 displays a menu of tasks available for print content selection, which may include user-defined tasks. An example of a user-defined task would be a user selecting content and executing a query to find comments about the content. The user may save the particular search query, or save search terms for criteria to find comments as a task. When a user selects a review task for a different content selection, reviews for the different content selection are retrieved.

The dynamic printed material application 202 receives user input selecting a defined gesture in step 354 and receives user input selecting a task or subtask from a menu in step 356. Outward facing camera 113 captures image data of the gesture performed by the user (which is notified to dynamic printed material application 202) in step 358 and dynamic printed material application 202 associates the gesture as a request for a task or sub-task selected in a menu in step 360.

Some printed materials, such as books and periodicals, may be printed with a layout that includes designated points for virtual content. For example, next to a photo (which has a tag with metadata identifying the photo and related virtual content or data) may be a space of a predetermined size into which the related virtual content fits. The space may also have indicia, such as an RFID tag or IR tag, identifying the virtual content to be displayed there. However, even for content that is printed in advance for enhancement by virtual data, the user may activate a task such as a search task and receive data for which the page is not pre-formatted. Software executing in the computing environment 54 on the display device system 8, the remote computer system 12, or both determines where to place the virtual data. The user may also specify placement by physical action. For example, the user may look at the virtual data for a duration of time and then look at the blank spots on the sheet or page. In another example, the user may point a finger at a virtual object and drag the finger to another point on the sheet or page.

Fig. 8B is a flowchart of an implementation example of a process for determining placement of dummy data relative to printed material. In this example, the dynamic print materials application 202 has a plurality of predetermined location options related to print content selection to select from. The user can move the virtual data from the predetermined position according to his or her preference. In this example, in step 353, the dynamic printed material application 202 determines whether an applicable job of execution requests an alternate location. For example, a task may be a personalization task with subtasks that change or insert role names into those specified by the reader or one or more users. If replacement is desired, dynamic application 202 selects to display virtual content at an alternate location for the print content in step 355. In step 357, in response to executing the task not requesting an alternate location, the dynamic print materials application 202 determines whether the virtual data content is properly placed in the inter-row location and still meets the visibility criteria. An interline position is a space between lines of text, or between a line of text or a picture, or between pictures. One example of visibility criteria is: the size of the virtual content that fits into the interline position is too small for a person with average vision to read in a comfortable reading position. Whether virtual data content fits in an inter-row position may be determined based on what percentage of the content may be displayed in the inter-row position and still be visible. Synonyms as defined are one example of content that can fit into an interline location and still meet visibility criteria. The inter-line position is usually not suitable for pictures. If the inter-line position is appropriate, then dynamic application 202 selects to display virtual content at the inter-line position for the print content at step 359.

If the inter-row location is not appropriate, then in step 361, dynamic application 202 determines if the virtual data content fits into any margin location and still meets the visibility criteria. If one or more satisfactory margin positions are available, then dynamic application 202 selects the satisfactory margin position closest to the print selection in step 363. If a satisfactory margin position is not available, then the dynamic printed material application 202 formats the virtual content into one or more segments in step 365, the segments having the layout properties of the current segment, and displays the one or more segments having the formatted virtual content after the current segment in the layout of the printed material in step 367. One example of a current fragment is a page. The layout properties of the page as a segment include typical page layout settings. Some examples of these settings are margin, page number placement, line spacing, spacing around the picture, font, and font size. Some examples of the layout of the printed material may be a newspaper, a book, a magazine, or a quiz card. In the example where the printed material is a book, one or more pieces that may be formatted using virtual content appear as additional pages of the book.

In the example of fig. 8B, the virtual data is formatted to appear to be within the perimeter of the physical printed material. In other examples, the floating position may also be a position option. For example, for a content selection for which an annotation already occupies the nearest margin, the margin may appear to be expanded to include a picture linked to the content selection. In another example, a floating explanatory paragraph may appear to pop up perpendicular to the page in the interline space near the concept it explains. In the embodiment of FIG. 15 below, a virtual version of the print content selection may be assigned a floating position that is linked to the user's field of view rather than the printed material itself. Fig. 9A-15 illustrate example embodiments of a task, where a user may request the task and the task may generate or bring up virtual data for display.

FIG. 9A is a flow diagram of an embodiment of a method for performing the task of up-to-date content selection. The dynamic print materials application 202 identifies the print content selection as being within an out-of-date segment of the print content item in step 362. One way in which print content selections may be identified as obsolete is that the publisher has identified obsolete content in a stored version of the layout of its print content version. For example, publishers provide descriptive metadata that includes location data within the layout of outdated information.

In one example, the dynamic printed material application 202 identifies obsolete segments in the printed content items by sending a search query including a printed content version identifier to a search engine, the search query requesting identification of the obsolete segments. In response to the search results identifying obsolete segments, the dynamic application 202 receives metadata having location data in the layout of the print content item.

The metadata may be in a standardized format using a markup language, such as extended markup language (XML), to interface with an application through an Application Programming Interface (API). in the above example, dynamic print material application 202 requests the most recent virtual data based on the print content version identifier and the location data.

When such a data access identifier is received by the publisher's database manager, the most up-to-date virtual data for replacing the outdated material is sent to the sender of the data access identifier. In step 364, the dynamic printed materials application 202 retrieves the virtual data with the latest information related to the print content selection, for example, from the publisher's database 207, and in step 366 displays the virtual data with the latest information in a location related to the print content selection. For example, a replacement or floating location may be specified for the most recent virtual data. In other examples, dynamic print materials application 202 may also search internet-indexed resources 209 for virtual data related to print content items and select to retrieve the latest content for the print content from an independent developer of virtual content.

FIG. 10A illustrates an example of outdated static printing material. In this example, a mathematical exercise book 380 for children printed 10 years ago includes a picture 384 of a giraffe₁The Giraffe has a picture 382₁Achieved by radar guns in experiments conducted 11 years ago by the investigator shown inThe fastest speed. Since then, the design of the radar gun was improved, while the same investigators performed the same experiments the last year and recorded the speed of even faster lions. Text fragment 381₁And 381₂Mathematical problems are provided and account for variations in the radar gun that enable the radar gun to capture faster speeds.

Fig. 10B illustrates an example of inserting virtual data to make the content of fig. 10A up-to-date and interactive virtual content. In this example, the math exercise book 380 virtually erased the giraffe's picture 384₁And with replacement image data 384 of the mountain lion in the layout position of the picture₂And (6) covering. In addition, the updated picture 382 is replaced₂A picture 382 of a researcher ten years ago is updated₁. Interaction point 386 is shown as interactive button hologram 386. From image data captured by outward facing camera 113, a physical action of finger 387 of hand 385 pressing a virtual button is detected, which physical action causes virtual calculator 388 to appear, which virtual calculator 388 may cause segment 381₁And 381₂The mathematical problem in (1) is more interesting to do.

When the see-through, mixed reality display device system is capable of producing three-dimensional (3D) virtual objects, a picture 347 of a lion₂And calculator 378 lies flush on the page as if printed on the page. The 3D hologram 382 appears to be produced by the page.

Another task that readers of printed content can now perform with display system 8 is one or more embodiments of a search task. Publishers of content and other indexers can supplement their stored layout with keywords associated with one or more print content selections, which can be used to search for information related to the print content selections. As discussed in fig. 9C, the printed material may be printed with indicia (e.g., invisible RFID or IR tags, or a subset of visual data) that identifies different print content selections. One example of a visible marker is a subset of the image data that acts as a signature for the image data. For example, when a print content item is identified, image data sub-elements along intersecting diagonals may identify a picture. The marking data is a quick reference which can reduce the processing time in selecting the print content in the identification layout. One or more keywords may be stored and updated for each tag.

FIG. 9B is a flow diagram of an embodiment of a method of performing a search based on at least one keyword associated with print content selection. In step 368, the dynamic printed materials application 202 identifies a search request for print content selection of a printed content item based on the physical action user input data. For example, the physical action user input may be based on eye tracking data, in particular gaze data. As some examples, the user may define that a gaze duration length, an underlining gesture, or a blinking sequence is an indication of a task request to search on a print content selection.

In step 370, the dynamic printed material application 202 can request and receive at least one keyword associated with the print content selection from one or more data stores related to the print content, and in step 372, formulate a search query based on the received at least one keyword. Some examples of data stores related to print content are internet indexed resources 209 and publisher databases 207 related to print content. In step 374, the dynamic printed material application 202 sends the search query to the search engine and displays the search results in the field of view of the see-through display in step 379. See, for example, virtual data 395 for a virtual paper sheet having text search results 397 at off-page positions along the margin closest to the print selection of male reviewer image data underlying the search.

Fig. 10A shows an example of static printed material that is out-of-date and includes invisible markings on the panelist's face 383 and 389 and clothing (390, 391). These marks may have been pre-printed with text. Further, the dashed lines representing the marks may represent marks generated after printing and associated with the layout of the print content item. One example of such a marker is the image data signature discussed above. Even used printed copies of books can be linked to and updated with keywords. For example, each time a new actor shows a character in a new movie version of "three firearms," the keyword or keywords associated with the character name may be updated for the 1937 impression of the novel.

FIG. 9C is a flow diagram of an embodiment of a method of performing a search based on at least one keyword associated with a tag selected for print content, the tag being within a sub-division that includes other tags. In step 369, indicia of print content selection within the subdivision of the print content item are identified from a plurality of indicia for the subdivision based on the physical action user input data. For example, a sensor (such as an IR or RFID sensing unit 144) detects a signal from the tag, the signal including data, the unit 144 converting the data to a processor readable form for the processing unit 210. In some examples, the data identifies print content selections within the segment and may include one or more keywords.

In another example, the visual markers may be stored as the structure data 200, and the object recognition software 192 identifies the visual markers from the image data captured by the camera 113 and notifies the processing unit 210 of the marker identifiers, which may send the marker identifiers to the publisher database 207 or to the internet-indexed resources 209 related to the print content to look for one or more associated keywords.

The subdivisions as discussed above may be paragraphs or pictures or verses. The gaze data allows the system to locate within the segment (pinpoint) the location that the user is focusing on, e.g., which word or words within a paragraph are focused on, or what object in the photograph the user is looking at. Keywords can be assigned at a very detailed level. Gaze durations, blinked fixations, and finger-pointed gestures for refinement and classification, and voice data may be used to select content within a segment.

In step 371, the dynamic print materials application 202 selects a search task for print content selection based on the physical action user input data. As discussed above, the selection of the printed content item and the search request may be indicated simultaneously. In step 373, the dynamic printed material application 202 requests and receives at least one keyword associated with the label from one or more data stores (e.g., 207 or 209), and in step 375 generates a search query based on the at least one keyword associated with the label. The search query is sent to the search engine in step 377, and in step 379, the dynamic application 202 causes the search results to be displayed in the field of view of the see-through display.

Fig. 10B shows an example of inserting virtual data that displays the search result as mentioned above. In the example of FIG. 10B, invisible markers 383, 389, 390, and 391 still signal outward despite the overlay of the dummy data. The processing unit 210 may also include keywords associated with the virtual data in the search query, particularly when the virtual data is identified as relevant to the print content selection. For example, the dynamic printed material application 202 determines this based on having the latest job also executed.

FIG. 11A is a flow diagram of an embodiment of a method for annotating static print content with virtual data. The dynamic printed material application 202 determines in step 392 based on the physical action user input that the task the user is requesting for print content selection is an annotation, and in step 394 causes the image generation unit 120 to display a virtual key entry input device for the user. For example, a virtual smartphone keypad may be displayed that a user can use with both hands to select a key press as if some were the very fast text inputter (texter). In other examples, a virtual QWERTY keyboard or other typical computer keyboard may be displayed for user interaction.

The dynamic printed material application 202 generates annotation data based on user input entered into the input device with the virtual keyboard in step 396 and displays the annotation data at the location indicated by the physical action user input in step 398. In step 400, annotation data is stored and linked to annotation data having a media independent data version of the content in the print content selection, e.g., a work or work version. By storing annotations with versions that are media independent, the user may recall the annotations regardless of the particular print layout of the work. Optionally, in step 401, annotation data may be linked with the print content selection in the print content item, and the annotation data is a version that is commonly used by the user at the current time.

In other examples, the annotation data may be input via a processing unit (e.g., mobile device 4 of fig. 1C) rather than a virtual typing input device. As also indicated in the example of fig. 12A, handwritten notes may be made into annotation data and displayed as the annotation data.

FIG. 11B is a flow diagram of an embodiment of a method for displaying stored annotations entered for one version of a work for another printed version of the work. In step 402, the dynamic printed material application 202 identifies different printed content items in the field of view of the see-through, mixed reality display device that include the same content selection. For example, different print content items have different print content item version identifiers 212 but the same composition identifier 220. The location data, work version, and works of the print content item may also be cross-referenced in databases 211, 213, and 215 by publishers, universities, libraries, or other entities that maintain resources of the internet index. In step 404, a user input is received requesting that user annotation data be displayed while the linked content selection is in the field of view of the display device. In step 406, the dynamic application 202 identifies the same content selection printed in the field of view, and in step 408, displays annotation data linked to the media independent version of the content selection at a location related to the location of the same content selection printed in a different printed content item.

12A, 12B, 12C, and 12D present examples of sequences for annotating printed materials with data from other printed materials.

Fig. 12A illustrates an example of a gesture to specify a selection of handwritten content. A display device 2 is shown with forward facing cameras 113l and 113r for capturing user finger gestures. Gestures performed by other body parts (such as hands, wrists, upper arms, and even feet, elbows, etc.) may also be used to control applications such as dynamic application 202. The hand and finger gestures allow a user to perform the gestures while maintaining a reading of the material in the field of view of the display. Lines 704l and 704r represent lines of sight of the eyes that approximate the gaze vector from the pupil or retina of the user. The notebook page 410 where the user's hand 480 is taking notes across the classroom is followed by the fingertips 482 in the virtually underlined handwritten text 412 to be selected₁To move. The outline tool displays a highlighted box 414 for outlining (outlining) the text that the user is selecting. To begin the sequence discussed in this set of figures, the user may have selected the annotation subtask from a menu of virtual annotation tasks displayed in the field of view or may have performed a gesture or say initiated the "annotation" subtask. As in the above example, start and stop gestures may also be used to demarcate the beginning and end of steps in a sequence. Handwritten notes on paper or cardboard or other material that can be printed can be considered a form of printed material, while handwritten notes on a computer screen will not be a print choice.

Dynamic printed material application 202 identifies letters in the handwriting based on user handwriting samples stored as structure data 200 and formats the selected handwriting into text 412 in a computer standard text data format₂And stores the content selection and print content selection metadata record 201. The handwritten content selection may also be marked as an annotation and linked to a list of annotations.

FIG. 12B shows an example of a virtual typing input device displayed for a user. In this example, image data of the user's thumbs 494l and 494r related to the buttons displayed on the virtual smartphone are captured by the outward facing camera 113. Other fingers may also be detected because not all users are skilled text-entry users, as are common 15-year-old people. The user types the comment "symbiotic" using this virtual typing device 481: an independent, beneficial relationship to each other; will be consulted ". In response to the user's text input, text is generated and displayed on virtual phone screen 483. The user can tap custom "send", "return" and such buttons as on a real phone and transmit the editing and completion of the text entered annotation.

FIG. 12C shows an example of a finger gesture used to select printed text to annotate. As in fig. 12A, the forward facing cameras 113l and 113r capture image data of the user's fingers 482 lining down the word "symbiosis" in the textbook as print selections 462.

FIG. 12D illustrates an example of virtual data displaying the formatted text version of the handwritten content selection of FIG. 12A as an annotation. As seen in the field of view of display device 2 and as captured by forward facing cameras 113l and 113r, after the annotation task is active and the user performs a selection gesture for text "symbiosis," the user's finger 482 is identified by the dynamic print materials application 202 from the image data as pointing to the user-specified top margin position of the page for the text version of the handwritten content selection. Although not shown, the user may have selected the text version 412 from a virtual menu or display of recently made annotations₂. In another example, when text version 412₂Is the last annotation made in the current session, the dynamic printed materials application 202 displays the text version. Text version 412₂Is displayed at the top of the page and writes "symbiotic: an independent, beneficial relationship to each other; will be consulted ".

In addition to providing additional information or interactive content to the static printed material, virtual versions of the static printed content material may be made to improve the visibility of the content due to changing the appearance properties of the content or the location of the content in the field of view of the display device.

Fig. 13A and 13B discuss an embodiment of a method for improving the readability of printed material by generating a virtual version of the material.

FIG. 13A is a flow diagram of an embodiment of a method for providing a virtual version of printed material in a comfortable reading position. In step 432, dynamic printed material application 202 identifies a location from the image data that may include an orientation of the printed material in a field of view of the see-through, mixed reality display device.

In step 434, dynamic printed material application 202 determines whether the position of the printed material in the field of view meets comfort criteria. Some examples of comfort criteria are the angle of the printed material (e.g., a book or magazine), and the determined angle of the text relative to a reference point of the display device (e.g., the optical axis 142 of the see-through display). Whether the angle of the printed material is within a comfortable reading area may also be determined based on the estimated gaze vector and head position data derived from the orientation sensing unit 132. For example, the reference head position may be a 0 degree head position, with 0 degree head position meaning looking straight ahead rather than looking at an angle up or down or left or right. From this reference to the 0 degree head position, the reading comfort region or comfort criterion may indicate a head position no more than 45 degrees from the 0 degree position. A gaze estimation vector that is more than 45 degrees from the optical axis 25 in any direction may be used as a threshold value indicating that the comfort criterion is no longer met.

The determined angle or orientation of the text may identify whether the text is upside down or at another uncomfortable angle. The depth distance may also be determined based on stereo viewing of image data applied to the capture device 113 or depth data obtained when the camera 113 is implemented with depth sensing capabilities. If the printed material is outside the depth comfort range (e.g., 1 to 3 feet), generation of a virtual version may be triggered. The arm length of the user (which may be available in user profile data 197) may be a guide for determining a depth comfort range.

If the location meets the reading location criteria, then in step 452, the dynamic printed material application 202 returns to step 432 for the next scheduled check. If the reading position criteria are not met, the dynamic printed material application 202 determines a new position for meeting the comfortable reading position criteria in step 436. For example, the new position orients the book within the distance boundaries of a comfortable angle and a reading comfort zone. In step 438, the dynamic printed material application 202 generates image data of a virtual version of the printed material. The virtual version may be generated based on image data of the printed material. Further, the dynamic printed material application 202 may generate a virtual version based on an electronic version of the printed material accessible to the dynamic printed material application 202. For example, a newspaper publisher may make an electronic version that is accessed with a mobile tag printed on a copy of their newspaper. The electronic version may have complete text of the printed material (e.g., a full day newspaper) along with its layout information. In step 440, the dynamic printed material application 202 causes the image generation unit 120 to display the virtual version at a new location in the see-through display and returns to the next scheduled check of step 432 in step 452.

FIG. 13B is a flow diagram of an embodiment of a method for providing a virtual version of printed content for improved visibility of the content. In step 442, the dynamic printed material application 202 identifies the size of the content data in the version in the field of view. In step 446 it is determined whether the visibility criteria are met. As mentioned above, in some examples, the visibility criteria determine whether the size of the content data is likely to be visible to an average person of the user's age at the location and depth of the printed material. The methods of fig. 13A and 13B are complementary. Performing fig. 13A first can adjust for any visibility issues, but even at a comfortable reading position, the text may still be too large or too small. If the visibility criteria are met, then in step 452, the dynamic printed material application 202 returns to step 432 for the next scheduled check.

If the visibility criteria are not met, then in step 448 the dynamic printed material application 202 generates a virtual version of the content data having the changed size to meet the visibility criteria and causes the virtual version of the content data having the changed size to be displayed in the see-through display in step 450. In step 452, the dynamic printing material application 202 returns to the next scheduled check of step 432.

Fig. 14 illustrates an example of providing virtual data for improved visibility and multi-user perspective shared views. In this living room scenario, grandmother 489 sits across from her grandson 484 while she is reading a book to which the interactive virtual content is available. Each person wears a display device system 8 in which all electronics are incorporated in the display device 2 as in the example of fig. 1B. In this version, the book 490 includes a processor and memory that stores virtual data for the book in a markup language. For example, extensible markup language (XML) may be used. In another example, a markup language such as Virtual Reality Modeling Language (VRML) may be used. In some examples, the book includes a password, and multiple users may enter the password and log into the same session. The virtual data is wirelessly transmitted to the logged in display devices that run the group reading plug-in application of the dynamic print materials application 202. Grandmother 489 and grandchild 484 are sharing the same application session, much like a player is playing an online game, but grandmother 489 controls the action.

Grandma 489 likes to see her grandchild to see the virtual data (such as pumpkin lamp 486)₁、487₁、488₁) And his reaction to events and dialog in the story. However, her grandson is just learning his letters and cannot yet read such a book. Grandma has selected a reader role for herself from the menu and assigned a participant role for her grandchildren. To make children interested in reading, their orientation is focused on the book by displaying virtual data as they read the book. The display device system 8 is capable of detecting other devices within a predetermined distance. For example, the display device systems may exchange identity tokens via bluetooth, WUSB, IR or RFID connections. The type and range of the proximity transceiver 144 may be selected to allow only a predetermined time periodConnections within a distance. Location data, such as from GPS transceiver 144 or cellular triangulation based on wireless transceiver signals 137, in combination with location data, such asApplications may also be used to identify devices that are within a predetermined distance of each other.

The virtual data is activated by the eye gaze of the reader on a certain part of the content of the story. The story is a work and the content is on page 485 of the printed content item, which is book 490. Grandma places the book on her knee, and the reading angle does not meet the comfort criteria. However, when a participant, such as grandchild 484, looks at the book in the field of view of his display device system 8, the book is where the virtual data originates, and the viewer's eye gaze activates the virtual content or data.

The process of figure 13A detects an uncomfortable reading position of grandmother. As discussed further below in fig. 15, repeated reading from the book by her may also trigger a "follow me" task. Printing page 485₁Virtual version 485₂Projected into the grandmother's eye such that the virtual page and any adjusted font size or contrast of the virtual page appear in the reading comfort zone in her display. Triggering virtual page 485₂Her gaze duration on the content also triggers the virtual pumpkin 486₁、487₁、488₁Appear for her grandchild participant to look from physical page 485 when he looks at the book₁The medium floats out. In her display, grandma sees a virtual pumpkin 486 from her perspective₂、487₂And 488₂. Also due to the process example of FIG. 15, virtual page 485₂Is positioned so that she can comfortably read the virtual page. Another criterion for placement is to not block participants in the field of view if space allows. The appearance layout variations discussed below may also be used to adapt to keep the participants in view.

Display device if physical book is still grandmotherThen the dynamic application 202 responds to the grandmother's physical action (e.g., flipping a page) on the physical book and to the currently displayed virtual version page 485₂Takes action instead of physical action.

FIG. 15 is a flow diagram of an embodiment of a method for providing a virtual version of print content linked to a user field of view. In step 502, in response to a triggering event, the following user tasks of the dynamic printed material application 202 are triggered. One example of a triggering event is the detection of a segment of printed material moving in and out of the field of view of the display device 2 multiple times. Some other examples of triggering events are user inputs requesting user tasks such as predefined or user-defined gestures or spoken commands such as "follow me".

In step 504, the dynamic printed material application 202 determines whether an electronic version of the printed content item including the printed material being viewed by the user is available. If available, dynamic application 202 may load at least a segment of the print content item into memory so as to be available as the user further reads and advances through the content embodied in the printed material. If an electronic version is available, dynamic application 202 displays image data including the content of the print content selection at a location in the field of view that is independent of the location of the print content item in step 514. In particular, in the example where the user has triggered a "follow me" task moving his or her head back and forth, if gaze duration data or gesture or voice data has not positively indicated a content selection, the dynamic application 202 selects as a print selection a segment of the print item that the user has been focusing on. In step 516, the display of the image data is updated in response to a user input of a physical action on the virtual version. If the print content item is still within view, the dynamic application 202 responds to physical action user input for both the print item version and the virtual version.

The physical action user input may include gestures to interact with the virtual object, such as pushing the virtual page around in view and pinning the virtual page at a "location" outside of the field of view tracked by the dynamic application 202. If the user moves her head to see the "location," application 202 causes the virtual page to be rendered. For example, a cook following a recipe may use the hand movements to push recipes into and out of view as she performs different steps of the recipe.

Optionally, in step 518, the dynamic application updates the display of image data in response to the user's layout change request. One example of a layout change request is to have virtual content appear on a transparency and increase the spacing between the text so that the user can see through the virtual content. The voice command may request transparency. Alternatively, the user may increase the spacing between portions of text to create a see-through window to see the object he or she is affecting. The spacing may be adjusted by a stretching gesture of pulling the virtual page in opposite directions with both hands. A mechanic working under a car may have a transparent version of a page of a handbook with a window that he inserts by a gesture where a flattened hand pushes down one segment of the page and then pushes up another segment of the page so that he can see in the middle. Another example of a gesture may also be two outstretched hands pushing multiple page portions (e.g., segments) in different directions.

In the event that an electronic version of the print content item is not available, in step 506, the dynamic application 202 stores image data of the print content selected segment currently in view, and in step 508 outputs instructions (e.g., audio or projected instructions through a display device) requesting the user to see one or more segments of the print content item that include content available in the current reading session. For example, before going down the car and starting work, the mechanic can see the page in the car manual related to his maintenance. The dynamic application 202 captures and stores image data of one or more segments when each of the segments is in the field of view of the outward facing camera in steps 510 and 512, and then proceeds to step 514. In one sense, the dynamic application 202 makes a copy of the page viewed by the user.

The tasks selected for execution on the physical content may also be performed on the virtual version. Virtual data resulting from any task execution may be stored and also viewed at a later time when the physical content selection is reviewed.

FIG. 16 is a block diagram of one embodiment of a computing system that may be used to implement one or more network accessible computing systems 12, which computing systems 12 may host at least some of the software components of the computing environment 54 or other elements depicted in FIG. 3. With reference to FIG. 16, an exemplary system for implementing the invention includes a computing device, such as computing device 800. In its most basic configuration, computing device 800 typically includes one or more processing units 802, and may also include different types of processors, such as a Central Processing Unit (CPU) and a Graphics Processing Unit (GPU). The computing device 800 also includes memory 804. Depending on the exact configuration and type of computing device, memory 804 may include volatile memory 805 (such as RAM), non-volatile memory 807 (such as ROM, flash memory, etc.) or some combination of the two. This most basic configuration is illustrated in fig. 16 by dashed line 806. Additionally, device 800 may also have additional features/functionality. For example, device 800 may also include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in FIG. 16 by removable storage 808 and non-removable storage 810.

Device 800 may also contain communication connections 812 that allow the device to communicate with other devices, such as one or more network interfaces and transceivers. Device 800 may also have input device(s) 814 such as keyboard, mouse, pen, voice input device, touch input device, etc. Output device(s) 816 such as a display, speakers, printer, etc. may also be included. All of these devices are well known in the art and need not be discussed at length here.

As discussed above, the processing unit 4 may be embedded in the mobile device 5. Fig. 17 is a block diagram of an exemplary mobile device 900 that may operate in embodiments of the present technology. Exemplary electronic circuitry of a typical mobile telephone is depicted. The phone 900 includes one or more microprocessors 912, and memory 910 (e.g., non-volatile memory such as ROM and volatile memory such as RAM) that stores processor readable code that is executed by one or more processors of the control processor 912 to implement the functions described herein.

The mobile device 900 may include, for example, a processor 912, memory 1010 including applications and non-volatile storage. The processor 912 may implement communications as well as any number of applications, including those described herein. The memory 1010 can be any variety of memory storage media types including non-volatile and volatile memory. The device operating system handles the different operations of the mobile device 900 and may contain user interfaces for operations such as making and receiving phone calls, text messaging, checking voicemail, and the like. The application 930 may be any kind of program, such as a camera application for photos and/or videos, an address book, a calendar application, a media player, an internet browser, games, other multimedia applications, an alarm clock application, other third party applications, such as skin applications and image processing software for processing image data to and from the display device 2, as discussed herein, and so forth. The non-volatile storage component 940 in memory 910 contains data such as web caches, music, photos, contact data, scheduling data, and other files.

The processor 912 also communicates with RF transmit/receive circuitry 906, which circuitry 906 in turn is coupled to an antenna 902, which also communicates with an infrared transmitter/receiver 908, with any additional communication channels 963 like Wi-Fi, WUSB, RFID, infrared or bluetooth, and with a movement/orientation sensor 914 like an accelerometer. An accelerometer is included in the mobile device to enable applications such as intelligent user interfaces that let the user input commands through gestures, indoor GPS functionality that calculates the movement and direction of the device after disconnecting from GPS satellites, and to detect the orientation of the device and automatically change the display from portrait to landscape when the phone is rotated. The accelerometer may be provided, for example, by a micro-electromechanical system (MEMS), which is a tiny mechanical device (micron-scale) built on a semiconductor chip. Acceleration direction, as well as orientation, vibration and shock can be sensed. The processor 912 also communicates with a ringer/vibrator 916, a user interface keypad/screen, a biometric sensor system 918, a speaker 920, a microphone 922, a camera 924, a light sensor 921, and a temperature sensor 927.

The processor 912 controls the transmission and reception of wireless signals. During a transmit mode, processor 912 provides a voice signal or other data signal from microphone 922 to RF transmit/receive circuitry 906. Transmit/receive circuitry 906 transmits the signal to a remote station (e.g., a fixed station, carrier, other cellular telephone, etc.) for communication via antenna 902. The ringer/vibrator 916 is used to signal an incoming call, text message, calendar reminder, alarm clock reminder, or other notification to the user. During a receive mode, the transmit/receive circuitry 906 receives voice or other data signals from a remote station via the antenna 902. The received voice signal is provided to the speaker 920 while other received data signals are also processed appropriately.

In addition, a physical connector 988 may be used to connect the mobile device 900 to an external power source, such as an AC adapter or powered docking station. The physical connector 988 may also be used as a data connection to a computing device. The data connection allows operations such as synchronizing mobile device data with computing data on another device.

A GPS receiver 965 that relays the location of the user application using satellite-based radio navigation is enabled for this service.

The example computer systems illustrated in the figures include examples of computer-readable storage devices. The computer readable storage device is also a processor readable storage device. Such devices include volatile and nonvolatile, removable and non-removable memory devices implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Some of the processor or computer readable storage devices are RAM, ROM, EEPROM, cache, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, memory sticks or cards, magnetic cassettes, magnetic tape, media drives, hard disks, magnetic disk storage or other magnetic storage devices, or any other device which can be used to store the desired information and which can be accessed by a computer.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims (10)

1. A method for rendering static print content dynamic with virtual data using a see-through, near-eye, mixed reality display device, comprising:

identifying a print content item in a field of view of a see-through, near-eye, mixed reality display device;

identifying a user selection of a print content selection within the print content item based on a physical action user input;

determining a task selected for the print content based on a physical action user input;

executing the task; and

displaying virtual data related to the print content selection in accordance with the task, wherein the task further includes displaying a virtual version of the print content item at a first location in a field of view of the see-through, near-eye, mixed reality display device that is independent of a second location of the print content item with respect to the field of view of the see-through, near-eye, mixed reality display device, and updating display of the virtual version of the print content selection in response to the physical action user input.

2. The method of claim 1, wherein the task updates the print content selection, further comprising:

identifying the print content selection as being within a stale segment of the print content item;

retrieving virtual data having the latest information for the section including the print content selection; and

displaying the virtual data having the latest information at a position related to the position of the print content selection.

3. The method of claim 1, wherein performing the task further comprises performing a search task by:

identifying, from a plurality of tags of a subdivision of the print content item, the print content selected tag within the subdivision based on physical action user input data;

selecting a search task selected for the print content based on physical action user input data;

requesting at least one keyword associated with the tag and receiving the at least one keyword from one or more data stores related to print content;

generating a search query based on the at least one keyword associated with the tag;

sending the search query to a search engine; and

displaying search results in the field of view of the see-through, near-eye, mixed reality display device.

4. The method of claim 1, wherein the task is a group reading task, and performing the group reading task further comprises:

receiving user input identifying a user associated with the see-through, near-eye, mixed reality display device as a reader or participant;

in response to a user input identifying the user as a reader, displaying a virtual version of the print content item and responding to a physical action user input directed to the virtual version of the print content item or the print content item if the print content item is within a field of view of a camera of the see-through, near-eye, mixed reality display device; and

in response to a user input identifying the user as a participant, displaying virtual data related to the print content item in the field of view in response to one or more notifications from another perspective, near-eye, mixed reality display device of the reader.

5. The method of claim 1, wherein the task is for providing a virtual version of print content linked to the user's field of view, further comprising:

displaying image data including the content of the print content selection at a position in the user's field of view that is independent of the position of the print content selection; and

updating display of the image data in response to the physical action user input.

6. The method of claim 5, further comprising:

determining whether an electronic version of the print content item is available;

in response to an electronic version of the print content item being available, displaying image data of the electronic version;

in response to the electronic version of the print content item being unavailable, storing image data of the print content selection,

outputting, to the user via the see-through, near-eye, mixed reality display device, instructions to view one or more segments of the printed content item that include content available in a current reading session;

capturing image data of each of the one or more segments while the respective segment is in a field of view of an outward-facing camera of the see-through, near-eye, mixed reality display device; and

the captured image data is stored.

7. A see-through, near-eye, mixed reality display device system for rendering static printed material dynamic, comprising:

a respective see-through display for each eye positioned by the support structure;

at least one outward facing camera positioned on the support structure for capturing image data in a field of view of the respective see-through display;

one or more software controlled processors communicatively coupled to a search engine having access to a data store comprising content, layout and virtual data for a work and print content items embodying the work;

the one or more software controlled processors are communicatively coupled to the at least one outward facing camera for receiving image data;

the one or more software controlled processors to identify a user selection of a print content selection based on the physical action user input and the image data;

the one or more software-controlled processors are to identify a print content item that includes the print content selection and a media-independent version of a work that includes the print content selection based on formulating one or more queries from the image data and sending the one or more queries to the search engine; and

the one or more software controlled processors cause at least one communicatively coupled image generation unit to display virtual data associated with one or both of the print content selection or the media independent version of the print content selection through each respective see-through display optically coupled to the at least one image generation unit, further comprising:

the one or more software-controlled processors cause at least one communicatively coupled image generation unit to display a virtual version of the print content item at a first location in a field of view of the respective see-through display that is independent of a second location of the print content item with respect to the field of view of the respective see-through display, and update display of the virtual version showing the print content selection in response to physical action input by the user.

8. The system of claim 7, further comprising:

a memory for storing data and software, the software including software for converting text in image data to text in a computer standardized text data storage format; and

the one or more software-controlled processors to identify print content items that include the print content selection and a media-independent version of a work that includes the print content selection based on formulating one or more queries from the image data and sending the one or more queries to the search engine further comprise:

the one or more software-controlled processors identify one or more version-identifying segments of the print content item in image data,

the one or more software-controlled processors formulate and send a query to the communicatively coupled search engine based on the one or more versioning segments, an

The one or more software controlled processors receive a print content item version identifier.

9. The system of claim 8, further comprising:

the one or more software-controlled processors request and receive from the data store based on the print content item version identifier: the print content selects location data within the layout of the print content item, a media independent work identifier, media independent version of work location data selected according to the media independent subdivision of the print content, and any applicable media independent work version identifier and work version location data according to the media independent subdivision.

10. A method for improving readability of statically printed material with virtual data using a see-through, near-eye, mixed reality display device system, the method comprising:

identifying printed material in a field of view of the see-through, near-eye, mixed reality display device system based on image data captured by one or more outward-facing cameras of the display device system;

determining whether the printed material located in the field of view satisfies readability criteria; and

in response to readability criteria not being met, displaying a virtual version of the printed material that meets readability criteria in the field of view, wherein the virtual version is displayed at a first location in a field of view of the see-through, near-eye, mixed reality display device system that is independent of a second location of the printed material with respect to the field of view of the see-through, near-eye, mixed reality display device system; and

responding to a physical action user input directed to one or both of the virtual version of the printed material or the printed material if the printed material is still within a field of view of the one or more outward-facing cameras.