JP6362325B2 - Object tracking - Google Patents

Description

  [0001] Tracking the position of an object and other states can occupy a great deal of effort in daily life. In addition, a lot of time may be spent trying to locate an object placed in the wrong place. For example, searching for the wrong car key, wallet, mobile device, etc. may cause people to lose productive time. Similarly, if you forget that the milk cart in the refrigerator at home is almost empty, if the shopper remembers the status of the cart, an additional outing to the store will be avoided. You may have to do. In some cases, such objects may be moved or emptied by someone other than the owner, thereby complicating the tracking task.

  [0002] Accordingly, various embodiments relating to automatic tracking of objects are disclosed herein. For example, one disclosed embodiment provides a method of operating a mobile computing device that includes an image sensor. The method includes: obtaining image data; identifying an inactive movable object in the image data; determining whether the inactive movable object is a tracked object; Storing information relating to the state of the inactive movable object when the inactive movable object is a tracking target object. The method further includes detecting a trigger for providing a notification of the state of the inactive movable object, and providing an output of a notification of the state of the inactive movable object.

  [0003] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Absent. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.

[0004] FIG. 1A illustrates a user of one embodiment of a see-through display device viewing a usage environment through a see-through display device. [0005] FIG. 1B illustrates the user's perspective of FIG. 1A and also illustrates one embodiment of a tracked object alert displayed on a see-through display device. [0006] FIG. 2 illustrates another embodiment of a tracked object alert displayed on a see-through display device. [0007] FIG. 3 illustrates a block diagram of one embodiment of a use environment for a see-through display device configured to track an object. [0008] FIG. 4 shows a flow diagram illustrating an embodiment for tracking an object with a see-through display device. [0009] FIG. 5 shows a block diagram illustrating one embodiment of a computing device.

  [0010] As noted above, tracking and remembering the state of an object (eg, position or other physical state) can take a significant amount of effort during daily life. In addition, failing to accurately track the state of an object can result in a loss of time and productivity. Accordingly, embodiments relating to automatically tracking the state of an object are disclosed herein. Briefly, a mobile device, such as a see-through display worn by a user, can include an image sensor that observes the user's environment. Video data from the image sensor can be processed to detect objects in the user's environment and identify the tracked object in the video data. The state information of the tracked object can then be stored, for example the position and other characteristics. As a result, when an alert trigger such as a user request or a clue related to the context is detected, an alert regarding the state of the tracking target object can be output. The state information can be stored locally and / or uploaded to a remote service. When multiple people utilize an object tracking device as disclosed herein, each person can see changes to objects made by other users through sharing object tracking information. In this way, it is possible for the user to find the latest location of the lost key, and a reminder that he should buy more milk while looking around the grocery store's dairy counter And / or other object state information can be tracked and recalled in any suitable manner.

  [0011] FIGS. 1A and 1B illustrate non-limiting exemplary usage scenarios. First, FIG. 1A shows a user wearing a head-mounted see-through display device 100 in the shape of glasses and watching the environment (living room) through the see-through display device 100 at a first time t1. As described in more detail below, the see-through display device 100 includes one or more outward-facing image sensors (RGB and / or depth sensors) that capture video images as the user is moving around. The acquired video image can be analyzed in real time or later to identify movable inanimate objects that are tracked by the user, as described in more detail below. In FIG. 1A, an exemplary embodiment of a tracked moveable inactive object is shown as a key 102 placed under a pile of magazines on a table, but any suitable object is tracked. It will be understood that you can.

  [0012] The identified object data and / or image data can be stored to provide future information, such as alerts, to the user regarding the object. For example, referring to FIG. 1B, the key 102 is covered with the newspaper 104 at a later time t2. Thus, the key is highlighted 106 on the see-through display device 100 in response to a user input requesting information about the position of the key (eg, a voice command detected through the microphone of the see-through display device 100).

  [0013] It will be appreciated that the see-through display device 100 may not track state information for all objects that the user observes or even interacts with. For example, in FIG. 1A, the environment observed by the user may be various other inactive objects other than the key 102 that the user may not want to track, such as magazines 108, flower pots 110, and bookshelves. The book 112 is included.

  [0014] Accordingly, the see-through display device 100 is able to track the state of an object that is considered sufficiently “important”, where the term “important” designates the object as a tracking target. Indicates whether sufficient implicit and / or explicit information has been received. Thus, before storing any information regarding the state of the observed inactive movable object, the see-through display device 100 determines whether the object is a tracked object and then if the object If is an object to be tracked, state information can be stored. The see-through display device 100 can also detect information that changes the “importance score” assigned to the object. As will be described in more detail below, such importance scores may be one or more thresholds for determining whether to track object state information and / or which object state information should be tracked. Can be compared with importance score.

  [0015] An inactive movable object can be defined as a tracking target in any suitable manner. For example, the status of an object to be tracked can be predefined (eg, by a developer), can be user defined, and / or can be learned adaptively. In an embodiment where some objects are predefined for tracking, the pre-definition applies to a group of objects that are considered important to most users, such as keys, wallets, financial transaction cards, etc. Can do. In addition, the user may have the ability to remove objects that meet a predefined important object definition from the tracked object list if tracking is not desired for that object.

  [0016] The object can be designated as a tracking target in any suitable manner. For example, the see-through display device 100 may allow the user to hold an object within the field of view of the image sensor of the see-through display device 100 or otherwise place it, scan the image of the object, and identify it for tracking The object designation mode can be included. As a more specific example, a user wearing a see-through display device 100 may hold an inactive movable object (eg, key, wallet, credit / debit card, etc.) in front of the see-through display device 100, for example, “ A voice command such as "Track this object" can be used to trigger the see-through display device to store the object as a tracked object.

  [0017] Furthermore, in some implementations, the user may be able to place time constraints and / or position constraints on the designation of the tracked object. For example, a user traveling abroad may specify a passport as a tracked object during a specified date range or until returning to a specified location (eg, home), and the specified date, location, or other Tracking can be automatically terminated when a condition is met. In addition, the user may have the option to change the status of inactive movable objects from tracked to untracked at any time. It is understood that these methods for user designation of tracked objects are presented for illustrative purposes, and the user can define the tracked object in any suitable way. right.

  [0018] As described above, in some implementations, the see-through display device 100 can adaptively learn which objects to track. This could help a person to find an object that was not explicitly defined as a tracked object but is still potentially important to a person. The see-through display device 100 can adaptively designate an inactive movable object as a tracked object based on any suitable criteria. For example, the see-through display device 100 designates an inactive movable object as a tracked object based on the number, pattern, and / or frequency of the object's sightings and / or user interactions with the object, and more Objects that are frequently seen and / or interacted with can be considered more likely to be important.

  [0019] Further, the see-through display device 100 recognizes certain locations (eg, home, work, and other locations), time, and / or circumstances (eg, ready to go to work) Whether or not an actively movable object should be designated as a tracked object can be determined based on where and / or circumstances the user interacts with the object or otherwise sees the object. For example, an object placed on a user's home bookshelf may be more likely to be an important object than an object placed on a merchandise shelf. As another example, an object placed in the immediate vicinity of the tracked object may be more likely to be important than other objects. For example, a user may put a key, a wallet, and a mobile device together on a table. As yet another example, an object moved by another person can be considered potentially important.

  [0020] Further, in some implementations, a user may be able to designate a location and / or a time as a designated critical location and / or designated critical time. In this way, objects that the user interacted with at the designated location and / or at the designated time may be more likely to be important than at times and / or places that are not designated as important. As an example, the user can designate a travel document carrier as an important object during the holidays. Similarly, an object that a user carries from home to another location may be more likely to be an important object than an object that originates from another location. For example, in a restaurant, it may be more desirable to track an object that a user has taken out of a coat pocket and placed on a table than a user's cup.

  [0021] As yet another example, an object that a user interacts at some time of the day (eg, before going out to work in the morning) can be an important object, but at other times the user May be higher than the interacting object. Furthermore, objects that the user interacts with in close proximity to changes in user context in time (e.g., before leaving home to go to work) may likely be important. This is because users may interact with keys, wallets, paper bags, smartphones, etc. before leaving home or work. Objects that the user interacts at the end of variable length events (time spent finding keys and / or looking for keys every morning before leaving home) are also considered more likely to be important be able to.

  [0022] An inactive movable object can be adaptively designated as a tracking target in any suitable manner. For example, as described above, the importance score is relative to an object detected and identified in the video image data, positional, temporal, related to the user's interaction with the see-through display device, It can be assigned based on behavior and / or other factors. If the importance score meets or exceeds the threshold importance score, the object can be designated for tracking, and more detailed information about the state of the object (e.g. latest position, history position for the object) , Values of physical properties that can change, etc.) can be stored. Further, in some implementations, different threshold importance scores can be used to define different levels of importance. This makes it possible to store different types of information for one object and provide different types of alerts based on different levels of importance.

  [0023] The pattern of user interaction with an object can be tracked in any suitable manner. For example, user behavior can be classified based on contextual information regarding time, contextual information regarding location, contextual information regarding the type of action being performed, and / or any other suitable contextual information. Exemplary user actions include having a meal, taking a shower, looking for lost objects, driving a car and going to work, and the like. By tracking and storing information related to such interactions over time, patterns of interaction with detected objects can be observed. For example, it can be observed through the see-through display device 100 that a user puts a key every day at 5:00 pm and looks for that key every morning at 8:00.

  [0024] Alerts can also be generated based on observed user patterns. For example, once a behavior pattern associated with user interaction with a key is recognized, a reminder regarding the location of that key can be automatically displayed every morning at 8:00 am. Such a pattern also builds inferences about where a lost object may exist if the object has moved since it was last observed and stored through a see-through display device Can be used for. An example of a displayed notification that provides suggestions on where to look for a lost object is shown in FIG. 2 in the form of a list 200 of places to look for the object. FIG. 2 also shows a notification 202 regarding where the object was last seen. It will be appreciated that the notification regarding where to look for the lost object may take any form. For example, such notifications may take the form of graphical instructions that guide the user to a promising location, such as an arrow and / or line to be followed, or a highlight that indicates the suggested location to look for. Can do.

  [0025] In some implementations, a common view of a location can be generated, for example, based on input of video data from multiple users each user uploading video data to a cloud-based service. This allows an object seen in one user's video stream to be associated with an object seen in another user's video stream, and therefore when the object is moved by another user. Even they can be tracked. In addition, object metadata so that multiple users can upload tracked object data instead of video data (eg, multiple devices can perform local recognition on the same object). Can be shared between users.

  [0026] As described above, in addition to position, other states of the object can be interpreted and monitored. As a non-limiting example, the liquid level of milk in a milk pack can be monitored. Similarly, a list of the contents of the refrigerator can be monitored over time, and it can always be notified that some items are missing. Based on these observed conditions, alerts can be triggered by contextual triggers. For example, in the case of an observed empty milk pack, a visual overlay, an automatically generated shopping list, and / or an audio reminder are generated when the user is walking near the grocery store's milk counter. Can. Triggers on the context of being at a grocery store's milk counter include, but are not limited to, an annotation or tagging of a particular type / brand of product by the grocery store or milk producer, objects observed at the milk counter Can be interpreted in any suitable manner, including by matching the appearance, size, and / or other physical and / or contextual characteristics with the characteristics of the milk object being tracked. Furthermore, as the user's shopping pattern compared to the fullness of the milk pack is observed over time (i.e., the user consistently buys more milk when the milk pack is empty or almost empty), The full state of the milk pack can be adaptively learned as the state of the tracking target object over time.

  [0027] FIG. 3 shows a block diagram of a usage environment 300 for automated tracking of inactive movable objects. Usage environment 300 illustrates any number N object tracking devices, depicted as object tracking device 1302 and object tracking device N 304, that communicate with a remote object tracking service 306 via a network 307, such as a computer network. It will be appreciated that the object tracking device may take any suitable form, including but not limited to the see-through display device 100 of FIG. It will further be appreciated that some embodiments may omit the remote object tracking service.

  [0028] The object tracking device 302 comprises one or more sensors 308. The sensor 308 may include an image sensor such as a two-dimensional image sensor 310 and / or a depth sensor 312 that collects video data of the user's local environment. Any suitable type and number of two-dimensional image sensors 310 and / or depth sensors 312 may be included. Examples of suitable two-dimensional image sensors include RGB sensors and gray scale sensors. Examples of suitable depth sensors include time of flight sensors, structured light sensors, and stereo depth sensors. It will be appreciated that any other suitable sensor may be included. Examples include, but are not limited to, microphones, global positioning system (GPS) sensors, motion sensors, inward image sensors that detect eye movement, and the like.

  [0029] The image data collected via the image sensor is supplied to an object recognition tracking module 314 that identifies an object imaged by the image sensor and detects state information about the object. The object recognition and tracking module 314 may further detect one or more output devices 316, such as a display 318 (eg, a see-through display or any other suitable device) based on the state of the object, as described in more detail below. Alerts can be provided to the display) and / or one or more speakers 320. The object recognition tracking module 314 can identify objects in any suitable manner, including, but not limited to, classifier functions pre-trained by the developer, and / or user-specific objects. Includes a classification function trained by the user. Here, the classification function compares and / or fits the observed object with the object model 315. The object recognition tracking module 314 can then output an object ID corresponding to the object identified in the image data to the object information storage device 322. Furthermore, if a new object (ie, an object that was not previously detected) is detected, the object recognition tracking module 314 can assign an ID to the object.

  [0030] The object information storage device 322 may be configured to store tracking target object data 324 including, but not limited to, tracking target object ID and status information. The object information storage device 322 can also store the non-tracking target object data 326 in the embodiment in which the user pattern is adaptively learned and the object is automatically designated as the tracking target. Here, the term “non-tracking target” indicates that state information is not stored for such an object. The non-tracking target object data 326 is drawn so as to store information regarding a plurality of non-tracking target objects drawn as the object 1 328 and the object N 332. Any suitable non-tracking object data can be stored. For example, importance score information 330 can be stored for each non-tracked object, where importance score information 330 is assigned to a recognized object based on user interaction with the object. Can then be used to determine whether the state of the object should be tracked. When the importance score exceeds the threshold importance score, the object can be designated as a tracking target object, and state information can be stored for the object.

  [0031] The tracked object data 324 may include any suitable data stored for any suitable number and type of tracked objects. In FIG. 3, a plurality of tracking target objects are shown as tracking target object 1 336 and tracking target object N 337. For the tracking target object 1, various state data 338 are also drawn. For example, but not limited to, location information 340 including the latest location and / or past location may be stored. Such location information can also include information regarding past user interactions with the object. This makes it possible to analyze user patterns to help find lost objects, for example. Any suitable location information can be stored. Examples include information regarding the physical location (eg, GPS coordinates) of the current usage environment and / or location data relating to the context (eg, location relative to other tracked / recognized objects).

  [0032] Further, information regarding the variable property value 342 of the object may be stored. For example, as noted above, any other suitable items related to the absence of a particular food item in the refrigerator, the level of milk in the milk pack, and / or the changing properties of the object Various values can be stored. It will be appreciated that these particular types of state information are described for exemplary purposes and that any other suitable state information may be stored.

  [0033] Information other than the state information can also be stored for the object. For example, in an embodiment that adaptively learns and designates an object as a tracking target, importance score data 344 is stored for the tracking target object to determine whether the designation of the object as the tracking target should be maintained. Can be done. For example, if a user stops interacting with an object that has not been lost, the object's score may decrease over time and the object may not be tracked. In addition, alert conditions 346 can be stored, where the alert conditions define not only the nature of the alert to be provided, but also when the alert should be triggered for a specified object. .

  [0034] In addition, the context data 350 can be stored as object information useful in determining an object importance score. For example, the context data 350 may define location 352, time 354, and / or other context information as “important” for purposes of assigning importance scores to objects. Examples of location 352 may include actual physical location, context rules regarding proximity to other tracked objects, and other such location data. Examples of time 354 are time intervals between clock / calendar time and / or observed object interaction and other events, eg position change (eg, interacting with object and then Contextual rules regarding the time between leaving the house). The context data 350 may further include context rules regarding how position, time, and other context information should be applied in determining the importance score of the object.

  [0035] In some implementations, analysis of image data for an object can be performed later rather than in real time. Accordingly, the object information storage device 322 can include image data 355 that has not been processed by the object recognition tracking module 314. In such an embodiment, the image analysis requested information about the object to locate the object in the image data by finding the relevant image data in the stored image data 355. Sometimes it can be implemented. To assist in finding relevant image data, various metadata such as location (eg, GPS data) and / or time when the image data was acquired can be stored along with the image data.

  [0036] In some implementations, object tracking and storage may be performed locally on the object tracking device 302. In other implementations, as described above, the object tracking device 302 can communicate with a remote object tracking service 306 over a network. Thereby, object data and image data can be shared between users. The remote object tracking service 306 can perform any of the object recognition, tracking, and alert generation functions described above with respect to the object tracking device 302. In addition, information received from a user can be stored in a user information store 356 that stores information for multiple users represented by user 1 358 and user N 359. It is drawn to be. Without limitation, object data 360 (eg, tracked and non-tracked object information), image data 362 (eg, point cloud depth data and / or 2D image data), and context data 364 (eg, user Any suitable information can be stored, including (location / time / other context) used to determine whether or not to track the interacted object. Further, the user data may also include information about other trusted users 366 that can share object data, image data, and / or other information. For example, a user may wish to access image data of other members of the family to help find lost objects. As a more specific example, if a user's spouse moves the user's key from the table into the drawer, the spouse's object data and / or image data is used to help the user find the key. It can be retrieved together with object data and / or image data.

  [0037] FIG. 4 shows a flow diagram illustrating an embodiment of a method 400 for tracking an object with a see-through display device. The method 400 includes, at 402, receiving image data of a background scene visible through a see-through display system from an image sensor. Here, the term “background scene” means the field of view of the real world scene located behind the see-through display for the user. Any suitable image data can be acquired including, but not limited to, 2D video data 404 (RGB and / or gray scale) and depth data 406.

  [0038] The method 400 then identifies, at 408, an inactive movable object in the image data in real-time or later upon receiving a trigger (eg, a user request to locate an object, a context trigger, etc.). And at 410, determining whether the detected inactive movable object is a tracking target object. As described above, inactive movable objects can be identified in any suitable manner, including, but not limited to, by a classification method that compares movable objects to an object model. .

  [0039] Similarly, as described above, an inactive movable object can be defined as a tracking target in any suitable manner. For example, as shown at 412, the user may request that an object be designated for tracking (through voice, gestures, and / or other appropriate user input) and scan an image of that object. Can do. In addition, some objects can be designated as important by the developer and / or manufacturer of the see-through display device.

  [0040] Further, as shown at 414, an inactive movable object may have a score assigned based on user interaction with the object ("importance score") that satisfies a threshold importance score: It can be specified as a tracking target. Any suitable factor or combination of factors can be used to determine such a score. For example, the importance score can be determined based at least in part on the number, frequency, and / or pattern of user interaction with the object over time, as shown at 416. Further, as shown at 418, positional information can also be used in determining the importance score. Such positional information may indicate whether the object is at a location designated as important, as indicated at 420, and / or if the object is subject to another tracking, as indicated at 422. It can include whether or not it is close enough to the target object, but is not limited thereto.

  [0041] Further, the importance score may also be assigned based at least in part on the time the user interaction occurred, as shown at 424. The time may be a specified critical time 426, a temporal proximity between user interaction with the object and a change in user context or other notable event, and / or any other suitable time. Factors can be included. Location and / or time can be designated as important in any suitable way, including but not limited to, by the user, by the developer and / or manufacturer, through adaptive learning, etc. Will be understood. It will also be appreciated that importance scores can be assigned in any other suitable manner. For example, in some implementations, a user can assign an importance score to an object. In such implementations, the user assigned score can be permanently assigned or can be an initial value that is later changed based on user interaction with the object. Similarly, in other implementations, the application can select the importance score based on any suitable factor other than location and / or time.

  [0042] If it is determined that the inactive movable object is a tracked object, then method 400 includes, at 430, storing information regarding the state of the object. The stored state information can include, but is not limited to, position information and / or information regarding the values of physical properties that can be changed by the object. Further, the status information can be stored locally, as shown at 432, and / or sent to a remote service for remote storage, as shown at 434.

  [0043] The method 400 then includes, at 436, detecting a trigger to provide an output of a tracked object state notification. Any suitable trigger can be used. For example, as shown at 438, the trigger may include user input requesting information regarding the state of the object, such as a request to locate a lost object. Further, as shown at 440, the trigger can include a detected value of a physical property that can change for an object that has met a threshold condition. As yet another example, as shown at 442, the trigger may include the detected time, location, and / or other objects whose context is related to the tracked object. For example, detecting that a user is walking through a dairy store in a grocery store has previously detected that the milk is gone / the milk level is low in the user's refrigerator. If so, you can trigger a milk reminder notification output. Similarly, when the user is at a location other than home, the detection of the trigger can include detecting that the user has moved a predetermined distance from the personal belongings. As a more specific example, a trigger can be detected when a user places a wallet at a restaurant and then walks away from the table or walks out of the building. It will be appreciated that the triggers described above are presented for illustrative purposes and are not intended to be limiting in any way.

  [0044] Upon detecting the trigger, the method 400 includes, at 444, providing an output of an object state notification. Any suitable output can be provided, including audio output 446 and / or video output 448. Examples of suitable video output include, but are not limited to, augmented reality images using a see-through display device, as described above with respect to FIG. Further, status information can be obtained from a local storage device or a remote service. If the information is obtained from a remote service, the information can be obtained from the user's account, another trusted user's account as described above, or any other suitable information source.

  [0045] The notification may include any suitable information. For example, as shown at 450, the notification can inform the user of the most recently stored location of the object. As a more specific example, the notification can highlight the physical location of the object in the background scene and / or include instructions (arrows, lines, text, etc.) that guide the user to the latest location. be able to. The notification can also include an alert that the user has left the object in a public place. Further, the notification can include information regarding the detected value of the physical property that may change. For example, if the milk pack meets an empty threshold condition, the notification may include a reminder to buy more milk and / or an option to make an online purchase of more milk.

  [0046] Further, in some implementations, the notification may include an image of the object being searched. This can help find an object that looks similar to the object but has something different. For example, if the user is looking for a set of keys and the display device has information about four sets of keys that look similar, the display device will show an image of the four sets of keys, It can allow the user to select the pair being searched. In addition, an image of the current arrangement of objects can be displayed. For example, if the set of keys being searched is currently on the night table, the display device can display an image of the keys on the night table to indicate to the user the actual location of the keys. .

  [0047] In this way, the state of the object is automatically tracked on behalf of the user, thereby locating the lost object and / or tracking the state of any other suitable object Can be facilitated. In some implementations, the methods and processes described above can be tied to a computing system that includes one or more computers. In particular, the methods and processes described herein can be implemented as computer applications, computer services, computer APIs, computer libraries, and / or other computer program products.

  [0048] FIG. 5 schematically illustrates a non-limiting computing system 500 that can implement one or more of the methods and processes described above. Computing system 500 is shown in a simplified form. It should be understood that almost any computer architecture can be used without departing from the scope of the present disclosure. In various embodiments, the computing system 500 is a mainframe computer, server computer, desktop computer, laptop computer, tablet computer, home entertainment computer, network computing device, mobile computing device, mobile communications device, see-through display. It may take the form of a device, a near-eye display device, a gaming device, etc., including but not limited to the see-through display described herein with reference to FIGS. Other computing devices are included.

  [0049] Computing system 500 includes a logical subsystem 502 and a data retention subsystem 504. The computing system 500 may optionally include a display subsystem 508, a communications subsystem 506, and / or other components not shown in FIG. The computing system 500 may also optionally include user input devices such as a keyboard, mouse, game controller, camera, microphone, and / or touch screen, for example.

  [0050] The logical subsystem 502 may include one or more physical devices configured to execute one or more instructions. For example, a logical subsystem executes one or more instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical structures. Can be configured as follows. Such instructions can be implemented to perform tasks, implement data types, convert the state of one or more devices, or otherwise reach a desired result.

  [0051] The logical subsystem may include one or more processors configured to execute software instructions. Additionally or alternatively, the logical subsystem can include one or more hardware logical machines or firmware logical machines configured to execute hardware or firmware instructions. The logical subsystem processor may be single-core or multi-core, and the program executed thereon may be configured for parallel or distributed processing. The logical subsystem may optionally include individual components distributed across two or more devices that may be remotely located and / or configured for collaborative processing. One or more features of the logical subsystem can be virtualized and executed by a remotely accessible networked computing device configured in the form of cloud computing.

  [0052] The data retention subsystem 504 is configured to retain one or more data and / or instructions that are executable by the logical subsystem to implement the methods and processes described herein. Physical and non-transitory devices can be included. When such methods and processes are implemented, the state of the data retention subsystem 504 can be converted (eg, to retain different data).

  [0053] The data retention subsystem 504 may include removable media and / or embedded devices. The data retention subsystem 504 includes, among other things, optical memory devices (eg, CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory devices (eg, RAM, EPROM, EEPROM, etc.), and / or magnetic memory devices ( For example, a hard disk drive, a floppy (registered trademark) disk drive, a tape drive, an MRAM, and the like can be included. Data retention subsystem 504 has the following characteristics: volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and content Devices with one or more of addressable can be included. In some implementations, the logical subsystem 502 and the data retention subsystem 504 can be integrated into one or more common devices, such as an application specific integrated circuit or system on chip.

  [0054] FIG. 5 is also a removable computer readable medium that can be used to store and / or transfer executable data and / or instructions to implement the methods and processes described herein. An aspect of a data retention subsystem in the form of a storage medium 510 is also shown. The removable computer readable storage medium 510 can take the form of, among other things, a CD, DVD, HD-DVD, Blu-Ray Disc, EEPROM, and / or floppy disk.

  [0055] It should be appreciated that the data retention subsystem 504 includes one or more physical and non-transitory devices. In contrast, in some embodiments, the instructional aspects described herein are pure signals (eg, electromagnetic signals, optical signals) that are not held by a physical device for at least some finite period of time. Etc.) can be transmitted in a temporary manner. Moreover, data and / or other forms of information related to the present disclosure can be transmitted by pure signals.

  [0056] The terms "module" and "program" may be used to describe one aspect of a computing system 500 that is implemented to perform one or more specific functions. In some cases, such modules and / or programs can be instantiated through a logical subsystem 502 that executes instructions retained by the data retention subsystem 504. It should be understood that different modules and / or programs can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module and / or program can be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module” and “program” are intended to encompass each one or collection of executable files, data files, libraries, drivers, scripts, database records, and the like.

  [0057] A "service" as used herein may be an application program that can be executed across multiple user sessions and available to one or more system components, programs, and / or other services. It must be recognized. In some implementations, a service can operate on a server in response to a request from a client.

  [0058] The display subsystem 508 can be used to present a visual representation of the data held by the data holding subsystem 504. When the methods and processes described herein change the data held by the data holding subsystem, thereby converting the state of the data holding subsystem, the state of the display subsystem 508 is similarly original. It can be transformed to visually represent changes in the data. The display subsystem 508 can include one or more display devices that utilize almost any type of technology, including but not limited to see-through display technology. Such a display device can be coupled in a shared enclosure with the logical subsystem 502 and / or the data retention subsystem 504, or such a display device can be a display device as a peripheral device. Good.

  [0059] When included, the communication subsystem 506 may be configured to communicatively couple the computing system 500 with one or more other computing devices. The communication subsystem 506 may include wired and / or wireless communication devices that are compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured for communication via a wireless telephone network, a wireless local area network, a wired local area network, a wireless wide area network, a wired wide area network, and the like. In some implementations, the communication subsystem may allow the computing system 500 to send and / or receive messages from other devices over a network such as the Internet. .

  [0060] The configurations and / or techniques described herein are exemplary in nature, and numerous variations are possible, and these specific embodiments or examples are understood in a limiting sense. It must be understood that it must not be done. The specific routines or methods described herein may represent one or more of many processing strategies. Thus, the various illustrated operations can be performed in the illustrated order, in other orders, in parallel, or in some cases omitted. Similarly, the order of the processes described above can be changed.

  [0061] The subject matter of this disclosure is not only the various processes, systems and configurations disclosed herein, and other features, functions, operations and / or characteristics, but any and all equivalents thereof. Includes all new and non-obvious combinations and sub-combinations of objects.

Claims (10)

A method of operating a mobile computing device with an image sensor comprising:
Obtaining image data;
Identifying in the image data a movable object tracked by a user ;
At least based on previous user interactions with the front KiUtsuri rotatably object, and determining whether before or KiUtsuri rotatably object is tracked object,
If before KiUtsuri rotatably object is tracked object, comprising the steps of storing information about the state before KiUtsuri rotatably object, the state, the latest physical location of the object in the environment of use Including steps, and
Detecting a trigger to provide a notification of state before KiUtsuri rotatably object,
Comprising the steps of providing an output of the notification state before KiUtsuri rotatably object, said signaling instructions for guiding the user to the stored latest physical location of the object in the use environment Including steps, and
Providing an output of one or more suggested current positions of the object if the object is no longer present at the stored latest position;
Including methods. Previous Previous KiUtsuri rotatably object before storing the information about the KiUtsuri rotatably object states further comprising determining whether a tracking target object, the method according to claim 1. Before further comprising KiUtsuri user input assigning a rotatably object as the tracking target object, and the step of receiving user input to assign before KiUtsuri rotatably object / or user selected severity scores A method according to claim 2 . Assigning a significance score before KiUtsuri rotatably object based on the previous user interactions with the front KiUtsuri rotatably object, KiUtsuri rotatably before on the basis of the importance score that satisfies the threshold importance score The method of claim 2 further comprising: designating an object as a tracked object. The importance score is assigned based at least in part on the number of user interactions with the front KiUtsuri rotatably object The method of claim 4. The importance score of the previous position of KiUtsuri rotatably object, and before KiUtsuri rotatably object and time that the user interaction occurs in assigned based at least in part to one or more of, claims 4. The method according to 4. The importance score is that the position of the front KiUtsuri rotatably object is designated as an important location, proximity to other tracked object before KiUtsuri rotatably object, and a front KiUtsuri rotatably object The method of claim 6, wherein the method is assigned based at least in part on one or more of a user interaction and a change in user context in time.   The method of claim 1, wherein the one or more suggested current positions of the object are based on a historical position of the object and / or user behavior. The state before KiUtsuri rotatably object, before further includes values of physical properties KiUtsuri may vary in rotatably object, detecting the trigger value threshold condition of the physical properties that may said change The method of claim 1, further comprising detecting that A sensor subsystem with an image sensor;
A display subsystem;
A logical subsystem configured to execute instructions;
A data retention subsystem with instructions;
And wherein the instructions are stored on the data retention subsystem and by the logical subsystem,
Receiving image data from the image sensor;
Through the image data, detects a plurality of user interaction with the mobile available-product bodies being tracked by the user,
Assign the importance score before KiUtsuri rotatably object based on the plurality of user interaction with the pre KiUtsuri rotatably object,
If the severity score satisfies a threshold importance score, it stores information about the values of physical properties that may vary before KiUtsuri rotatably object,
Detecting the trigger for providing notification of the values of physical properties which may the change of front KiUtsuri rotatably object,
Before KiUtsuri rotatably feasible to provide an output of the notification of the value of a physical property of the can change the object, a mobile computing device.

Images