WO2018090015A1

WO2018090015A1 - Camera-to-screen pattern recognition system

Info

Publication number: WO2018090015A1
Application number: PCT/US2017/061569
Authority: WO
Inventors: John A. SANDS; Kyle Fitzpatrick; Paul Lacey; Andrew J. WORKMAN; John-Mark COLLINS; Jeffrey K. SCHNEIDER; Scott Banks; Kurt D. Martin
Original assignee: Museum Of Bible Inc
Current assignee: Museum Of Bible Inc
Priority date: 2016-11-14
Filing date: 2017-11-14
Publication date: 2018-05-17
Anticipated expiration: 2019-05-14

Abstract

A recognition system that enables the identification of a mobile device (MD) by a large video display (LVD) so the LVD can distinguish multiple MDs placed on its surface and allow the MDs and LVD to communicate independently via a back-end system.

Description

CAMERA-TO-SCREEN PATTERN RECOGNITION SYSTEM

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 62/421,752 filed November 14, 2016, which is hereby incorporated by reference.

FIELD

[0002] The present invention relates generally to systems and methods for identifying users by mobile devices and of interactively guiding the user through a facility or exhibit.

BACKGROUND OF THE INVENTION

[0003] Interactive surfaces have been developed which provide a tabletop touch screen by which a user can interact with a computing device. A user may wish to connect a mobile device, such as a mobile telephone, with the interactive surface and as the interactive surfaces are touch sensitive they can detect when a mobile telephone or other object is placed on their surface. However, automatic synchronization of the interactive surface and the device can only occur if the device is physically augmented with a tag (e.g. an RFID tag) because the interactive surface cannot distinguish the device on its surface from other devices which are discovered over the wireless link.

[0004] Additionally, proposed systems for identifying mobile devices without tags can be problematic when there are numerous mobile devices that might be used to interact with an interactive surface. For example, if patrons in a museum desire to use mobile devices to interact with interactive surfaces as they move around the museum, each interactive surface may have to distinguish between thousands of mobile devices in trying to identify a single patron's mobile device. Thus, the interaction can be slow and unresponsive. Accordingly, it is desirable to have interactions between mobile devices and interactive surfaces which avoid such problems.

SUMMARY OF THE INVENTION

[0005] Generally, this disclosure relates to a mobile device (MD) to large video display

(LVD) recognition system that facilitates communication between a mobile device and a large video display. Many of the embodiments limit the facilitation to only mobile devices within a predefined geographical area around the large video display screen. In the embodiments, communication can be initiated by placing the mobile device against the large video display. The LVD senses positioning of the MD and determines the physical characteristics of the MD by one of many methods, such as capacitive touch, location, visual analysis, pressure/weight sensitivity. Identifying the specific physical characteristics of the device provides the ability to have the LVD emit a unique light pattern frequency directly in front of the MD's camera, essentially hiding the pattern from the user. The pattern can be displayed with planned variability on the video screen that is recognized by the camera of the placed mobile device. The system allows for specific identification of the MD to the LVD system for the purposes of MD-to-LVD communication, determining the exact location and position of the MD on-screen, allowing a customized menu to appear in relation to the device, with the menu sized according to the device's physical characteristics, as well as the ability to transfer information between the MD and the LVD through the custom menu. [0006] More specifically, one set of embodiments provide for a method comprising determining a group of mobile devices paired with a first large video display. From the group of mobile devices, a first sub-group of mobile devices is selected. Broadly, the first sub-group is selected to provide a more limited group of mobile devices that are available for interaction with the first large video display than the group of paired mobile devices. For example, the first subgroup can be selected based on shared geolocation data such that the first sub-group includes only those mobile devices within a predetermined geographical range of the first large video display. After selection of the first subgroup, a target device is identified from the first sub-group by sending a variable light signal from the first large video display to the optical sensor of the target mobile device.

[0007] In the above embodiments, identifying the target device can comprise placing the target device on the first large video display such that the placement and orientation of the target device is detected by the first large video display. The detection can include detecting a location for the optical sensor. After the detection, the variable light signal can be sent to the location of the optical sensor such that the variable light signal is hidden from detection by a different optical sensor by the placement of the target device on the first large video display.

[0008] Further, the above embodiments can comprise displaying a custom menu for the target device. The custom menu can coordinate with the placement and orientation of the target device on the first large video display.

[0009] In some of the embodiments, the first large video display can be part of a system of large video display units. The system can include at least the first large video display and a second large video display. The group of mobile devices paired with the first large video display can be paired with the system such that the group of mobile devices is also paired with the second large video display.

[0010] In such embodiments, a second sub-group of devices can be selected from the group of devices based on shared geolocation data such that the second sub-group of devices includes only those devices within a predetermined geographical range of the second large video display. Further, the first sub-group can be used to determine mobile devices eligible for interaction with the first large video display, and the second sub-group can be used to determine mobile devices eligible for interaction with the second large video display.

[0011] Also in these embodiments, the first large video display can be in a first room and the second large video display can be in a second room. In such cases, the first sub-group can consist of only mobile devices in the first room, and the second sub-group can consist of only mobile devices in the second room.

[0012] In another set of embodiments, there is provided a method comprising pairing a target mobile device of a patron with a large video display system of a facility. In the method, there is a plurality of mobile devices paired with the large video display system, the target mobile device has an optical sensor, the video display system includes at least a first display unit and a second display unit. The pairing can include placing the target mobile device on the first display unit such that the placement and orientation of the first mobile device are detected by the first display unit including detecting a location for the optical sensor. Also, the method can include sending a variable light signal from the first display unit to the location of the optical sensor location such that variable light signal is hidden from detection by a different optical sensor by the placement of the target mobile device on the first display unit. In response to the pairing, a first custom menu is displayed for the target mobile device. The first custom menu can be coordinated with the placement and orientation of the target mobile device on the first display unit. The first custom menu can include options to create a tour of the facility. The options include one or more of the following: selection of an order for touring exhibits, exhibits of interest, time allotment, meal planning, and rendezvous point. Thereafter, the target mobile device interacts with the second display unit.

[0013] The step of interacting the target mobile device with the second display unit can comprise determining a second portion of the plurality of mobile devices based on shared geolocation data. The second portion includes only those devices within a second predetermined geographical range of the second display unit. The second portion can include the target mobile device. Further, the determination of the second portion is used to limit the number possible mobile devices for interacting with the second display unit.

[0014] Typically, the step of interacting the target mobile device with the second video display unit further comprises selecting the target mobile device from the second portion of mobile devices.

[0015] More specifically, the step of interacting the first mobile device with the second video display unit can comprise placing the target mobile device on the second display unit such that the placement and orientation of the target mobile device are detected by the second display unit including detecting a location for the optical sensor. Next, a variable light signal can be sent from the second display unit to the location of the optical sensor location such that variable light signal is hidden from detection by a different optical sensor by the placement of the target mobile device on the second display unit. Based on the light variable signal, the target mobile device is identified from the first portion of mobile devices. Additionally, a second custom menu for the target mobile device can be displayed on the second display unit. The custom menu can be coordinated with the placement and orientation of the target mobile device on the second display unit.

[0016] In the above embodiments, the first display unit can be in a first room, the second display unit can be in a second room, and the second predetermined geographical range can be the second room such that only mobile devices in the second room are included in the second portion of devices.

[0017] Further, the above embodiments can further comprise determining a first portion of the plurality of mobile devices based on shared geolocation data, wherein the first portion includes only those devices within a first predetermined geographical range of the first display unit, and wherein the determination of the first portion is used to limit the number possible mobile devices for interacting with the first display unit.

BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Fig. 1 is a flowchart describing a method in accordance with one embodiment.

[0019] Fig. 2 is an illustration of users interaction with a large video display in accordance with certain embodiments.

DETAILED DESCRIPTION OF THE INVENTION

[0020] The present disclosure may be understood more readily by reference to the following description. In addition, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, those of ordinary skill in the art will understand that the embodiments described herein can be practiced without these specific details. In other instances, methods, procedures and components have not been described in detail so as not to obscure the related relevant feature being described. Additionally, the description is not to be considered as limiting the scope of the embodiments described herein.

[0021] Generally, the system enables the identification of a mobile device (MD) located within a predefined geographical area by a large video display (LVD) so the two systems can communicate via a back-end system.

[0022] The mobile devices useful in the methods disclosed herein are mobile devices with wireless communication (WiFi, Bluetooth, or similar). The mobile devices are ones that are easily carried and include a visual display. Also, for use of some of the pairing and interaction features described below, it is generally preferred that the mobile device has a camera or similar optical sensor for detecting light and light patterns. Suitable devices include smart phones, portable media players, tablet computers, and similar devices.

[0023] Large video displays useful in the methods disclosed herein are interactive screens, such as touchscreen monitors and touchscreen computing devices, which can emit light, display images and interact through user touch without use of a mouse, touchpad or similar devices (other than a stylus). Generally, the large video displays will be table monitors but can include vertical displays or displays mounted at an angle to horizontal. The large video displays comprises an interactive surface on which a mobile device can be placed, and may use touch sensing the mobile device placement, location and orientation on the large video display. Any suitable touch sensing technology may be used which is capable of detecting a mobile device placed on its surface, such as optical touch sensing (e.g. by detecting shadows caused by objects on the surface or by detecting light reflected back by objects on the surface). In another example, a pressure switch, weight sensor, proximity sensor, light sensor or microphone may be used to detect when the target device is placed onto the initiator device or otherwise brought into proximity to the large video display. In a further example, IR or visible emitters and receivers may be used (e.g. placed around the edge of a display) and presence may be detected when the beam from a transmitter to a receiver is broken by the presence of an object. This beam-break technique may also provide approximate position information for the target object.

[0024] Turning now to FIG. 1, one embodiment of the method of facilitating interaction between a large video display system and a mobile device is illustrated. Users are given a mobile device (MD) by an organization for use within a facility to facilitate learning or navigation (such as a museum). Alternatively, users may use their own personal MD devices by downloading an appropriate application for the MD. In step 101, users sign into the device and/or initiate the application for interaction with the large video display (LVD) system. The system includes one or more large video displays or display units, which will typically be a large touch table (large video display with touch sensing capabilities arranged parallel to the floor) as described above.

[0025] At this point, the MD can identify its location relative to the display unit. That is the

MD can share its geolocation data so that the LVD system recognizes that the MD is within a predetermined geographical range of the display unit, as further explained below. More typically, and as illustrated in FIG. 1 , this shared positioning is used after the initial pairing with the LVD system. [0026] For pairing with the LVD system, the user sets the MD down on the display unit, step 102. In some embodiments, the users place their MD on the display unit in predefined areas of the display indicated by a visual graphic. Alternatively, in other embodiments, the users can place their MD anywhere on a display unit, which uses capacitive sensing or other techniques to detect the placement and orientation of the MD. Next, the display unit detects the MD's specific physical characteristics by one of many methods, such as capacitive touch, location, visual analysis, pressure/weight sensitivity, as illustrated in step 104. Physical characteristics detected for the device can include one or more of length, width, MD optical sensor position, button layout and weight. This allows the LVD system to determine the exact location and orientation of the MD on the display unit. In step 106, the LVD system displays a variable light signal through the display unit to the MD's specific camera location, of which the MD's camera will read/detect the signal (step 108). Identifying the specific physical characteristics of the device provides the ability to have the LVD emit a unique light pattern frequency directly in front of the MD's optical sensor, essentially hiding the pattern from any other optical sensor including eye of the user and other mobile device cameras. Such light signal can be of any detectable type of signal, for example: unique pattern of colors or frequency of one color flashes, or one fixed light signal in a specific patterned frequency (in this case, the pairing will be made by the co-timing of the placement on the display unit with the light signal detection of the MD).

[0027] Recognition of the visual signal then occurs in step 110. If the pattern is not recognized by the MD, the LVD system will re-emit the pattern to the MD, as in step 112. Otherwise, if the signal is recognized, passive handshaking then takes place between the MD and LVD via a wireless communication channel. In step 1 14, the MD sends its identification data, including data identifying the light signal detected wirelessly to the LVD system that will then communicate the device identification to the display unit (step 116). Continuous handshaking occurs wirelessly to communicate between the LVD system and MD. Once the handshaking is established, the LVD system and MD communicate to interact and update each other (step 1 18).

[0028] Additionally, the identification data can include sharing geolocation data of MD, if this has not been shared previously as described above. Including the MD's geolocation allows the reuse of detectable light signals on other display units and increases reliability of the system by reducing the number of possible mobile device candidates.

[0029] As can be seen from FIG. 2, a user 202 places a mobile device 204 on the display unit 200. After handshaking or pairing is established, as described above, menu 206 can be displayed for the user 202. Generally, a specific, custom menu 206 in relation to a specific MD 204 is displayed on the display unit, with the menu appearing to 'fit' the device's size, placement and orientation. Thus, the menu 206 can appear adjacent to and at least partially around the MD as illustrated in FIG. 2. This is especially important when several devices are placed on the display unit at the same time. This allows input of items selected on the menu to be communicated to the MD and vice versa. There can be several users per table, so when a device is placed, the LVD will transmit a separate, distinct light signal to each device so as to initiate handshaking, so as to distinguish specific MD's placed on the LVD and establish 1-1 communication between each MD and the LVD.

[0030] Also, the menu can be customized for the specific user based on the identification information shared between the MD 204 and LVD system. The menu can be further changed and customized by interaction with user 202. [0031] The method and system of the current disclosure can be better understood from the following specific application; however, the system and method has wider application that the specific embodiment described below.

[0032] In one embodiment, patrons within a facility, which in this embodiment is a museum, utilize MD devices interacting with a LVD system to facilitate learning or navigation. The MD devices may be ones supplied by the museum or may be the patron's own personal MD device by downloading an appropriate application onto the MD. Generally, the museum will have a large number of patrons with MD devices interacting with the LVD system. Naturally, this can create system delays as the patron moves around the museum from exhibit to exhibit and from room to room. For example, if a patron moves from one room to another or changes from one exhibit to another, the patron's MD will typically interact with a different display unit in the LVD system. Unfortunately, it can take time to identify the patron's MD or target MD from the MD of the other patrons. This identification time can create unwanted delays in the patron's experience. In order to reduce identification times, the LVD system can use the geolocation data of the plurality of MD's paired with the LVD system so that only those patrons' whose MD is in a particular room or exhibit are searched to identify mobile devices for interaction with the display unit at the particular room or exhibit.

[0033] Thus, upon receiving an MD or activating the museum application on the MD, a patron sets his MD down on a first display unit in an entry portion of the museum for pairing. During pairing, the placement and orientation of the first mobile device is detected by the first display unit including detecting a location for the optical sensor. Next, the first display unit sends a variable light signal to the location of the optical sensor location. Preferably, the variable light signal is hidden from detection by a different optical sensor by the placement of the target mobile device on the first display unit. In response to the pairing, a first custom menu is displayed for the target mobile device. The first custom menu can be coordinated with the placement and orientation of the target mobile device on the first display unit. The first custom menu can include options to create a tour of the museum and have the contents of the tour transferred to the patron's MD. The options included one or more of the following: selection of an order for touring exhibits, exhibits of interest, time allotment, meal planning, and rendezvous point.

[0034] After creation of the customized tour, the patron takes the MD and begins his tour based on the selections made on the touch table. Thus, the patron can use the MD to move around the museum and explore the exhibits. During the tour, the patron may want to change the options selected for the tour, change the order or time schedule, or otherwise change the attributes of the tour. Additionally, at selected exhibits, the patron may have the option to receive additional information by further interacting with the display unit associated with that exhibit. Accordingly, at some point during the tour, the patron may find it desirable to interact with a second display unit located in a different area of the museum from the first display unit.

[0035] To facilitate interactions with such a second display unit, the LVD system determines which mobile devices, out of all the mobile devices currently paired with the LVD system, are in a predetermined geographical range of the second display unit. Generally, the predetermined geographical location can be a particular room in which the second display unit is located, a particular exhibit area in which the second display unit is located or can be a certain predefined radius around the second display unit. [0036] Any mobile devices currently in the predetermined geographical area are designated as being in a sub-group of mobile devices which can interact with the second display unit. Accordingly, this sub-group includes only those devices within the predetermined geographical range around the second display unit. Thus, the determination of the sub-group can be used to limit the number of possible mobile devices for interacting with the second display unit. Generally, the sub-group can be updated continuously or periodically to add or remove mobile devices. Thus, the sub-group can be determined prior to a mobile device being placed on the display unit.

[0037] If the patron needs to react with the second display unit, the LVD system selects the patron's MD from the sub-group of mobile devices instead of the group of all mobile devices connected to the LVD system. More specifically, the patron wishing to interact with the second display unit, places his MD on the second display unit such that the placement and orientation of the target mobile device are detected by the second display unit including detecting a location for the optical sensor. Next, a variable light signal can be sent from the second display unit to the location of the optical sensor location such that variable light signal is hidden from detection by a different optical sensor by the placement of the MD on the second display unit. Based on the light variable signal, the patron's MD is identified from the sub-group of mobile devices. Additionally, a custom menu for the MD can be displayed on the second display unit. The custom menu can be coordinated with the placement and orientation of the patron's MD on the second display unit.

[0038] The custom menu at the second display unit allows the patron to receive additional information about the exhibit or modify his tour. Thus, refinements of the tour can be made without having to return to the first display unit at the entry to the museum. [0039] In a second example, customers can install an app on their own mobile devices, which they can use to place against a kiosk (local display unit) in a mall or other sales setting. Once the kiosk recognizes the device is in place, a welcome message appears and the user has access to sales information, coupons, maps and other information related to their shopping, which can be transferred to their devices.

[0040] In the shopping mall, there can be several video display units located at the various entrances and at other convenient locations. Similar to the embodiment described for the museum, a group of all the mobile devices currently paired with the large video display system is determined. Then for each local display unit location, a sub-group of mobile devices is selected. Broadly, the sub-group is selected so as to provide a more limited group of mobile devices that are available for interaction with the local display unit than the group of all paired mobile devices. For example, the sub-group can be selected based on shared geolocation data such that the first subgroup includes only those mobile devices within a predetermined geographical range of the local display unit. After selection of the subgroup, a mobile device trying to interact with a local display unit is identified from the first sub-group by sending a variable light signal from the first large video display to the optical sensor of the target mobile device.

[0041] Pairing with the LVD system can be at activation of the application on the MD so that the geolocation data is shared before interaction with a kiosk. Alternatively, pairing can be established when the MD interacts with the first kiosk; however, this may create a slower identification of the customer's MD.

[0042] While the methods are described in terms of "comprising," "containing," or

"including" various components or steps, the methods also can "consist essentially of or "consist of the various components and steps. Also, the terms in the claims have their plain, ordinary meaning unless otherwise explicitly and clearly defined by the patentee. Moreover, the indefinite articles "a" or "an," as used in the claims, are defined herein to mean one or more than one of the elements that it introduces. If there is any conflict in the usages of a word or term in this specification and one or more patent(s) or other documents that may be incorporated herein by reference, the definitions that are consistent with this specification should be adopted.

Claims

What is claimed is:

1. A method of comprising:

determining a group of mobile devices paired with a first large video display;

selecting a first sub-group of mobile devices from the group of devices based on shared geolocation data such that the first sub-group includes only those mobile devices within a predetermined geographical range of the first large video display; and

identifying a target device from the first sub-group by sending a variable light signal from the first large video display to the optical sensor of the target mobile device.

2. The method of claim 1, wherein the first large video display is in a first room, and the predetermined geographical range is the first room such that only mobile devices in the room are included in the first sub-group of devices.

3. The method of claim 1, wherein the step of identifying a target device further comprises:

placing the target device on the first large video display such that the placement and orientation of the target device is detected by the first large video display including detecting a location for the optical sensor; and

sending the variable light signal to the location of the optical sensor such that the variable light signal is hidden from detection by a different optical sensor by the placement of the target device on the first large video display.

4. The method of claim 3, further comprising: displaying a custom menu for the target device on the first large video display, wherein the custom menu is coordinated with the placement and orientation of the target device on the first large video display.

5. The method of claim 1, wherein the first large video display is part of a system of large video display units including at least the first large video display and a second large video display, and wherein the group of mobile devices paired with the first large video display is paired with the system such that the group of mobile devices is also paired with the second large video display.

6. The method of claim 5, further comprising:

selecting a second sub-group of devices from the group of devices based on shared geolocation data such that the second sub-group of devices includes only those devices within a predetermined geographical range of the second large video display.

7. The method of claim 6, wherein the first sub-group is used to determine mobile devices eligible for interaction with the first large video display, and the second sub-group is used to determine mobile devices eligible for interaction with the second large video.

8. The method of claim 7, wherein the first large video display is in a first room, the second large video display is in a second room, the first sub-group consists of only mobile devices in the first room, and the second sub-group consists of only mobile devices in the second room.

9. A method comprising:

pairing a target mobile device of a user with a large video display system of a facility, wherein there is a plurality of mobile devices paired with the large video display system, the target mobile device has an optical sensor, the video display system includes at least a first display unit and a second display unit, and the pairing includes:

placing the target mobile device on the first display unit such that the placement and orientation of the first mobile device are detected by the first display unit including detecting a location for the optical sensor;

sending a variable light signal from the first display unit to the location of the optical sensor location such that variable light signal is hidden from detection by a different optical sensor by the placement of the target mobile device on the first display unit; and

displaying a first custom menu for the target mobile device on the first display unit, wherein the first custom menu is coordinated with the placement and orientation of the target mobile device on the first display unit, and wherein said first custom menu includes options to create a tour of the facility, and the options include one or more of the following: an order for touring exhibits, exhibits of interest, time allotment, meal planning, and rendezvous point; and thereafter

interacting the target mobile device with the second display unit.

10. The method of claim 9, further comprising determining a first portion of the plurality of mobile devices based on shared geolocation data, wherein the portion includes only those devices within a first predetermined geographical range of the first display unit, and wherein the determination of the first portion is used to limit the number possible mobile devices for interacting with the first display unit.

11. The method of claim 9, wherein the step of interacting the target mobile device with the second display unit comprises:

determining a second portion of the plurality of mobile devices based on shared geolocation data, wherein the second portion includes only those devices within a second predetermined geographical range of the second display unit, wherein the second portion includes the target mobile device; and wherein the determination of the second portion is used to limit the number of possible mobile devices for interacting with the second display unit.

12. The method of claim 11, wherein the first display unit is in a first room, the second display unit is a second room, and the second predetermined geographical range is the second room such that only mobile devices in the second room are included in the second portion of devices.

13. The method of claim 12, wherein the step of interacting the target mobile device with the second video display unit further comprises:

selecting the target mobile device from the second portion of mobile devices.

14. The method of claim 11, wherein the step of interacting the first mobile device with the second video display unit further comprises:

placing the target mobile device on the second display unit such that the placement and orientation of the target mobile device are detected by the second display unit including detecting a location for the optical sensor; sending a variable light signal from the second display unit to the location of the optical sensor location such that variable light signal is hidden from detection by a different optical sensor by the placement of the target mobile device on the second display unit;

identifying the target mobile device from the second portion of mobile devices based on the sending of the variable light signal; and

displaying a second custom menu for the target mobile device on the second display unit, wherein the second custom menu is coordinated with the placement and orientation of the target mobile device on the second display unit.

15. The method of claim 14, wherein the first display unit is in a first room, the second display unit is in a second room, and the predetermined geographical range is the second room such that only mobile devices in the second room are included in the second portion of devices.

16. The method of claim 15, further comprising determining a first portion of the plurality of mobile devices based on shared geolocation data, wherein the first portion includes only those devices within a first predetermined geographical range of the first display unit, and wherein the determination of the first portion is used to limit the number possible mobile devices for interacting with the first display unit.