US20180129295A1

US20180129295A1 - Gesture-based control and usage of video relay systems

Info

Publication number: US20180129295A1
Application number: US15/687,716
Authority: US
Inventors: Dennis Episkopos; Cliff Pearce; Francis Kurupacheril
Original assignee: Purple Communications Inc
Current assignee: Purple Communications Inc
Priority date: 2016-08-15
Filing date: 2017-08-28
Publication date: 2018-05-10
Also published as: US20180129294A1; US10423237B2; US20180048859A1

Abstract

A system comprising a web-enabled video camera, at least one microphone, a remote server, and a command processing unit, which allow users to communicate remotely with video and audio. The system further is controlled by either gestures from a user, or a remote control, and can have gestures customized by the user for specific inputs. The system contains a menu the user may navigate, and multiple system states which provide different functionality and recognize different gesture commands.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No. 15/677,877 titled “GESTURE-BASED CONTROL AND USAGE OF VIDEO RELAY SYSTEMS”, filed on Aug. 15, 2017, which claims the benefit of, and priority to, U.S. provisional patent application Ser. No. 62/375,243, titled “GESTURE BASED CONTROL AND USAGE OF VIDEO RELAY SYSTEMS” and filed on Aug. 15, 2016, the entire specification of which is herein incorporated by reference in its entirety.

BACKGROUND OF THE INVENTION

Field of the Art

The disclosure relates to the field of video relay systems for hearing-impaired users, and more particularly to the field of gesture-controlled video relay systems.

Discussion of the State of the Art

It is currently available for deaf individuals to communicate with other individuals across distances with computer systems (by which it is meant a computing device which contains a central processing unit, memory, and interface elements such as a keyboard) that include a remote control, and allow them to input commands and information so that the deaf individual may communicate with other users and even with translators. Currently, the systems in place are cumbersome to their users in that the remote may be lost easily, which makes the system unusable. Further, the systems take several steps with a time-consuming interface in order for the user to communicate, and typing in letters on a QWERTY keypad on a small device can be painful for some users. Remotes in this field also only possess the capability to execute a certain set of commands, and cannot be programmed for new commands or capabilities by the user, and require an independent energy source such as batteries.
In these cases, the user is inconvenienced and encumbered in such a way as to make communication take more time and energy from the user of the system. Communication can be time-sensitive and being required to use a system which may be disabled due to loss of a remote or lack of batteries, or which causes discomfort from small keypads, can hinder communication.
Therefore, what is clearly needed is a system that does not rely on a small, easily misplaced remote or keypad, does not require separate batteries, and does not cause the user physical discomfort or difficulty in using the system.

SUMMARY OF THE INVENTION

Accordingly, the inventor has conceived and reduced to practice, in a preferred embodiment of the invention, a system and various methods for gesture-based control and usage of video relay systems, which allow a deaf and possibly mute user to communicate to other users across the Internet, with a system possessing several modes of interaction and requiring no tactile interaction from the user. The following non-limiting summary of the invention is provided for clarity, and should be construed consistently with embodiments described in the detailed description below.
To address the problem of the remote described above for communication systems, American Sign Language (ASL) gesture recognition is provided in a novel system that provides clear program flow for a user to initiate communications at a distance with another user. Instead of being required to solely use a remote, the system is capable of recognizing hand gestures as well as ASL-specific commands that the user makes in front of a connected web-enabled camera (colloquially known as a “webcam”), allowing the user to use the software with easily learned motions if they do not wish to use the remote control. Common examples of motions that a user may take advantage of are pointing their thumb to their left while facing the webcam, which takes them back to a previous menu option in the system, or being able to define certain gestures that may be used to hang up a call. Use of these and more gestures allows users to naturally and easily use the system, without the need for a small, battery-enabled remote, however such a remote is still supplied with the system in case users may desire to use the remote anyway.
According to an aspect of the invention, a system for gesture-based control and usage of video relay systems, comprising: a command processing unit comprising at least a processor, a memory, and a plurality of programming instructions stored in the memory and operating on the processor, wherein the programmable instructions, when operating on the processor, cause the processor to: receive a first video stream from a connected camera hardware device; analyze at least a portion of the first video stream to identify a plurality of hand gestures; correlate at least a portion of the identified hand gestures with a plurality of electronic commands; transmit at least a portion of the first video stream via a network; receive a second video stream via a network; and direct the navigation of at least a software-based menu interface, the navigation being based at least in part on at least a portion of the electronic commands, is disclosed.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The accompanying drawings illustrate several aspects and, together with the description, serve to explain the principles of the invention according to the aspects. It will be appreciated by one skilled in the art that the particular arrangements illustrated in the drawings are merely exemplary, and are not to be considered as limiting of the scope of the invention or the claims herein in any way.

FIG. 1 is a block diagram illustrating an exemplary system architecture for gesture recognition and communication, according to an aspect of the invention.

FIG. 2 is a flow diagram illustrating an exemplary embodiment of system states for the software, according to an aspect of the invention.

FIG. 3 is a flow diagram illustrating an exemplary method for turning on the system using gesture recognition, according to an aspect of the invention.

FIG. 4 is a flow diagram illustrating an exemplary method for the main menu of the system, according to an aspect of the invention.

FIG. 5 is a flow diagram illustrating an exemplary architecture for the Main Menu, showing the various items it may have available for the user, according to an aspect of the invention.

FIG. 6 is a flow diagram illustrating an exemplary method for the QWERTY Mode of the system, according to an aspect of the invention.

FIG. 7 is a flow diagram illustrating an exemplary method for the Dial Mode of the system, according to an aspect of the invention.

FIG. 8 is a flow diagram illustrating an exemplary method for the Connection Mode of the system, according to an aspect of the invention.

FIG. 9 is a flow diagram illustrating an exemplary method for the Privacy Mode of the system, according to an aspect of the invention.

FIG. 10 is a block diagram illustrating an exemplary hardware architecture of a computing device used in various aspects of the invention.

FIG. 11 is a block diagram illustrating an exemplary logical architecture for a client device, according to an aspect of the invention.

FIG. 12 is a block diagram showing an exemplary architectural arrangement of clients, servers, and external services, according to an aspect of the invention.

FIG. 13 is another block diagram illustrating an exemplary hardware architecture of a computing device used in various aspect of the invention.

DETAILED DESCRIPTION

The inventor has conceived, and reduced to practice, in a preferred embodiment of the invention, a system and various methods for gesture-based control and usage of video relay systems.
One or more different aspects may be described in the present application. Further, for one or more of the aspects described herein, numerous alternative arrangements may be described; it should be appreciated that these are presented for illustrative purposes only and are not limiting of the aspects contained herein or the claims presented herein in any way. One or more of the arrangements may be widely applicable to numerous aspects, as may be readily apparent from the disclosure. In general, arrangements are described in sufficient detail to enable those skilled in the art to practice one or more of the aspects, and it should be appreciated that other arrangements may be utilized and that structural, logical, software, electrical and other changes may be made without departing from the scope of the particular aspects. Particular features of one or more of the aspects described herein may be described with reference to one or more particular aspects or figures that form a part of the present disclosure, and in which are shown, by way of illustration, specific arrangements of one or more of the aspects. It should be appreciated, however, that such features are not limited to usage in the one or more particular aspects or figures with reference to which they are described. The present disclosure is neither a literal description of all arrangements of one or more of the aspects nor a listing of features of one or more of the aspects that must be present in all arrangements.
Headings of sections provided in this patent application and the title of this patent application are for convenience only, and are not to be taken as limiting the disclosure in any way.
Devices that are in communication with each other need not be in continuous communication with each other, unless expressly specified otherwise. In addition, devices that are in communication with each other may communicate directly or indirectly through one or more communication means or intermediaries, logical or physical.
A description of an aspect with several components in communication with each other does not imply that all such components are required. To the contrary, a variety of optional components may be described to illustrate a wide variety of possible aspects and in order to more fully illustrate one or more aspects. Similarly, although process steps, method steps, algorithms or the like may be described in a sequential order, such processes, methods and algorithms may generally be configured to work in alternate orders, unless specifically stated to the contrary. In other words, any sequence or order of steps that may be described in this patent application does not, in and of itself, indicate a requirement that the steps be performed in that order. The steps of described processes may be performed in any order practical. Further, some steps may be performed simultaneously despite being described or implied as occurring non-simultaneously (e.g., because one step is described after the other step). Moreover, the illustration of a process by its depiction in a drawing does not imply that the illustrated process is exclusive of other variations and modifications thereto, does not imply that the illustrated process or any of its steps are necessary to one or more of the aspects, and does not imply that the illustrated process is preferred. Also, steps are generally described once per aspect, but this does not mean they must occur once, or that they may only occur once each time a process, method, or algorithm is carried out or executed. Some steps may be omitted in some aspects or some occurrences, or some steps may be executed more than once in a given aspect or occurrence.
When a single device or article is described herein, it will be readily apparent that more than one device or article may be used in place of a single device or article. Similarly, where more than one device or article is described herein, it will be readily apparent that a single device or article may be used in place of the more than one device or article.
The functionality or the features of a device may be alternatively embodied by one or more other devices that are not explicitly described as having such functionality or features. Thus, other aspects need not include the device itself.
Techniques and mechanisms described or referenced herein will sometimes be described in singular form for clarity. However, it should be appreciated that particular aspects may include multiple iterations of a technique or multiple instantiations of a mechanism unless noted otherwise. Process descriptions or blocks in figures should be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process. Alternate implementations are included within the scope of various aspects in which, for example, functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those having ordinary skill in the art.

Conceptual Architecture

FIG. 1 is a block diagram illustrating an exemplary system 100 interacting with a user 110 to capture hand gestures for long-range communication, according to an aspect of the invention. According to the aspect, a web-enabled camera 120 may capture video of a user 110, and images captured from the webcam may be sent to command processing unit 130 that may be configured to transmit video and audio to another remote user 150. According to the aspect, a computing device 100 typically comprises at least a memory 11 and a processor 12 (as described below, referring to FIG. 10), configured to operate a software operating system (OS), for example including (but not limited to) ANDROID™, APPLE IOS™, WINDOWS™, or various forms of LINUX™.
Command processing unit 130 may then recognize gestures from the images provided by the webcam 120, recognizing commands from a user, using image recognition. Various implementations of image recognition software may be used according to various use cases or arrangements. These gestures can provide, according to a preferred aspect of the invention, different functionality depending on the state of the system, as shown below. When the system is in Connection Mode 240 as in FIG. 2, which can be activated according to methods described in aspects detailed below, the images captured by the webcam 120 of a user 110 may be sent to a remote server 140. This remote server can run various kinds of Session Initiation Protocol (SIP) software operating with call routing rules according to various arrangements, as is common in the art. A user may then, through such a server, communicate visually and audibly with another remote user 110, and every user involved in a communication will see each other's captured video displayed through video output 160, such as a TV through an HDMI connection, an embedded laptop monitor, or other video output devices in use today and the future with computing devices that may run the system described herein.

Detailed Description of Exemplary Aspects

The system may begin in a listening state 220 as shown in FIG. 2 and, according to an aspect shown in FIG. 3, an implementation of the system may await a repetition of gestures 310 that may be defined or configured by a user. Such gestures may be determined upon installation of the software as in an initial configuration as is common with some types of software, or may be configured in a main menu, as a configuration application 561 as in FIG. 5, as one of several applications capable of being run by the system. When a user produces these gestures 310 with a short delay between each repetition 320, the system initializes and turns on all other components 330 as shown in FIG. 3. When initialized, the system enters 340 a main menu 210 as shown in FIG. 3 and FIG. 2. According to an aspect, the system may use at least one web-enabled camera 120 to detect these gestures as shown in FIG. 1. Upon entering main menu 210, a user is given a list of menu items to choose from according to an aspect in FIG. 5. Users may use many different gestures when on the Main Menu 210, listed on FIG. 4, to either produce functionality on their own, or to navigate the menu items available to users. According to an aspect of the invention, a user may produce gestures 410 which may be repeated 420, defined either in a configuration utility on installing the system as is common with many software applications and systems, or with a configuration utility which may be available as an application 561 as shown in FIG. 5. According to a preferred aspect of the invention, the system may enter QWERTY Mode 230, 421 if the delay between these gestures is decidedly short, or it may enter dial mode 422, as shown in FIG. 4.
Alternatively, according to an aspect, a user may make a thumb gesture 430 or substitute a different ASL sign depending on system configuration, and this will cause the menu cursor to move in the specified direction by the hand sign 431 according to a preferred aspect as shown in FIG. 4. A user may also, according to an aspect of the invention described in FIG. 4, point forward to the screen with their finger 440, and the generated action would select the highlighted menu item 441 and execute the selected item (for example, if an application were highlighted, the application would start running on the computer system). A user may also, according to an aspect of the invention described in FIG. 4, point their thumb straight out to the left 450, and the generated action would be to go back to the previous item in the menu 451, and this motion has further functionality in further system states as well, according to preferred aspects, in that it may return the user to a previous screen 451.
Further according to the aspect, a user may, as described in FIG. 4, point their thumb to themselves or away from themselves 460, and the generated action would be to adjust the focus of the web camera 461, either closer or farther away based on the direction of the thumb. A user may also, according to an aspect of the invention described in FIG. 4, produce a gesture configured in the system 470, either during system installation, or by running a configuration utility 561, and the gesture can perform three different functions 471. If a user is being called by another user, the gesture will answer the call 471. If a user is currently speaking to another user, it will hang up the call 471. Lastly, if neither a call is taking place nor a user is being called, the gesture may allow a user to place a call instead, directly from main menu 471.
According to an aspect of the invention, a main menu 210 may have several different items listed for a user to choose from. As shown in FIG. 5, a user may choose from contacts 520, in which the event generated will list on the screen all the contacts 521 that are saved by the user. Further, the same user may then add another contact 522, according to a preferred aspect of the invention. A user could also, according to a preferred aspect of the invention specified in FIG. 5, choose history 530, from the main menu, which will allow a user to view their call connection history 531, listing both incoming and outbound calls made. A user could also, according to a preferred aspect of the invention specified in FIG. 5, choose mail 540, which will initiate a mail interface 541 for users. A user could also, according to a preferred aspect of the invention specified in FIG. 5, choose a search option 550, which, if executed, can search a user's contacts, or their mail, or search online 551 such as via a service such as YELP™. Lastly, a user may also, according to an aspect of the invention described in FIG. 5, choose apps 560, which will bring up many different applications that a user may choose from 561, which may include such things as a configuration utility or other applications as decided by the vendor.
According to further aspects of the invention, when a user may enter QWERTY Mode 610 from the main menu 421, a user may then produce two gestures for additional functionality.
A user may, according to a preferred aspect of the invention as specified in FIG. 6, produce ASL gestures 630, 631, to produce commands to the system 632 without the use of a remote. A user may further motion to the left with their thumb 620, and, as mentioned earlier in the disclosure, this will return a user to the main menu 621.
According to further aspects of the invention, when a user may enter dial mode 710 from main menu 422, a user may then produce two gestures for additional functionality. A user may, according to a preferred aspect of the invention as specified in FIG. 7, produce ASL numeric gestures 730, 731, to dial numbers and call others 732 without the use of a remote or number pad. A user may further motion to the left with their thumb 720, and, as mentioned earlier in the disclosure, this will return a user to main menu 721.
According to further aspects of the invention, when a user may enter a connection mode 240 from main menu 470, 732, as shown in FIG. 4 and FIG. 7 when a user desire to place a call or dials a number to call, they may further use two gestures to produce further functionality 810 as shown in FIG. 8. If a user snaps their fingers 830, the user may enter a privacy mode 831, as illustrated in FIG. 2 in the state diagram of the system, 250. If, instead, a user executes a repeated configured gesture 820, with a short delay between the repetitions 821, any call the user is currently in hangs up and the user returns to the initial listening mode 822.
Further aspects of the invention illustrated in FIG. 9 show privacy mode 250. In privacy mode a user may snap their fingers 830 only once 910, to turn video on and off 911, thereby preserving privacy. Further, a user may snap their fingers twice 921 with a small delay between each snap 920, to mute and unmute their audio 922. Privacy mode is an extension of connection mode, and hanging up a call is the same as in connection mode as shown in FIG. 8, 820, 821, 822.

Hardware Architecture

Generally, the techniques disclosed herein may be implemented on hardware or a combination of software and hardware. For example, they may be implemented in an operating system kernel, in a separate user process, in a library package bound into network applications, on a specially constructed machine, on an application-specific integrated circuit (ASIC), or on a network interface card.
Referring now to FIG. 10, there is shown a block diagram depicting an exemplary computing device 10 suitable for implementing at least a portion of the features or functionalities disclosed herein. Computing device 10 may be, for example, any one of the computing machines listed in the previous paragraph, or indeed any other electronic device capable of executing software- or hardware-based instructions according to one or more programs stored in memory. Computing device 10 may be configured to communicate with a plurality of other computing devices, such as clients or servers, over communications networks such as a wide area network a metropolitan area network, a local area network, a wireless network, the Internet, or any other network, using known protocols for such communication, whether wireless or wired.
In one embodiment, computing device 10 includes one or more central processing units (CPU) 12, one or more interfaces 15, and one or more busses 14 (such as a peripheral component interconnect (PCI) bus). When acting under the control of appropriate software or firmware, CPU 12 may be responsible for implementing specific functions associated with the functions of a specifically configured computing device or machine. For example, in at least one embodiment, a computing device 10 may be configured or designed to function as a server system utilizing CPU 12, local memory 11 and/or remote memory 16, and interface(s) 15. In at least one embodiment, CPU 12 may be caused to perform one or more of the different types of functions and/or operations under the control of software modules or components, which for example, may include an operating system and any appropriate applications software, drivers, and the like.
CPU 12 may include one or more processors 13 such as, for example, a processor from one of the Intel, ARM, Qualcomm, and AMD families of microprocessors. In some embodiments, processors 13 may include specially designed hardware such as application-specific integrated circuits (ASICs), electrically erasable programmable read-only memories (EEPROMs), field-programmable gate arrays (FPGAs), and so forth, for controlling operations of computing device 10. In a specific embodiment, a local memory 11 (such as non-volatile random access memory (RAM) and/or read-only memory (ROM), including for example one or more levels of cached memory) may also form part of CPU 12. However, there are many different ways in which memory may be coupled to system 10. Memory 11 may be used for a variety of purposes such as, for example, caching and/or storing data, programming instructions, and the like. It should be further appreciated that CPU 12 may be one of a variety of system-on-a-chip (SOC) type hardware that may include additional hardware such as memory or graphics processing chips, such as a QUALCOMM SNAPDRAGON™ or SAMSUNG EXYNOS™ CPU as are becoming increasingly common in the art, such as for use in mobile devices or integrated devices.
As used herein, the term “processor” is not limited merely to those integrated circuits referred to in the art as a processor, a mobile processor, or a microprocessor, but broadly refers to a microcontroller, a microcomputer, a programmable logic controller, an application-specific integrated circuit, and any other programmable circuit.
In one embodiment, interfaces 15 are provided as network interface cards (NICs). Generally, NICs control the sending and receiving of data packets over a computer network; other types of interfaces 15 may for example support other peripherals used with computing device 10. Among the interfaces that may be provided are Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, graphics interfaces, and the like. In addition, various types of interfaces may be provided such as, for example, universal serial bus (USB), Serial, Ethernet, FIREWIRE™, THUNDERBOLT™, PCI, parallel, radio frequency (RF), BLUETOOTH™, near-field communications (e.g., using near-field magnetics), 802.11 (Wi-Fi), frame relay, TCP/IP, ISDN, fast Ethernet interfaces, Gigabit Ethernet interfaces, Serial ATA (SATA) or external SATA (ESATA) interfaces, high-definition multimedia interface (HDMI), digital visual interface (DVI), analog or digital audio interfaces, asynchronous transfer mode (ATM) interfaces, high-speed serial interface (HSSI) interfaces, Point of Sale (POS) interfaces, fiber data distributed interfaces (FDDIs), and the like. Generally, such interfaces 15 may include physical ports appropriate for communication with appropriate media. In some cases, they may also include an independent processor (such as a dedicated audio or video processor, as is common in the art for high-fidelity A/V hardware interfaces) and, in some instances, volatile and/or non-volatile memory (e.g., RAM).
Although the system shown in FIG. 10 illustrates one specific architecture for a computing device 10 for implementing one or more of the inventions described herein, it is by no means the only device architecture on which at least a portion of the features and techniques described herein may be implemented. For example, architectures having one or any number of processors 13 may be used, and such processors 13 may be present in a single device or distributed among any number of devices. In one embodiment, a single processor 13 handles communications as well as routing computations, while in other embodiments a separate dedicated communications processor may be provided. In various embodiments, different types of features or functionalities may be implemented in a system according to the invention that includes a client device (such as a tablet device or smartphone running client software) and server systems (such as a server system described in more detail below).
Regardless of network device configuration, the system of the present invention may employ one or more memories or memory modules (such as, for example, remote memory block 16 and local memory 11) configured to store data, program instructions for the general-purpose network operations, or other information relating to the functionality of the embodiments described herein (or any combinations of the above). Program instructions may control execution of or comprise an operating system and/or one or more applications, for example. Memory 16 or memories 11, 16 may also be configured to store data structures, configuration data, encryption data, historical system operations information, or any other specific or generic non-program information described herein.
Because such information and program instructions may be employed to implement one or more systems or methods described herein, at least some network device embodiments may include nontransitory machine-readable storage media, which, for example, may be configured or designed to store program instructions, state information, and the like for performing various operations described herein. Examples of such nontransitory machine-readable storage media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto-optical media such as optical disks, and hardware devices that are specially configured to store and perform program instructions, such as read-only memory devices (ROM), flash memory (as is common in mobile devices and integrated systems), solid state drives (SSD) and “hybrid SSD” storage drives that may combine physical components of solid state and hard disk drives in a single hardware device (as are becoming increasingly common in the art with regard to personal computers), memristor memory, random access memory (RAM), and the like. It should be appreciated that such storage means may be integral and non-removable (such as RAM hardware modules that may be soldered onto a motherboard or otherwise integrated into an electronic device), or they may be removable such as swappable flash memory modules (such as “thumb drives” or other removable media designed for rapidly exchanging physical storage devices), “hot-swappable” hard disk drives or solid state drives, removable optical storage discs, or other such removable media, and that such integral and removable storage media may be utilized interchangeably. Examples of program instructions include both object code, such as may be produced by a compiler, machine code, such as may be produced by an assembler or a linker, byte code, such as may be generated by for example a JAVA™ compiler and may be executed using a Java virtual machine or equivalent, or files containing higher level code that may be executed by the computer using an interpreter (for example, scripts written in Python, Perl, Ruby, Groovy, or any other scripting language).
In some embodiments, systems according to the present invention may be implemented on a standalone computing system. Referring now to FIG. 11, there is shown a block diagram depicting a typical exemplary architecture of one or more embodiments or components thereof on a standalone computing system. Computing device 20 includes processors 21 that may run software that carry out one or more functions or applications of embodiments of the invention, such as for example a client application 24. Processors 21 may carry out computing instructions under control of an operating system 22 such as, for example, a version of MICROSOFT WINDOWS™ operating system, APPLE OSX™ or iOS™ operating systems, some variety of the Linux operating system, ANDROID™ operating system, or the like. In many cases, one or more shared services 23 may be operable in system 20, and may be useful for providing common services to client applications 24. Services 23 may for example be WINDOWS™ services, user-space common services in a Linux environment, or any other type of common service architecture used with operating system 21. Input devices 28 may be of any type suitable for receiving user input, including for example a keyboard, touchscreen, microphone (for example, for voice input), mouse, touchpad, trackball, or any combination thereof. Output devices 27 may be of any type suitable for providing output to one or more users, whether remote or local to system 20, and may include for example one or more screens for visual output, speakers, printers, or any combination thereof. Memory 25 may be random-access memory having any structure and architecture known in the art, for use by processors 21, for example to run software. Storage devices 26 may be any magnetic, optical, mechanical, memristor, or electrical storage device for storage of data in digital form (such as those described above, referring to FIG. 10). Examples of storage devices 26 include flash memory, magnetic hard drive, CD-ROM, and/or the like.
In some embodiments, systems of the present invention may be implemented on a distributed computing network, such as one having any number of clients and/or servers. Referring now to FIG. 12, there is shown a block diagram depicting an exemplary architecture 30 for implementing at least a portion of a system according to an embodiment of the invention on a distributed computing network. According to the embodiment, any number of clients 33 may be provided. Each client 33 may run software for implementing client-side portions of the present invention; clients may comprise a system 20 such as that illustrated in FIG. 11. In addition, any number of servers 32 may be provided for handling requests received from one or more clients 33. Clients 33 and servers 32 may communicate with one another via one or more electronic networks 31, which may be in various embodiments any of the Internet, a wide area network, a mobile telephony network (such as CDMA or GSM cellular networks), a wireless network (such as Wi-Fi, WiMAX, LTE, and so forth), or a local area network (or indeed any network topology known in the art; the invention does not prefer any one network topology over any other). Networks 31 may be implemented using any known network protocols, including for example wired and/or wireless protocols.
In addition, in some embodiments, servers 32 may call external services 37 when needed to obtain additional information, or to refer to additional data concerning a particular call. Communications with external services 37 may take place, for example, via one or more networks 31. In various embodiments, external services 37 may comprise web-enabled services or functionality related to or installed on the hardware device itself. For example, in an embodiment where client applications 24 are implemented on a smartphone or other electronic device, client applications 24 may obtain information stored in a server system 32 in the cloud or on an external service 37 deployed on one or more of a particular enterprise's or user's premises.
In some embodiments of the invention, clients 33 or servers 32 (or both) may make use of one or more specialized services or appliances that may be deployed locally or remotely across one or more networks 31. For example, one or more databases 34 may be used or referred to by one or more embodiments of the invention. It should be understood by one having ordinary skill in the art that databases 34 may be arranged in a wide variety of architectures and using a wide variety of data access and manipulation means. For example, in various embodiments one or more databases 34 may comprise a relational database system using a structured query language (SQL), while others may comprise an alternative data storage technology such as those referred to in the art as “NoSQL” (for example, HADOOP CASSANDRA™, GOOGLE BIGTABLE™, and so forth). In some embodiments, variant database architectures such as column-oriented databases, in-memory databases, clustered databases, distributed databases, or even flat file data repositories may be used according to the invention. It will be appreciated by one having ordinary skill in the art that any combination of known or future database technologies may be used as appropriate, unless a specific database technology or a specific arrangement of components is specified for a particular embodiment herein. Moreover, it should be appreciated that the term “database” as used herein may refer to a physical database machine, a cluster of machines acting as a single database system, or a logical database within an overall database management system. Unless a specific meaning is specified for a given use of the term “database”, it should be construed to mean any of these senses of the word, all of which are understood as a plain meaning of the term “database” by those having ordinary skill in the art.
Similarly, most embodiments of the invention may make use of one or more security systems 36 and configuration systems 35. Security and configuration management are common information technology (IT) and web functions, and some amount of each are generally associated with any IT or web systems. It should be understood by one having ordinary skill in the art that any configuration or security subsystems known in the art now or in the future may be used in conjunction with embodiments of the invention without limitation, unless a specific security 36 or configuration system 35 or approach is specifically required by the description of any specific embodiment.
FIG. 13 shows an exemplary overview of a computer system 40 as may be used in any of the various locations throughout the system. It is exemplary of any computer that may execute code to process data. Various modifications and changes may be made to computer system 40 without departing from the broader scope of the system and method disclosed herein. Central processor unit (CPU) 41 is connected to bus 42, to which bus is also connected memory 43, nonvolatile memory 44, display 47, input/output (I/O) unit 48, and network interface card (NIC) 53. I/O unit 48 may, typically, be connected to keyboard 49, pointing device 50, hard disk 52, and real-time clock 51. NIC 53 connects to network 54, which may be the Internet or a local network, which local network may or may not have connections to the Internet. Also shown as part of system 40 is power supply unit 45 connected, in this example, to a main alternating current (AC) supply 46. Not shown are batteries that could be present, and many other devices and modifications that are well known but are not applicable to the specific novel functions of the current system and method disclosed herein. It should be appreciated that some or all components illustrated may be combined, such as in various integrated applications, for example Qualcomm or Samsung system-on-a-chip (SOC) devices, or whenever it may be appropriate to combine multiple capabilities or functions into a single hardware device (for instance, in mobile devices such as smartphones, video game consoles, in-vehicle computer systems such as navigation or multimedia systems in automobiles, or other integrated hardware devices).
In various embodiments, functionality for implementing systems or methods of the present invention may be distributed among any number of client and/or server components. For example, various software modules may be implemented for performing various functions in connection with the present invention, and such modules may be variously implemented to run on server and/or client components.
The skilled person will be aware of a range of possible modifications of the various embodiments described above. Accordingly, the present invention is defined by the claims and their equivalents.

Claims

What is claimed is:

1. A system for gesture-based control and usage of video relay systems, comprising:

a command processing unit comprising at least a processor, a memory, and a plurality of programming instructions stored in the memory and operating on the processor, wherein the programmable instructions, when operating on the processor, cause the processor to:

receive a first video stream from a connected camera hardware device;

analyze at least a portion of the first video stream to identify a plurality of hand gestures;

correlate at least a portion of the identified hand gestures with a plurality of electronic commands;

transmit at least a portion of the first video stream via a network;

receive a second video stream via a network; and

direct the navigation of at least a software-based menu interface, the navigation being based at least in part on at least a portion of the electronic commands.

2. The system of claim 1, wherein the command processing unit is further configured to navigate multiple system states based at least in part on at least a portion of the electronic commands.

3. The system of claim 1, wherein the command processing unit is further configured to direct the operation of a phone dialer based at least in part on at least a portion of the electronic commands.

4. The system of claim 1, wherein the electronic commands may be configured by a user and stored for future use.

5. A method for gesture based control and usage of video relay systems, comprising the steps of:

(a) receiving, at a command processing unit, a first video stream captured from a camera hardware device;

(b) analyzing at least a portion of the first video stream to identify a plurality of hand gestures;

(c) processing at least a portion of the hand gestures to produce a plurality of electronic commands;

(d) transmitting at least a portion of the first video stream via a network;

(e) directing the navigation of a software-based menu interface, the navigation being based at least in part on at least a portion of the electronic commands;

6. The method of claim 5, further comprising the step of navigating multiple system states based at least in part on at least a portion of the electronic commands.

7. The method of claim 5, wherein the command processing unit is further configured to direct the operation of a phone dialer based at least in part on at least a portion of the electronic commands.

8. The method of claim 5, wherein the electronic commands may be configured by a user and stored for future use.