HK1060231A - Electronic chat and instant messaging with ink data - Google Patents

Description

Electronic chat and instant messaging with ink data

Technical Field

The present invention relates to the field of communications using computing and/or communication devices. More particularly, the present invention relates to ink data communication between multiple parties using computing and/or communication devices on a network.

Background

Chat room services from internet service providers, as well as online services, provide an informal public meeting place where multiple participants from around the world can converse through text, audio, or through a shared whiteboard. Chat rooms may be maintained by a single server or by multiple servers connected by a network spread over a large area.

Typical chat room services depend on two basic factors of the network protocol on which the internet is based, i.e., the transmission control protocol/internet protocol (TCP/IP): a client and a server. Clients run software that allows them to connect to the server. Other implementations allow peer-to-peer chat communications. In a client/server implementation, the server accepts connections of one or more clients at the same time. Multiple servers are typically interconnected. Clients may access conferences and users on other servers from one server. Multiple servers provide a support structure that allows chat room services to work. The server maintains information of currently available chat rooms. Whenever a new chat room is created, information about it is passed to the other various servers on the network. The server also manages which clients are currently connected and which options and features they have established. When such information changes, all such information is exchanged between the servers. A typical chat system, Internet Relay Chat (IRC), is described in internet RFC1495 "internet relay chat protocol".

Current chat room services provide little support for non-ASCII characters, which makes it difficult to communicate in languages other than english, such as ideographic languages. In addition, current chat room services provide little support for sharing graphical or ink data messages between clients. Conventional ink data refers to a simple set of information about the motion of a pen device on an electronic tablet.

In the past, some services have allowed multiple clients to send graphical messages between clients using a shared whiteboard. Each client system has a camera directed to the whiteboard at the client location. The image captured by the camera will be sent to the server. The server superimposes the images received by each camera on each other and sends the new images to the respective clients for display. "Video Draw by John c.tang and Scott l.minneman" printed in CHI' 90 Conference Proceedings: a Video Interface For a collaborative driver "describes such a system: video mapping on Xerox PARC (VideoDraw at Xerox PARC).

There are several disadvantages to using a shared whiteboard. One drawback to utilizing a shared whiteboard is that linear conversations cannot be easily conducted over a shared graphics space. When two or more clients share a whiteboard, it is difficult for a person to determine which client is posting to the shared whiteboard, and to keep track of the chronological order in which the postings are made. Another disadvantage of using a shared whiteboard is that after the whiteboard is filled, the clients have to wait for other clients to wipe off the content on their whiteboard before the next contribution can be made.

One special case of utilizing chat is traditional instant messaging, whereby one client can send and receive messages to and from another client on the fly. Traditionally, Instant Messaging (IM) is used to quickly communicate short text messages between two networked users. Longer or more complex messages or documents may be communicated between network users in a less real-time manner using conventional e-mail. Conventional email systems support the delivery of documents structured in a variety of formats, including text, graphics, GIF, JPEG, bitmap, EXE, and many others. However, the traditional IM and chat infrastructure is more limited to a very few formats, mainly ASCII text and/or voice formats.

With the growth of the IM and chat user base, it will become increasingly important to support data types other than simple ASCII text and/or voice. Currently, one such data type that is not supported by conventional IM or chat systems is a handwritten data type, such as ink data. The use of ink data is particularly important in applications where traditional textual data is not very efficient. For example, many languages other than English utilize symbols, wherein drawing symbols with hands is more efficient than typing on a text-oriented keyboard. Also for example, some portable devices such as Personal Digital Assistants (PDAs), cellular telephones, handheld devices, pagers, etc., may be more efficiently utilized with pen input devices. Pen input devices are more efficient at inputting ink data than conventional text unless handwriting recognition software is utilized. However, handwriting recognition software typically consumes many system resources.

Therefore, there is a need for a means for incorporating non-traditional data types, such as ink data, into traditional IM and chat infrastructures.

In addition, there is a need for a method and apparatus for enabling ink data communication between multiple parties using computers on a network.

Drawings

The present invention is illustrated by way of example, and not by way of illustration, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

FIG. 1 illustrates an exemplary block diagram of a computer system implementing one embodiment of the invention;

figure 2 illustrates a data distribution system implementing one embodiment of the present invention using a unicast protocol.

Figure 3 illustrates a data distribution system implementing one embodiment of the present invention using a multicast protocol.

Figure 4 illustrates the functional structure and process flow of one embodiment of the present invention.

Detailed Description

A method and apparatus for enabling communication of ink data between parties using computing and/or communication devices on a network is disclosed. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that these specific details need not be employed to practice the present invention. In other instances, well-known structures, materials, circuits, processes, and interfaces have not been shown or described in detail to avoid unnecessarily obscuring the invention.

Referring to FIG. 1, an exemplary computer system that may be implemented in accordance with an embodiment of the invention is shown at 100. Computer system 100 includes a processor or CPU 101 that processes digital data. Processor 101 may be a Complex Instruction Set Computing (CISC) microprocessor, Reduced Instruction Set Computing (RISC) microprocessor, Very Long Instruction Word (VLIW) microprocessor, processor implementing a combination of instruction sets, or other processor device. Processor 101 is connected to a CPU bus 110 that transfers data between processor 101 and other components in computer system 100.

For the illustrated embodiment, memory 113 includes Dynamic Random Access Memory (DRAM) devices, Static Random Access Memory (SRAM) devices, or other memory devices. The memory 113 stores information or other intermediate data during execution by the processor 101. The bridge memory controller 111 connects the CPU bus 110 and the memory 113. Bridge memory controller 111 controls data traffic between processor 101, memory 113, and other components in computer system 100 and bridges signals from these components to high speed I/O bus 120.

For the illustrated embodiment, the high speed I/O bus 120 supports peripheral operations at high data throughput rates. Bus 120 may be a single bus or a combination of multiple buses. By way of illustration, bus 120 may include a Peripheral Component Interconnect (PCI) bus, a Personal Computer Memory Card International Association (PCMCIA) bus, or other bus. Bus 120 provides a communications link between the components in computer system 100.

A network interface 121 is connected to bus 120. The network interface is used to link the computer system 100 to a server or a network of computers and to provide communication between these machines. The network interface 121 may be a telephone modem, cable modem, Integrated Services Digital Network (ISDN) connection, or other network interface. A display device controller 122 is connected to the high speed I/O bus 120. A display device controller 122 is connected to the bus 120. The display device controller 122 allows the display device to be connected to the computer system and serves as an interface between the display device and the computer system 100. The display device receives information and data from processor 101 through display device controller 122 and displays such information and data to a user of computer system 100. A graphical input device 124 is connected to the bus 120. The graphical input device 124 is used to input graphical images and/or handwritten or ink data to the computer system 100. The graphical input device 124 may be, for example, a camera and whiteboard or electronic pen and tablet or other graphical input device. Graphical input devices 124 may also include conventional handwriting input devices such as a touch screen, a digitizer, an electronic tablet, a mouse, an electronic pen, a light pen, and the like.

In the depicted embodiment, bus bridge 123 connects a high speed I/O bus 120 to an I/O bus 130. Bus bridge 123 includes a translator to bridge signals between high speed I/O bus 120 and I/O bus 130. The I/O bus 130 is used to pass signals between peripherals operating at lower throughput rates. The I/O bus 130 may be a single bus or a combination of multiple buses. By way of illustration, the bus 130 may comprise an Industry Standard Architecture (ISA) bus, an Extended ISA (EISA) bus, or a Micro Channel Architecture (MCA) bus. Bus 130 provides a communications link between the components in computer system 100. The data storage device 131 may be a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device. Keyboard interface 132 allows a keyboard to connect to computer system and sends signals from the keyboard to computer system 100. An optional audio controller 133 for coordinating the recording and playing of sounds is also connected to the I/O bus 130. A pointing device, such as a conventional mouse or trackball, may also be connected to I/O bus 130.

The present invention relates to the use of computer system 100 to enable ink data communication between multiple parties on a network. According to one embodiment, computer system 100 operates to enable ink data communications in response to processor 101 executing sequences of instructions contained in memory 113. These instructions may be read into memory 113 from other computer-readable media, such as data storage device 131, or from a network. Execution of the sequences of instructions contained in memory 113 causes the processor to perform the functions described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, the present invention is not limited to any specific combination of hardware circuitry and software. It will be apparent to those of ordinary skill in the art that the present invention may be used with a wide variety of other computing and/or communication devices, such as chat devices, Personal Digital Assistants (PDAs), palm-top computers, or the like.

Fig. 2 illustrates a data distribution system implementing the present invention using a unicast protocol. Data distribution system 200 is a network that includes a plurality of clients 100 and 201 and 204. Clients 100 and 201-204 may be implemented by a computer system such as the one shown in FIG. 1. The data distribution system 200 also includes a server 210. Each client maintains a separate connection with the server 210. Lines 221-225 represent the connections between clients 100 and 201-204, respectively, and server 210. For example, connection 221-225 may be an Internet connection using Transmission Control Protocol (TCP) or User Datagram Protocol (UDP). According to one embodiment of the invention, graphical or ink data messages generated from a client are sent to the server 210 in the form of data packets over the client's connection. The graphical or ink data message is sent to a message queue in the server 210 where other graphical or ink data messages from other clients are stored. Copies of the graphical or ink data messages received from the various clients are then sent from the server 210 to the clients 100 and 201 in the order in which they were received 204. Clients 100 and 201 store the graphical or ink data messages received from server 210 in their own receive message queues 204. Thus, each client receives a copy of the message sent to the server 210 in the order in which the server 210 receives the messages. Clients 100 and 201 display the graphical or ink data message packets stored in their receive message queues on a display device 204.

According to one embodiment of the invention, server 210 may be connected to a plurality of other servers in a similar configuration to form a network of servers that share graphics or ink data. According to another embodiment of the present invention, connection 221-225 may be a non-Internet connection over a Local Area Network (LAN) or a Wide Area Network (WAN). The server 210 and clients 100 and 201 and 204 may be implemented by any known circuitry. It should be appreciated that data distribution system 200 may be a single computer system and a plurality of clients coupled to the computer system that may communicate with each other using interprocess communication.

Figure 3 illustrates a data distribution system implementing one embodiment of the present invention using a multicast protocol, such as the multicast internet protocol. Data distribution system 300 is a network that includes a plurality of clients 100 and 201 and 204. Clients 100 and 201-204 may be implemented by a computer system such as the one shown in FIG. 1. Clients 100 and 201 and 204 are connected to transmission medium 310. The transmission medium 310 may be, for example, an internet network, or a Local Area Network (LAN) or a Wide Area Network (WAN). According to one embodiment of the present invention, a first graphical or ink data message is transmitted as a graphical or ink data packet from a first client 100 to a plurality of other clients 201 and 204 on a data distribution system 300 via a transmission medium 310. Each of the clients 100 and 201 and 204 has a received message queue in which messages sent from other clients are stored in the order of reception. A client that sends a message, such as client 100, also stores copies of the message it sent to other clients in its received message queue. Client 201-204 also executes a similar protocol when sending graphical or ink data messages. Each client 100 and 201-204 may display the graphical or ink data messages stored in its received message queue on a display device.

One embodiment of the present invention is designed to augment current chat and instant messaging infrastructures, allowing users to communicate with ink data obtained through graphical or handwriting input devices, such as touch screens, digitizers, electronic tablets, mice, light pens, and the like. The use of simple ink data is well known in the art. For example, it is well known to capture handwritten input in electronic form by capturing information related to the motion of an electronic pen on an electronic tablet. This type of information is commonly referred to as ink data. In conventional techniques, ink data is simply represented as the x, y position of the pen in a particular time increment. In more sophisticated conventional ink data capture devices, the capture device captures and reports other parameters to the processing device through a conventional interface, including hand position, time, pen pressure, style, color, and other attributes related to pen motion. While these ink data capture devices exist in the prior art, their use in instant messaging or chat systems is not known or suggested.

Conventional chat systems or instant messaging systems have traditionally been used primarily to transfer text from one network user to another during a chat session. Many traditional instant messaging people have added voice transfer capabilities, but no one has now provided or suggested the use of ink data formats in instant messaging or chat sessions. However, the ink data format is particularly well suited for integration with instant messaging or chat infrastructure for at least the following reasons: first, the ink data format is an ideal tool for sharing creative activities such as drawings, architecture, and design. Second, the ink data format is an ideal tool to facilitate communication with avatar content, such as indicating directions on a map, changing the layout of a floor plan, and the like. Third, the ink data format is the most efficient way for people in many countries outside the united states to communicate. Even in the united states, there are millions of people who cannot type at a fast enough speed, but for them handwriting is often a very natural and comfortable communication medium.

Referring now to FIG. 4, a block diagram illustrates the basic architecture of one embodiment of the present invention. Input device 410 represents any of a variety of conventional electronic pen input devices, handwriting input devices, and other types of conventional ink data capture devices. These input devices 410 generate raw ink data inputs that generally include x-coordinate, y-coordinate, and time information from an ink data capture device. The ink data input device 410 operates with a hardware interface software layer or driver 420 to format the raw ink data into a format suitable for processing at a higher functional level. The x, y coordinate information is used to determine the location of the electronic pen tip at a particular time. Using the time information provided by the input device 410, the handwritten strokes of the electronic pen may be reproduced with high accuracy, thereby increasing the intelligibility and aesthetic impact of the representation of the ink data. This improved intelligibility applies to human and machine based handwriting recognition. If a more complex ink data input device 410 is utilized, the input device 410 may provide additional information such as pen pressure, drawing style or line type information, color information, and other types of ink data information to the hardware interface layer 420. It will be apparent to those of ordinary skill in the art that providing additional ink data parameters will improve the rendering and presentation of ink data information by a remote user.

Once the hardware interface layer 420 combines the ink data into the original ink data packet, the packet is passed to the ink management layer 430 as shown in FIG. 4. Generally, the ink management layer 430 performs initial processing of the raw ink data received from the hardware interface layer 420. Depending on the type of data received from the input device 410 and the amount of data, it may be desirable to compress or compact the raw ink data received through the hardware interface layer 420. For example, using conventional techniques, the ink management layer 430 may filter out copied original ink data or insert or smooth out original ink data that may have lost some data. In addition, the ink management layer 430 may utilize conventional data compression techniques to reduce the amount of data transferred to higher functional levels without reducing the information represented by the original ink data. As an additional functional level, the ink management layer 430 may also encrypt the original ink data using well-known techniques. In addition, the ink management layer 430 may apply conventional handwriting recognition techniques to convert the ink data into textual form. With the configuration user interface provided by the present invention, any of the following processing functions may be selectively provided or disabled: such as compact, encryption, and handwriting recognition. The ink management layer 430 also converts the raw ink data into a network transmittable format. Upon completion of its processing of the input raw ink data, the ink management layer 430 generates processed ink data, which is passed to the network interface layer 440. The network interface layer 440 converts the processed ink data into a form compatible with the local network communication protocol. In some conventional systems, the network interface layer 440 may not be needed if the computing device implementing the present invention already has an established internet connection. These devices, such as desktop personal computers, may already be capable of transmitting processed ink data over a network. However, if such functionality is not implemented or not available to the particular system utilizing the present invention, the network interface layer 440 may convert the processed ink data into a form that may be transmitted over a network to a remote user. Once the network interface layer 440 produces a network-transferable form of processed ink data, the network-transferable ink data is transferred to a messaging layer 450, such as an Instant Messaging (IM) interface layer 450. The interface layer 450 provides the conventional system infrastructure for transferring instant messages between two users on a computer network. The present invention extends the conventional instant messaging interface layer 450 to include the transfer of processed ink data using conventional instant messaging infrastructure. In this manner, a user at a location on the network having an input device 410 that generates ink data may communicate the ink data to a remote network user via the instant messaging infrastructure 450. It will be apparent to those of ordinary skill in the art that other conventional messaging infrastructures may be similarly utilized. For example, interface layer 450 may also be an interface layer that implements chat functionality. In this embodiment, the network interface layer 440 will provide the chat interface layer with processed ink data that the layer sends to other message recipients during a particular chat session. Meanwhile, the invention expands the traditional chat interface layer, thereby being capable of transmitting the processed ink data to other chat receivers. In another embodiment of the present invention, the interface layer 450 may be a conventional email delivery system. In this case, the network interface layer 440 provides the processed ink data to an email interface layer that incorporates the processed ink data into a conventional email message and transmits the email message with embedded ink data to a remote network user by employing conventional email transmission mechanisms. At the same time, an extension to the conventional system is necessary for the following cases: enabling processed ink data to be embedded in a conventional email message for transmission by an email application. Thus, the generation and transmission of ink data using a conventional chat or email infrastructure is described.

Referring again to FIG. 4, the illustrated embodiment of the invention includes functionality on the receiver side for receiving chat or instant messages that include ink data. On the receiver side, the interface layer 450 will receive messages from remote users using conventional techniques. Additionally, the interface layer 450 may be an instant messaging interface, a chat interface, or an email interface. Using any of these conventional tools, a receiver at the interface layer 450 receives a message that includes processed ink data. Next, the incoming ink data message is passed to the network interface layer 440. The network interface layer 440 unwraps the input ink data message and extracts the ink data. Ink data packets received via the incoming message are passed from the network interface layer 440 to the ink management layer 430. The ink management layer 430 is responsible for decoding the incoming ink data messages. According to the ink data processing performed by the message sender, if the sender has encrypted the ink data message, the receiver-side ink management layer 430 will decrypt the input ink data. In addition, the ink management layer 430 on the receiver side will decompress the received ink data using conventional techniques. The ink management layer 430 will also apply conventional handwriting recognition techniques to the ink data. The ink management layer 430 on the receiver side will also further process the received ink data to facilitate rendering of the ink data on a display device 460 on the receiver side, if desired. Once the ink management layer 430 on the receiver side has processed the input ink data, the decoded ink data is transferred to the hardware interface layer 420 on the receiver side. The hardware interface layer 420 on the receiver side also formats the ink data for compatibility with the particular display device 460 on the receiver side. The hardware interface layer 420 then transmits the received ink data to the display device 460 for display to the network user at the receiving end. Thus, ink data captured or generated by a networked sender is transmitted to a network receiver via conventional instant messaging, chat, and/or email infrastructure for display at the receiver.

Accordingly, a method and apparatus for enabling ink data communication between multiple parties using computing and/or communication devices on a network is disclosed. Although the present invention has been described herein with reference to certain preferred embodiments, numerous modifications and variations will readily occur to those skilled in the art. Accordingly, all such variations and modifications are intended to be included within the scope of the invention as defined in the following claims.

Claims (21)

1. An electronic chat system, comprising:

a hardware interface layer for receiving raw ink data from an input device;

the ink management layer is used for processing the original ink data into a form usable by the chat interface layer; and

and the network interface layer is used for transmitting the processed ink data to the chat interface layer on the remote computer through a network.

2. The electronic chat system of claim 1 wherein the chat interface layer further comprises an instant messaging layer for receiving the processed ink data.

3. The electronic chat system of claim 1, further comprising:

a chat subsystem for displaying the processed ink data as part of a multi-user chat session.

4. The electronic chat system of claim 1, further comprising:

an instant messaging subsystem for displaying the processed ink data as part of an instant messaging session.

5. The electronic chat system of claim 1 wherein the ink management layer further comprises a data compression component for compressing the raw ink data.

6. The electronic chat system of claim 1 wherein the ink management layer further comprises a data encryption component for encrypting the raw ink data.

7. The electronic chat system of claim 1, further comprising:

a receiver for receiving processed ink data transmitted by a remote computer system and for converting said received ink data into a form displayable as part of a chat session.

8. The electronic chat system of claim 7, further comprising:

and the data decompression component is used for decompressing the ink data received from the network.

9. The electronic chat system of claim 7, further comprising:

and the data decryption component is used for decrypting the ink data received from the network.

10. The electronic chat system of claim 1 wherein the ink management layer further comprises a handwriting recognition component.

11. A method, comprising:

receiving raw ink data from an input device;

processing the original ink data into a form usable by a chat interface layer; and

the processed ink data is transmitted over a network to a chat interface layer on a remote computer.

12. The method of claim 11, wherein the chat interface layer further comprises an instant messaging layer for receiving the processed ink data.

13. The method of claim 11, further comprising:

displaying the processed ink data as part of a multi-user chat session.

14. The method of claim 11, further comprising:

displaying the processed ink data as part of an instant messaging session.

15. The method of claim 11, further comprising compressing the raw ink data.

16. The method of claim 11, further comprising encrypting the original ink data.

17. The method of claim 11, further comprising:

processed ink data sent by a remote computer system is received and the received ink data is converted into a form that can be displayed as part of a chat session.

18. The method of claim 17, further comprising:

decompressing ink data received from the network.

19. The method of claim 17, further comprising:

decrypting the ink data received from the network.

20. The method of claim 11, further comprising performing handwriting recognition on the received ink data.

21. An article of manufacture comprising a computer readable medium on which processing instructions are encoded, the processing instructions comprising:

a hardware interface component for receiving raw ink data from an input device;

an ink management component for processing the raw ink data into a form usable by the chat interface component; and

a network interface component for transmitting the processed ink data over a network to a chat interface component on a remote computer.