WO2012153167A1

WO2012153167A1 - System and method for real-time transmission of multimedia messages

Info

Publication number: WO2012153167A1
Application number: PCT/IB2011/052028
Authority: WO
Inventors: Danny KOK
Original assignee: Talkbox Limited
Priority date: 2011-05-09
Filing date: 2011-05-09
Publication date: 2012-11-15
Also published as: TW201246972A; CN102780687A

Abstract

A method for real-time transmission of a multimedia message is disclosed. The method records the multimedia message upon receiving a start indication from a sender at a sender device. The method then divides the multimedia message into a plurality of packets, streams the plurality of packets from a sender device to a server during recording of the multimedia message. The method then determines whether at least one recipient of the multimedia message is connected to the server. If the recipient is connected to the server, the method then streams the plurality of packets from the server to a recipient device during recording of the multimedia message. A system for real-time transmission of a multimedia message is also disclosed.

Description

SYSTEM AND METHOD FOR REAL-TIME TRANSMISSION OF

MULTIMEDIA MESSAGES

FIELD OF INVENTION

[0001] This invention relates to a system and method for transmission of data, and in particular a system and method for transmission of multimedia messages.

BACKGROUND OF INVENTION

[0002] Conventional systems for transmission of multimedia messages usually send the whole message at once. Such systems have too much delay for achieving real-time transmission of multimedia messages.

SUMMARY OF INVENTION

[0003] In the light of the foregoing background, it is an object of the present invention to provide an alternate system and method for real-time transmission of multimedia messages.

[0004] Accordingly, the present invention, in one aspect, is a method for real-time transmission of a multimedia message. The method records the multimedia message upon receiving a start indication from a sender at a sender device. The method then divides the multimedia message into a plurality of packets, streams the plurality of packets from a sender device to a server during recording of the multimedia message. The method then determines whether at least one recipient of the multimedia message is connected to the server. If the recipient is connected to the server, the method then streams the plurality of packets from the server to a recipient device during recording of the multimedia message.

[0005] In an exemplary embodiment of the present invention, the start indication is a detection of the sender's finger at a predetermined start location. Recording is continued while said end indication is not received. The end indication is a detection of the user's finger leaving a predetermined continuing location. [0006] In another exemplary embodiment, the method further allows playing of the multimedia message at the recipient device during recording of the multimedia message.

[0007] In another implementation, the method further transcodes the multimedia message from a recorded format to a predetermined format of the sender device before the dividing step, and transcodes the multimedia message from the predetermined format to a playable format of the recipient device after the streaming step from the server.

[0008] In another exemplary embodiment, the method further receives a cancel indication from the sender device. The cancel indication is then transmitted to the at least one recipient device through the server, and the data packets received at the recipient device are deleted.

[0009] According to another aspect of the present invention, a system for real-time transmission of multimedia messages is disclosed. The system comprises a sender device comprising a multimedia input mechanism, at least one recipient device, and a server adapted to connect to the sender device and the recipient device. The sender device records the multimedia message and divides the multimedia message into a plurality of packets. The sender device further streams the plurality of packets to the server. The server determines whether at least one recipient of the multimedia message is connected to the server, and streams the plurality of packets to the recipient device if said recipient is connected to said server, during recording of said multimedia message.

[0010] In one embodiment, the server further stores the multimedia message if at least one recipient is not connected to the server.

[0011] In an exemplary embodiment, the sender device and the at least one recipient device comprise a transcoding module for transcoding the multimedia message from a device dependent format to a predetermined format for optimized transmission.

[0012] There are many advantages to the present invention. A main advantage of the present invention is that the multimedia message is streamed from the sender device to the recipient device while the sender is still recording the multimedia message. Comparing with conventional methods, when the sender finishes the recording of the multimedia message, a large portion of the message has already been streamed to the recipient, while conventional methods needs to wait for the end of the recording before starting to transmit the message to the recipients. The present invention also allows the recipient to play the message while the sender is still recording the message, further being more real-time.

[0013] Another advantage of the present invention is that the streaming of a message can be cancelled upon sender's action. This feature is useful for the sender to call back undesired messages when necessary during the recording of the message. The received data packets of the multimedia message are deleted from the recipient devices and further playback is prevented.

BRIEF DESCRIPTION OF FIGURES

[0014] Fig. 1 is a block diagram of a system for real-time transmission of multimedia messages, according to an exemplary embodiment of the present invention.

[0015] Fig. 2 is an exemplary user interface of a device in the system of Fig. 1.

[0016] Fig. 3 is a flow chart showing a method for real-time transmission of multimedia messages, according to an exemplary embodiment of the present invention.

[0017] Fig. 4 is an exemplary user interface showing cancellation of sending of a multimedia message, according to an embodiment of the present invention.

[0018] Fig. 5 is an exemplary packet structure of a multimedia message, according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] As used herein and in the claims, "comprising" means including the following elements but not excluding others.

[0020] As used herein and in the claims, "connect" refers to connect through either a wired or wireless network connection unless otherwise stated.

[0021] Referring now to Fig. 1, the first embodiment of the present invention is a system comprising a sender device 20, a central server 22 and at least one recipient device 24. In various embodiments, the sender device 20 and the recipient device 24 can be smart phones, personal computers, or other devices with data input/output and data transmission capabilities. In an exemplary embodiment, the sender device 20 and the recipient device 24 both have a display 26 for showing a user interface 28.

[0022] Fig. 2 shows an exemplary user interface 28 according to an embodiment of the present invention. The user interface 28 comprises a list of avatars 30 representing a plurality of users for example. In an embodiment, the user interface 28 of the sender device 20 shows a plurality of recipients, while the user interface of the recipient device 24 shows the sender along with other recipients. That is, the sender and all the recipients can see each other, and they are said to have formed a conversation.

[0023] The user interface 28 also comprises a dialog box 31 with a time indicator 32 beside each avatar 30. The dialog box 31 shows whether the user is currently recording a multimedia message at his device, and the time indicator 32 shows the current length of the recorded multimedia message. The user interface 28 also comprises a "Hold to Talk" button 34 at the bottom of the screen in this specific embodiment as shown. This will be explained in more detail below.

[0024] Fig. 3 shows a flow chart for transmission of a multimedia message according to an exemplary embodiment of the present invention. The sender device 20 first determines whether a start indication is received (step 100). In an exemplary embodiment, the start indication is the detection of the sender's finger touching the "Hold to Talk" button 34 as shown in Fig. 2. In another embodiment, the start indication is the detection of the sender's finger pressing a physical start button or record button.

[0025] Upon receiving the start indication, the sender device 20 then starts recording the multimedia message or "message" (step 102). In an exemplary embodiment, the multimedia message is an audio message. In other embodiments, the multimedia message can be a video message, an image, or a combination of any kind of multimedia content. In one embodiment, the audio message is recorded through a microphone and the video message is recorded through a digital camera.

[0026] The sender device 20 then determines whether an end indication is received (step 104). In an exemplary embodiment, the end indication is the detection of the sender's finger releasing the touch screen. In other words, as long as the sender's finger is still touching the touch screen, the recording of the multimedia message will continue. This is also called a push-to-talk mechanism.

[0027] Before the end indication is received, and while the recording of the multimedia message continues, the sender device 20 divides the multimedia message into a plurality of data packets, or in other words, packetizes the message (step 106). The sender device 20 then streams the data packets to the server 22 as each data packet is created (step 108). The server 22 then determines each recipient of the message specified in the data packet is connected to the server 22 (step 110). If the recipient is connected to the server 22, the server 22 then relays the data packets to the recipient device 24 (step 112). While the recording of the multimedia message continues, the time indicator 32 beside the sender at the user interfaces 28 of both the sender device 20 and the recipient devices 24 will be updated with the current recorded time with each data packet received.

[0028] When the end indication is received, the sender device 20 stops the recording of the multimedia message (step 114). The end indication shows up at the user interface 28 of both the sender device 20 and the recipient devices 24. The sender device 20 then streams the remaining data packets to the recipient 24 through the server 22 in a similar manner as mentioned above (step 116). [0029] Once the entire multimedia message is received at the recipient device 24, the recipient device 24 groups all data packets received into a complete multimedia message for playing (step 118). The multimedia message can be played automatically or upon receiving a command from the recipient.

[0030] In an alternative embodiment of the present invention, the recipient device 24 plays the multimedia message to the recipient before the end indication and all data packets are received, i.e. continuous playback. The recipient device 24 determines whether continuous playback is possible based on the speed of the internet connection and also the amount of buffered data in the recipient device 24. After the end indication and all data packets have been received, the recipient device 24 still groups the multimedia message together for storing in the recipient device 24. A recipient on the recipient device 24 has the option to choose between this configuration and the configuration as mentioned above.

[0031] In an exemplary embodiment, before dividing the multimedia message into data packets and transmitting the data packets to the server 22, the sender device 20 first transcodes the multimedia message from a recorded format into a predetermined format to optimize the transmission. In a specific embodiment, the predetermined format is iLBC (Internet Low Bit Rate Codec) format for audio messages. Data in iLBC format has a relatively low sampling rate and bit rate, therefore the amount of data transmitted is minimized while keeping the audio message at a comprehensible quality. Similarly, when the data packets reach the recipient device 24, the recipient device 24 transcodes the data packets into a playable format of the recipient device 24.

[0032] The service platform for different devices may be different. As such, the recorded format and the playable format as mentioned above may also be different. In such situation, transcoding the data into a common format for transmission is even more advantageous, as each device also does not need to care about the service platform of other devices since the received format is always the same, e.g. iLBC. The device only needs to be adapted to transcode the data between the device's preferred format and iLBC, instead of the need to accommodate for every single format for different service platforms. [0033] In an exemplary embodiment, the multimedia message recorded is saved in the sender device 20 and each recipient device 24. In a further embodiment, if the recipient device 24 is not connected to the server 22 at the time of recording of the multimedia message, the multimedia message will be stored in the server 22 in the predetermined format, e.g. iLBC. When the server 22 determines that the recipient 24 is connected to the server 22, the server 22 then sends the multimedia message to the recipient 24.

[0034] In an exemplary embodiment, the sender on the sender device 20 first determines a list of recipients before transmitting the multimedia message to the recipient devices 24. In one embodiment, the sender is able to add additional recipient devices 24 to the conversation at any time, and the recipient device 24 will join the conversation after receiving the first multimedia message from the sender device 20. In one embodiment, the maximum number of users in a single conversation is nine.

[0035] In one embodiment, recipients of a multimedia message are selected from a contact list of the sender device 20. In various embodiments, the contact list can be generated by synchronizing with a contact list in other programs such as social networking services, directly adding a user through entering the user's name or email address, mapping the address book of the sender device 20, or a combination of the above.

[0036] The contact list of a device may comprise other entities more than users. In an exemplary embodiment, the contact list also comprises "agents" with specific functionalities. For example, an agent can link to a social networking site, such that when the agent is selected as a recipient of the multimedia message, the multimedia message will show up at the social network site as a link to the message. Other functionalities include interacting with other users in a different way e.g. broadcasting messages, subscribing to specific channels, providing business functions through integrating with enterprises etc. In one embodiment, the multimedia message is sent to servers of the social networking site or specific channels, or a specific module in said sender device for broadcasting messages or providing business functions. [0037] In an exemplary embodiment, a geographic location of the sender device 20 is transmitted to the recipient devices 24 along with the multimedia message. In a further embodiment, the geographic location is gathered via location service provided by the platform operating system. The location of the sender device 20 is then shown at the recipient devices 24 through a map service application upon receiving an indication by the recipient on the recipient device 24.

[0038] In an exemplary embodiment, the sending of a multimedia message can be cancelled by the sender. The sender device 20 first receives a cancel indication from the sender. In one embodiment, the cancel indication is a detection of the sender's finger releasing from the touch screen at a predetermined location, e.g. a location showing a microphone icon 36, as shown in Fig. 4. In another embodiment, the cancel indication is a detection of sender pressing on another physical button simultaneous to the recording button, and releasing both buttons at the same time.

[0039] Upon receiving the cancel indication, the sender device 20 transmits the cancel indication to the server 22. The server 22 then relays the cancel indication to the recipient devices 24. The recipient devices 24 then displays that the multimedia message is cancelled, and then automatically removes all received data packets for the multimedia message. In one embodiment, the dialog box 31 and the time indicator 32 of the avatar 30 representing the sender is removed to show that the multimedia message is cancelled. In one embodiment, all other information transmitted with the multimedia message e.g. the geographic location of the sender device 20 is also removed automatically.

[0040] In an exemplary embodiment, if continuous playback is selected by the recipient, once the cancel indication is received by the recipient device 24, the continuous playback is stopped immediately and all received data packets are removed, even though the recipient device 24 has already played a portion of the multimedia message before receiving the cancel indication.

[0041] In an exemplary embodiment, each multimedia message is made up of a plurality of data packets, as shown in Fig. 5. Each data packet 37 consists of a header 38 and a body 40. The header comprises information to the message such as a message ID, sender ID and receiver IDs, timestamp, geographical data and message chuck number for ordering the data packets. The body comprises start or end flags, or the actual multimedia binary data.

[0042] The multimedia message starts with a message start flag 42. The message data body 44 follows the message start flag 42. In an exemplary embodiment, the message data body 44 comprises a plurality of TCP packets 46, each with a TCP packet header 48 and a TCP packet body 50. Each TCP packet header 48 comprises a total number of TCP packets in the message data body 44, a packet token randomly generated and dedicated to the message, a packet chunk number and a packet data length. The TCP packet body 50 contains the multimedia binary data with length as specified by the packet data length.

[0043] After the message body data 44, a message end flag 52, a metadata start flag 54, a metadata body 56 and a metadata end flag follows to complete the entire multimedia message. In one embodiment, the metadata body 56 contains all non-streamable contents, e.g. images, geographic location, text, markups etc., while the message data body contains all streamable content, e.g. video or audio data.

[0044] In one embodiment, the message header 38 further contains a receiver type information, identifying whether the multimedia message belongs to a group conversation or not. In another embodiment, the message header 38 further contains a client local ID information for identifying any transmission failures of the multimedia message.

[0045] In an exemplary embodiment, the sender device 20 is allowed to send multimedia messages in real-time while using a third-party application. The third-party application needs to be incorporated with the present invention, for example a hold-to-talk button 34 is displayed to the sender on the third-party application. The sender can then stream the multimedia message in real-time to selected recipients whenever the sender sees something worth sharing when browsing on the third-party application. In one embodiment, information regarding the third-party application, such as any links provided on the display of the sender, is also contained in the header of the data packets. [0046] The exemplary embodiments of the present invention are thus fully described. Although the description referred to particular embodiments, it will be clear to one skilled in the art that the present invention may be practiced with variation of these specific details. Hence this invention should not be construed as limited to the embodiments set forth herein.

Claims

What is claimed is:

1. A method for real-time transmission of a multimedia message, comprising the steps of:

a) recording said multimedia message upon receiving a start indication from a sender at a sender device;

b) during recording of said multimedia message, dividing said multimedia message into a plurality of data packets;

c) during recording of said multimedia message, streaming said plurality of data packets from a sender device to a server;

d) determining whether at least one recipient of said multimedia message is connected to said server;

e) during recording of said multimedia message, streaming said plurality of data packets from said server to a recipient device, if said recipient is connected to said server.

2. The method according to claim 1, further comprising the steps of:

a) stopping said recording of said multimedia message upon receiving an end indication; and

b) continuing said streaming of said plurality of data packets until a last packet of said multimedia message has been streamed to said recipient device, if said recipient is connected to said server.

3. The method according to claim 2, wherein said start indication is a detection of said sender's finger at a predetermined start location, and said recording is continued while said end indication is not received, said end indication being a detection of said user's finger leaving a predetermined continuing location.

4. The method according to claim 1 , further comprising the step of during said recording of said multimedia message, allowing playing of said multimedia message at said recipient device.

5. The method according to claim 1, further comprising the step of transcoding said multimedia message from a recorded format into a predetermined format of said sender device before said step b), and the step of transcoding said multimedia message from said predetermined format to a playable format of said recipient device after said step e).

6. The method according to claim 1, further comprising the step of storing said multimedia message at said server if at least one said recipient is not connected to said server, said method further comprising the step of sending said multimedia message to said at least one recipient upon detecting said at least one recipient being connected to said server.

7. The method according to claim 1, further comprising the steps of:

a) receiving a cancel indication at said sender device;

b) transmitting said cancel indication to said at least one recipient device through said server;

c) deleting said packets received at said at least one recipient device.

8. The method according to claim 7, wherein said cancel indication is a detection of said sender's finger releasing from a predetermined cancel location.

9. The method according to claim 1, wherein a geographic location of said sender is sent to said at least one recipient along with said multimedia message.

10. The method according to claim 1, wherein said at least one recipient comprises a user, a predetermined social network, a specific channel, a predetermined function, or a combination of the above.

11. The method according to claim 1 , wherein each said data packet comprises a header and a body, said header comprises a message ID, a sender ID, a receiver ID, a geographic location of said sender and a message chunk number.

12. The method according to claim 11, wherein said plurality of data packets comprise a message data body packet, said body of said message data body packet further comprises a plurality of TCP packets each comprising a TCP packet header and a TCP packet body, said TCP packet header comprises a total number of TCP packets in said message data body packet, a packet token dedicated to said multimedia message, a packet chunk number and a packet data length.

13. A system for real-time transmission of multimedia messages comprising:

a) a sender device comprising a multimedia input mechanism; b) at least one recipient device;

c) a server adapted to connect to said sender device and said at least one recipient device;

wherein said sender device records said multimedia message and divides said multimedia message into a plurality of data packets, said sender device further streams said plurality of packets to said server; said server determines whether at least one recipient of said multimedia message is connected to said server, and streams said plurality of packets to said recipient device if said recipient is connected to said server, during recording of said multimedia message.

14. The system according to claim 13, wherein said server further stores said multimedia message if at least one said recipient is not connected to said server.

15. The system according to claim 13, wherein said sender device and said at least one recipient device comprise a transcoding module for transcoding said multimedia message between a device dependent format and a predetermined format for optimized transmission.

16. The system according to claim 13, wherein said at least one recipient device deletes said data packets upon receiving a cancel indication from said sender device.

17. The system according to claim 13, wherein said at least one recipient device comprises a user device, a social networking site server, a specific channel server, a specific module for performing a predetermined function, or a combination of the above.

18. The system according to claim 13, wherein each said data packet comprises a header and a body, said header comprises a message ID, a sender ID, a receiver ID, a geographic location of said sender and a message chunk number.

19. The system according to claim 13, wherein said plurality of data packets comprise a message data body packet, said body of said message data body packet further comprises a plurality of TCP packets each comprising a TCP packet header and a TCP packet body, said TCP packet header comprises a total number of TCP packets in said message data body packet, a packet token dedicated to said multimedia message, a packet chunk number and a packet data length.

14