GB2526330A

GB2526330A - Method and apparatus for synchronizing the display of video data on two display devices

Info

Publication number: GB2526330A
Application number: GB1409048.4A
Authority: GB
Inventors: Romain Guignard; Pascal Viger; Stã Phane Baron
Original assignee: Canon Inc
Current assignee: Canon Inc
Priority date: 2014-05-21
Filing date: 2014-05-21
Publication date: 2015-11-25
Anticipated expiration: 2034-05-21
Also published as: GB201409048D0; GB2526330B

Abstract

In a method and a device for synchronizing the display of video data on different display devices, a source device provides the same video data stream to the display devices through a communication network. One of the display devices (first display screen device) is provided with picture acquisition (camera) capabilities. By taking a distant picture of the second device display and comparing it with the local picture being displayed at the same (acquisition) moment or time on the first display, these images may be localized within a sequence. The time drift between the two different displays may be determined, allowing a command to modify a display timer to be provided to the source device or display itself. Precision may be improved by iterating the process with dichotomous delay applied to at least one display. Drift determination using local and distant (other display captured) images may comprise determining distance (in terms of pictures / frames) between points of interest (POI) in respective pictures; POIs may be successive predefined patterns. The first display may be the image content source, providing a form of screen mirroring between portable devices, such as tablets and mobiles, and large screen TVs or projectors.

Description

METHOD AND APPARATUS FOR SYNCHRONIZING THE DISPLAY OF

VIDEO DATA ON TWO DISPLAY DEVICES

The present invention concerns a method and a device for synchronizing the display of video data on different display devices. A source device provides the same video data stream to the display devices through a communication network. One of the display devices is provided with picture acquisition capabilities.

Nowadays, most of people own a smartphone or a tablet. This kind of devices groups much functionality for entertainment: photos, videos, music, games, web and so on. Moreover, due to their connectivity (3G, 4G, wi-fi), users can access a lot of contents coming from Internet. Although, the first aim of those handheld devices was the mobility, they now supplant the computer in many usages even in home environment. However, despite all the advantages of those devices, the users would like to take advantage of their different applications on a large screen like TV set or video projector. To make up for this lack, the manufacturers have developed application allowing the copy of the smartphone or tablet screen on a big screen thanks to the wireless connectivity.

This trendy application is named screen mirroring.

Unfortunately, the mirroring applications suffer with a delay between both screens displaying. This delay may be really disturbing for high interactivity application such as gaming, remote control and so on. This synchronization issue is the result of the time or latency difference of the full path between source and sink for both displays. This latency value takes into account the encoding time, the transmission time, the decoding time, and the input lag of the display. The input lag is the time between the picture arrival, for instance at the HDMI port of the display, and the effective display of this picture. In a typical TV, a lot of processing is performed to improve the quality and fluidity of the video.

For instance, to convert input video frame rate, typically 30 or 60 Hz, to the display frame rate of at least 100 or 200 Hz. The buffering of pictures is necessary to perform interpolation between frames meaning that at least two pictures are buffered. Thereby, new frames are created in order to achieve the display frame rate. In this scenario, the problem of synchronization is due to the different video path for a local and distant display comprising the transmission time.

An extension of the default mirroring application set up with one video source having its own display and one video display, could be done by having the source being separated from the handheld. The video source, a still camera or a video camera for example, broadcasts the video toward both displays. The first one, a handheld device like smartphone or tablet, may be used as a remote controller of the source. And the second one may be a large screen to have a convenient display. In this second embodiment wherein the smartphone is used as a remote controller to pilot the streaming from the source to the wireless display, the problem of synchronization leads to difference between the picture displayed onto the smartphone and the picture displayed onto the wireless display. This default becomes really visible when the user applies a pause to the video streaming and both frozen pictures are different on the smartphone and on the TV. In this case, the transmission and encoding time can be considered as equivalent for both displays. Actually, the video source is shared and the localization of both sinks is typically closed hence we can consider an equal travel time. So the synchronization issue is mainly due to the input lag and the buffering of the receiver. Whatever the reason, the problem of synchronization remains the same.

The present invention has been devised to address one or more of the foregoing concerns.

According to a first aspect of the invention there is provided a method of synchronizing the display of video data on at least two display devices, the same video data stream being provided by a source device to the display devices through a communication network, one of the display devices being provided with picture acquisition capabilities, the method comprising by a first display device comprising picture acquisition capabilities: acquiring using the picture acquisition capabilities a distant picture corresponding to a picture displayed by a second display device; identifying a local picture displayed by the first display device at the time of acquisition of the distant picture; determining a time drift between the first and the second display devices using the local and the distant pictures; and providing a command to modify a display timer in the data stream provided to at least one of the display devices based on the determined time drift. Accordingly, the video on both screens may be synchronized easily without requiring extra device. The measurement is transparent for the user.

In an embodiment, the command to modify the display timer is provided to the source device.

In an embodiment, the command to modify the display timer is provided to a display device.

In an embodiment, determining a time drift between the first and the second display devices using the local and the distant pictures comprises: determining points of interest in the distant picture; determining points of interest in the local picture; and determining the distance in term of pictures between the local picture and the distant picture in the sequence of pictures composing the video data stream.

In an embodiment, the sequence of pictures composing the video data stream is composed of successive predefined patterns for an easy determination of the points of interest.

In an embodiment, the time of acquisition of the distant picture is chosen at a predefined time of the display of local pictures.

In an embodiment, the precision of the determined time drift is improved by iterating on: providing to the source device a command to modify a display timer in the data stream provided to at least one of the display devices based on dichotomous time drifts; acquiring using the picture acquisition capabilities a distant picture corresponding to a picture displayed by a second display device; identifying a local picture displayed by the first display device at the time of acquisition of the distant picture; and determining a time drift between the first and the second display devices using the local and the distant pictures.

In an embodiment, the time of acquisition of the distant picture is chosen to be the middle of time of display of local pictures.

In an embodiment, the first display device comprises the source device.

In an embodiment, the synchronization process is repeated periodically.

Accordingly, any drift occurring after a first synchronization may be corrected.

According to another aspect of the invention there is provided a first display device comprising: picture acquisition capabilities; a synchronization module for synchronizing the display of video data on said first display device and on a second display device, the same video data stream being provided by a source device to the display devices through a communication network; an acquisition module for acquiring using the picture acquisition capabilities a distant picture corresponding to a picture displayed by a second display device; an identifying module for identifying a local picture displayed by the first display device at the time of acquisition of the distant picture; a time drift module for determining a time drift between the first and the second display devices using the local and the distant pictures; and a command module for providing a command to modify a display timer in the data stream provided to at least one of the display devices based on the determined time drift.

In an embodiment, the command module provides the command to modify the display timer to the source device.

In an embodiment, the command module provides the command to modify the display timer to a display device.

In an embodiment, the time drift module comprises: a module for determining points of interest in the distant picture; a module for determining points of interest in the local picture; and a module for determining the distance in term of pictures between the local picture and the distant picture in the sequence of pictures composing the video data stream.

In an embodiment, the device comprises for improving the precision of the determined time drift an iteration module for iterating on: providing to the source device a command to modify a display timer in the data stream provided to at least one of the display devices based on dichotomous time drifts; acquiring using the picture acquisition capabilities a distant picture corresponding to a picture displayed by a second display device; identifying a local picture displayed by the first display device at the time of acquisition of the distant picture; and determining a time drift between the first and the second display devices using the local and the distant pictures.

In an embodiment, the first display device comprises the source device.

In an embodiment, the device comprises a module for repeating periodically the synchronization process.

According to another aspect of the invention there is provided a computer program product for a programmable apparatus, the computer program product comprising a sequence of instructions for implementing a method according to the invention, when loaded into and executed by the programmable apparatus.

According to another aspect of the invention there is provided a computer-readable storage medium storing instructions of a computer program for implementing a method according to the invention.

At least parts of the methods according to the invention may be computer implemented. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit", "module" or "system". Furthermore, the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer usable program code embodied in the medium.

Since the present invention can be implemented in software, the present invention can be embodied as computer readable code for provision to a programmable apparatus on any suitable carrier medium. A tangible carrier medium may comprise a storage medium such as a floppy disk, a CD-ROM, a hard disk drive, a magnetic tape device or a solid state memory device and the like. A transient carrier medium may include a signal such as an electrical signal, an electronic signal, an optical signal, an acoustic signal, a magnetic signal or an electromagnetic signal, e.g. a microwave or RF signal.

Embodiments of the invention will now be described, by way of example only, and with reference to the following drawings in which: Figure Ia and lb illustrates two different video streaming systems in which the invention can be implemented; Figure 2 is a block diagram illustrating a schematic configuration of a communication apparatus representing a transmitting device or a receiving device in an embodiment of the invention; Figure 3 illustrates the impact of the difference in the display timing of two screens; Figure 4 illustrates the main step of the synchronization process in an embodiment of the invention; Figure 5 illustrates the iteration process in an embodiment of the invention; Figure 6 illustrates the main steps of the iteration process in an embodiment of the invention.

The gist of the invention is to take advantage of the fact that one of the display device to be synchronized also owned some picture acquisition capabilities. This device is typically a smartphone or a tablet computer. It is therefore possible to ask to the user to take a picture of the second device using the handheld device. As the handheld device is responsible for the second device picture acquisition and for the display of the video stream, it is possible to know exactly what picture is being displayed on the handheld at the time of the picture acquisition of the distant picture displayed on the second display device. By comparing the distant picture to pictures in the sequence of pictures constituting the video stream, it is possible to determine in term of number of picture the drift between the two displays.

The time of display of a given picture in a video stream is typically managed by timestamps incorporated in the video stream giving the time of display of each picture. It may, for example, be the PTS (Presentation Time

B

Stamp) information in a MPEG 2 video stream. These time stamps are interpreted according to the clock of the display. They guarantee the synchronization of the different contents, typically audio and video, on a given display. But, as they are interpreted locally on the display, they do not guarantee the synchronization between different displays. However, by amending these time stamps in a video stream, it is possible to shift in time the display of the video stream. The display may be advanced or retarded by adding a negative or positive offset to all the time stamps in the video stream.

Of course, the offset that can be added is limited due to physical constraints. A picture to be displayed should be received and decoded, namely to be ready for display, before being actually displayed limiting the advance that can be obtained. On another hand, the amount of delay that can be added to the display is typically limited by the size of the buffer used to store pictures ready for display before actually displaying them. It comes that the synchronization may be actually done by retarding the display of the video stream on the display in advance, or by advancing the display of the video stream on the late display or by a combination of both. A command to introduce a given shift between the displays of the two video streams is typically sent to the source device. This source device will interpret this command, typically by adding a positive or negative offset to the display timer of one or both of the video streams broadcast to the two displays.

Pictures are displayed at a given rate on both displays. This means that each picture in the stream is displayed during a given period of time according to this rate. Therefore, it is possible to choose the time of acquisition of the distant picture relatively to the period of time when the local picture is displayed on the handheld display. For example, it is possible to choose the middle of the period of display of the local picture as the time of acquisition of the distant picture.

Once the shift between both displays has been determined in term of number of pictures, it is possible to refines the precision of this shift by instructing different shifts to the source device. Then for each shifts introduced between the two displays, it is possible to identify the shift corresponding to the beginning of the display of the distant picture corresponding to the local picture.

Advantageously, this approach is done by applying dichotomous shifts until we get the desired precision. This precision may be arbitrary; it is only limited by the precision of the clock used to encode the display time stamps. This clock is typically a 90 kHz clock. The precision is improved by iterating on providing to the source device a command to modify a display timer in the data stream provided to at least one of the display device based on dichotomous time drifts and by determining the shift between the two display in term of number of pictures for this time drift.

A detailed embodiment will now be described.

Figure Ia and lb describes two different video streaming systems in which the invention can be implemented. Those distribution systems are used to display simultaneously the content of a video source on two screens. These communication systems are only examples of systems where the present invention can be implemented.

Figure Ia describes a system wherein the source is merged with one display in the same device 120 and a second screen 110 is also used to display video sent by the source. The dotted arrow 140 represents the video streaming from the source/display 120 to the display 110.

Figure lb describes a system consisting in a source node 130 that simultaneously sends video stream towards handheld display device 120 (smartphone or tablet) and towards a large screen device 110. Dotted arrows and 150 represent the sending of both streams. A command channel between the handheld device 120 and the source device 130 is represented by the dashed arrow 160. Indeed the handheld device 120 is typically used as a remote controller of the source device 130.

In both systems, handheld device 120 is provided with picture acquisition capabilities, typically an embedded camera, allowing the device to take picture of the distant display 110.

Figure 2 is a block diagram illustrating a schematic configuration of a communication apparatus 200 representing a transmitting device or a receiving device (such as the camera device 120 or the display 110) adapted to incorporate the invention. Reference 202 is a RAM which functions as a main memory, a work area, etc., of CPU 201, and the memory capacity thereof can be expanded by an optional RAM connected to an expansion port (not illustrated). CPU 201 is capable of executing instructions on powering up the communicating apparatus from program ROM 203. After the powering up, CPU 201 is capable of executing instructions from the main memory 202 relating to a software application after those instructions have been loaded from the program ROM 203 or the hard-disc (HD) 206 for example. Such software application, when executed by the CPU 201, causes the steps of the flowcharts shown in one or more of the figures 4 and 6 to be performed.

Reference 204 represents the network interfaces that can be a single network interface, or composed of a set of different network interfaces (for instance several wireless interfaces, or different kinds of wired or wireless interfaces). Data packets are written to the network interface for transmission or read from the network interface for reception under the control of the software application running in the CPU 201. Reference 205 represents a user interface to display information to, and/or receive inputs from, a user.

I/O module 207 represents a module able to receive or send data from/to external devices (like video sensor or display).

Figure 3 describes the impact of the difference in the time of display of two screens. The time of display takes into account the encoding time, the transmission time, the decoding time, and the input lag of the display. The input lag is the time between the picture arrival, for instance at the HDMI port of the display] and the effective display of this picture. In a current typical TV, a lot of processing is performed to improve the quality and fluidity of the video. For instance, to convert input video frame rate, typically 30 or 60 Hz, to the display frame rate of at least 100 or 200 Hz. The buffering of pictures is necessary to perform interpolation between frames. Thereby, new frames are created in order to achieve the display frame rate. The difference in the time of display is mainly due to the difference in the input lag for each screen, indeed, each screen owns its proprietary technology with different processing.

Figure 3 is composed of three parts. The first one on the top of the picture depicts the display 320 of a video sequence on the screen of a device with an acquisition capability. This first device 120 is also used as a remote controller to pilot the video streaming on a second device 110 either directly because the device 120 streams the video to the second screen of the device or by interaction with an external video source, e.g. a video camera 130.

The second part, in the middle of the figure, describes the display 310 of the same video sequence on the screen. The third part on the bottom of the figure is a timeline that points out the major temporal events.

At tO referenced 330, the source 130 sends the video streams towards both screens. At ti referenced 331, the second screen 110 displays the first picture of the video sequence. The difference ti-to represents the global elapsed time between the sending of the video by the source and the display of the picture at the second screen (transmission time + decoding time + input lag of the display). At t2 332, the first screen starts to display the video sequence.

The difference in the display timing is represented by the duration t2-tl. One of the goal of the invention is to measure this difference and to compensate it. The duration of one picture 333 in the video sequence, or more clearly the frequency of pattern change, is made advantageously proportional to the acquisition frequency of the device 120 in order that the device 120 should be able to detect all pictures. With a real video stream, the only important point is that the shutter speed allows the discrimination of successive frames i.e. the shutter speed should be quicker than the frame rate to avoid superposition of successive frames in the resultant picture.

At t3 referenced 334, the device 120, which has a picture acquisition capability, takes a picture 390 of the second screen of the device 110 and looks in its current buffer the picture currently displayed 370, or the device 120 performs a print screen simultaneously to the distant picture snapshot. To perform the snapshot of the second screen of the device 110, the user points the distant screen and the picture is automatically taken. By comparison of both pictures, the local picture 370 and the distant one 390, the system computes a timing difference. For that purpose, the device performing the computation has to know the video sequence order to establish the timing difference associated to the pattern difference. Unfortunately, the accuracy of the measurement is linked to the picture duration. With only one measurement, the system is only able to know the difference in term of number of pictures.

In another embodiment of the measurement process, the device 120, which has picture acquisition capabilities, takes a picture 390 of the screen of the other device 110 and searches in its current buffer the picture having the same characteristics. These characteristics correspond typically to some point of interest. For example, for the snapshot of the distant screen at t3, the picture 390 is similar to the picture 380 of the local screen. In this case, the difference or time drift is determined by comparison of the timestamp t3, the time at which the picture is taken, and the time t4, referenced 335, of the display of the picture 380. As in the former embodiment, this one shot measurement suffers to an accuracy issue.

In relation with the next figures, we will describe an embodiment of the invention explaining how to improve the measurement accuracy. This may be done by an iteration process of the one shot measurement briefly describes here above.

Figure 4 describes the main steps of an algorithm according to an embodiment of the invention. The execution of this algorithm is shared between the source node and the node performing the measurement. Both functionalities (source and measurement) may take place in the same device.

The algorithm starts at step 400 which may correspond to a procedure of connection between the different nodes (source/sink coupling, session establishment, security setup...). Once the connection is established between the nodes of the system, the video streaming can start.

At the step 401, the video source (120 or 130 for respectively Figure Ia or Ib) generates the video stream. The video stream generated by the video source may have different shapes. The video stream can be a real content or a video pattern allowing an easiest analysis in the further steps. The picture with vertical line used for the examples of Figures 3 and 5 is an example of possible video pattern. The usage of video pattern may be interesting for a quick startup.

Once the system is well calibrated, the synchronization process could be performed on the real content of the video stream to check whether the system drifts or not. The video source may also insert pattern in the real video stream as a subliminal picture.

In some cases, the difference of display time between both screens may be very long e.g. if one screen applies huge post processing to improve the quality or fluidity of the video sequence while the other screen displays the video sequence on the fly without post processing. In that case, the system should prevent a pattern loop effect i.e. the system finds the good pattern correspondence but not in the same cycle. We consider here that the video stream is constituted with a periodically repeated sequence of patterns.

Consequently that leads to a synchronization with an offset which will not be detected by the system. For instance, the same pattern sequence is repeated every 30 ms and the difference in between screen is 45ms. The system will compute a difference of l5ms (45ms modulo 3Oms) because it finds the same pattern in a different 3Oms cycle. To avoid the pattern loop effect, the system can produce a video stream with different pattern duration 333 according the iteration along the synchronization process. The pattern duration is the number of picture with the same pattern content introduced successively in the sequence. For instance, at each iteration of the synchronization process, the pattern duration decreases. Thus, at the beginning, long pattern duration avoids the pattern loop effect. Then, the system improves the accuracy by decreasing the pattern duration up to a minimum value which corresponds to one picture of each pattern. Another method to prevent pattern loop effect is to have a large set of different pattern to be able to cope with a huge delay between screens.

Then the system goes to the step 402 wherein the video source distributes video to both display devices (110 and 120). The system produces two video streams with a common presentation time i.e. the Presentation Time Stamp contained in the MPEG2TS header. This timestamp is in unit of 90kHz (ITIJ-T H.222.0). Of course, the use of the presentation time stamp of MPEG 2 is only the example used in the present embodiment to manage the time of display of images. In some other embodiment, others way to control this time of display may be used according to the actual standard used to encode and broadcast the video stream. The video content may also be different in term of format to match the actual format of the display device. If the source is independent of one of the display device, as in the system illustrated on Figure Ib, the video source device 130 should send each video stream (140 and 150) to its respective display device (110 and 120). In the case where both streams are strictly the same, the video source can broadcast the video stream to all display devices. When they receive the video stream, the display devices carry out decoding process, if any, and display the video.

Upon reception of a trigger sent by the source, or by auto-triggering, the display device with picture acquisition capabilities 120 invites the user to point the distant display 110. The display device with a picture acquisition capability takes a picture of the distant display device 110 in a step 403. The timet3 at which the picture is taken is time stamped. When the display device 120 recovers the distant picture of the distant display 110, it analyzes the picture, in step 404, in order to extract points of interest (P01) of the picture 390. Next, the system goes to the step 405 in which the display device 120 searches a picture with the same characteristics, namely the same P01, in its display buffer. This picture can be already present in the display buffer -the display device with a picture acquisition capability 120 is in advance compared to the other screen -or may require a delay to obtain the matched picture. Moreover, the display device 120 can compute the P01 on the fly and so store the P01 and associated time of display rather than store the whole video sequence.

Concerning the P01, a lot of technicstalgorithms are available in the state of the art to compute P01 and so find similarities in pictures. Those technics may be based on the corner or edge detection, intensity modification and so on. Some algorithms are mentioned as an example: Harris detector, Features from Accelerated Segment Test (FAST), Maximally Stable Extremal Regions (MSER)... Any of these technics may be used in the scope of the invention. The system can also detect the scene change and use it as synchronization point. In this last case, the system of acquisition is set in video mode rather than in picture mode in order to detect the scene change of the distant screen. With the video stream containing patterns, the P01 computation is much easier in comparison to real" video stream indeed the video patterns are built in order to simplify the P01 search.

When the display device 120 has found a similar local picture corresponding to the distant one, it computes the delay between the display timing for both pictures in step 406. The computation is done either by comparison of the timestamp t3 with the timestamp t4 corresponding to the display of the local picture 380 in the device 120, and/or based on the gap in picture number between the picture 370 and the picture 380 knowing the frame per second. In the example of Figure 3, there is a gap of one picture i.e. a delay from 16 to 32 milliseconds for a frame rate corresponding to a display time of 16 milliseconds for each picture. Both alternatives give a different view of the same result.

Once the delay or time shift is computed, the display device 120 sends a command message to the video source in the step 407 to compensate the time shift. The command message is composed of the delay value itself. This delay value may be the determined time shift itself or a value based on this time shift in particular when iterating to improve the accuracy of the determined time shift.

This iterative process will be detailed below in relation with Figure 5 and 6.

Advantageously it also comprises the range of confidence. This command message may also be composed of the gap in term of picture. In case of the system illustrated in Figure Ia wherein the source and the display device are the same, this command is an internal command from the software module implementing the measurement to the software module managing the sending of the video stream.

When the source receives the delay, the video source adjusts in a step 408 the video distribution timing to compensate the display timing gap between devices 110 and 120. To do this, the video source, in this embodiment, has to modify the presentation timestamp (or decoding timestamp) for instance contained in the MPEG2TS header. This timestamp is in unit of 90kHz (ITU-T H.222.0) which ensures a sufficient accuracy for the current invention. For instance, the presentation timestamp of the device 120 is used as reference thereby the source device modifies the presentation timestamp of the bitstream intended for the other display device 110. In some cases, it may be necessary to modify the display time of both display devices (110 and 120) e.g. the delay to compensate is really large and the correction should be distributed on both displays or even it is more efficient to modify the display time to postpone a stream rather anticipate the other stream. Indeed, the system has to keep a minimum latency (encoding, transmission, decoding) in order to maintain stability. Moreover, the buffer capabilities of the display device should be taken into account for the timing modification. For instance in case of postpone, the display device has to store more data in its buffer before displaying. The gap between both timestamps corresponds to the delay received from the display device with an acquisition capability 120. Then, the system tests in a step 409 whether the synchronization process is over.

It is worth noting that the synchronization process may be conducted in a one iteration process. But doing this the accuracy of the determined time shift corresponds to an integer number of frame. An iterating process, as detailed below in relation with Figure 5 and 6 allows to improve this accuracy.

In the context of the iterating process, several thresholds may be reached to stop the synchronization process. For instance, the system reaches a predefined accuracy value e.g. 1 millisecond of exactitude or a ratio of the duration of a frame. Another threshold may be the number of iteration of the adjustment process. This test can be very useful if the system reaches an oscillation point. The system can also test the convergence of different measurement. So if the synchronization process is completed, the algorithm ends in step 410. Otherwise, the system goes back to the step 401.

In this algorithm, the step 408 is performed by the source but the delay may also be managed by the handheld device or TV through a dedicated application. In that case, the delay is added at decoding time by the rendering application within the display device. The delay command is then sent to the display device instead of the source device.

Figure 5 depicts an example of iteration process. This figure shows the way to achieve synchronization between the display devices with a dichotomy method. The aim is, knowing the time of acquisition of the distant picture relatively to the duration of display of the local picture, to try different time shift in order to align the local picture with the distant picture with this time of acquisition. Typically, in the described embodiment, the time of acquisition is chosen to correspond to the middle of the display of a local picture. By trying dichotomous time shift, and for each time shift determining which distant image is displayed at the time of acquisition, it is possible to identify the time shift corresponding to the beginning of the display of the distant image corresponding to the local one. When the alignment is found it is possible to exactly match the beginning of the display of both the local image and the distant one. For the sake of clarity, an example of this process will now be described.

Numbers 501 to 508 represent the iteration number in the process of synchronization. The number 511 to 518 represent the pictures recorded for the first screen of the device 120 which owns the acquisition capabilities. The number 521 to 528 represent the picture recorded for the second screen (screen of the distant wireless device 110). The last number family 531 to 538 represents the delay applied by the source between both video streams. This delay is the difference of the presentation timestamp between both video streams.

At the first iteration 501, the source applies a null delay 531 between both video streams. When comparing the distant image 521 and the local one 511, it may be determined a time drift of one picture using the method described in relation to Figure 4. Indeed, the picture 511 has two vertical lines while the picture 521 has three vertical lines. It means that the stream displayed on the screen 1 is late regarding the same stream displayed on the screen 2. As shown in Figure 3, the system, namely the screen which owns the acquisition capabilities, takes the middle of the picture of the first screen as time reference.

This picture difference delay brings back in time corresponds to a range from 8, the reference frame varying from -8 to 8 millisecond, to 24 milliseconds, based on the fact that the frame rate is 16 milliseconds and the 0 corresponds to the middle of the frame.

For the following of the iterative process, the system tries to find the boundary between the picture with one and two vertical lines for the second screen. Indeed, the picture with two vertical lines is considered as the reference for the first screen. Next, for the second iteration 502, the video source applies a delay 532 of -32 milliseconds to the second stream, the stream which is provided to the screen 2. The time to correct a time drift of two pictures in order to reach the limit between pictures with one and two vertical lines as previously explained. By the modification of the timing relationship, the screen 2 (picture 522) becomes late compared to the screen 1 (picture 512). This time drift of one picture brings back in time is equivalent to a range from -8 to -24 milliseconds.

Next, the system applies in iteration three 503, a correction 533 of -16 milliseconds. As a result, the system computes a new picture delay difference of 0 picture between picture 513 and picture 523. It may be deduced that the boundary we are looking for is now delimited in the range from -32 to -16 milliseconds. It is worth noting that, strictly speaking, these two steps, 502 and 503, are not mandatory. Indeed, applying a delay being a multiple of the image rate do not bring additional information. From the first step, we already know that the boundary we are looking for is delimited in the range from -32 to -16 milliseconds. Steps 502 and 503 are described for illustrative purpose, they confirm information obtained from the first step 501.

Thereby, by dichotomy the system applies a new delay 534 of -24 milliseconds in the fourth iteration 504. With this new delay 534, the system computes a picture difference of one picture between picture 514 and picture 524. Thus the boundary is now located in the range from -24 to -16 milliseconds.

According to this new range, the system defines a new delay 535 of -20 milliseconds in the fifth iteration 505. The system computes a time shift of 0 pictures between picture 515 and picture 525. With this new measurement, the system determines a new range for the pictures boundary from -24 to -20 milliseconds.

In the sixth iteration 506, the system applies a delay 536 of -22 milliseconds. Then, the system detects a time drift of one picture between picture 516 and picture 526 that corresponds to a range from -22 to -20 milliseconds for the picture boundary that the system tries to find.

For the iteration 507, the system modifies the timing relationship in between both stream by a delay 537 of -21 milliseconds which gives the limit in between the picture with one and two vertical lines for the second screen with an accuracy of 1 millisecond. Finally, in the iteration 508, the system offsets this boundary value with half the frame rate (8 milliseconds in our current example) to temporally align the middle of the picture in the second screen with the middle of the picture in the first screen. Thereby, the system achieves a good synchronization with 1 millisecond accuracy between both screens by delaying the stream of -13 milliseconds. In our example, the algorithm has been stopped when reaching the accuracy of one millisecond. This is arbitrary, the accuracy that can be reached is only limited by the precision of the display timer, 90 kHz

in our example.

Figure 6 is an example of algorithm of the iteration process which improves the dual screen synchronization accuracy. This example sums up the examples of the former figure. In this figure, the first screen is the screen having the acquisition capability (120) and the second screen corresponds to the TV set (110).

The algorithm starts by the step 600 in which the system may for instance initialize internal variables. Then, the system retrieves the delay between screens in step 601. This delay message contains at least the gap in term of number of picture. This delay message is stored 602 in order for the source to have the historic of the delay and so the delay evolution.

In the step 603, the system checks according to the delay value received in step 601 whether the 2nd screen is late compared to the P' screen. The first goal is to find the limit between the reference picture (i.e. the picture of the first screen which is displayed at the instant of the snapshot of the second screen) and the picture just before, "late" means that the second stream is late according to this limit. If the test is negative, the system increases the presentation time stamp of the second stream 604 in order to correct the gap in between the both stream displaying. If the test is positive, the system decreases the presentation time stamp of the second stream 605 in order to correct the gap in between the both stream displaying. Those modifications (increasing or decreasing) of the presentation time stamp are performed according the current and previous values of the recovered delays from step 601.

As described in the previous figure, the current delay value and the previous ones are used to find, by a dichotomy method, the beginning of the reference picture i.e. the picture of the first screen which is displayed at the instant of the snapshot of the second screen. After the system has performed the computation of the correction to apply to the timing relationship in between the streams in the step 604 or 605, the system tests in step 606 whether this correction reaches the expected accuracy. For instance, in Figure 5, the system targets accuracy around millisecond; thereby, in the iteration 7, the system locates the beginning of the reference picture with a confidence range of +1-1 millisecond. If the result of the test 606 is negative, the system goes back to the step 601 otherwise, the system jumps to the step 607. In the step 607, the system adds an offset to the computed correction. This offset corresponds to the half of the frame duration (8 milliseconds for a 60 frame per second, 16 milliseconds for a 30 frame per second). This offset allows the temporal alignment of the middle of the picture in the second screen with the middle of the picture in the first screen. For instance, align picture 336 and picture 335 in Figure 3. Afterwards, the algorithm ends at the step 608. It is worth noting that the offset is directly correlated to the reference time chosen for the acquisition.

An offset of half the frame duration corresponds to the choice of the middle of the display time of one picture as the time of acquisition. Another choice would lead to another offset.

Any step of the algorithm shown in Figure 4 or 6 may be implemented in software by execution of a set of instructions or program by a programmable computing machine, such as a PC ("Personal Computer), a DSP ("Digital Signal Processor") or a microcontroller; or else implemented in hardware by a machine or a dedicated component, such as an FPGA ("Field-Programmable Gate Array") or an ASIC ("Application-Specific Integrated Circuit').

Although the present invention has been described hereinabove with reference to specific embodiments, the present invention is not limited to the specific embodiments, and modifications will be apparent to a skilled person in the art which lie within the scope of the present invention.

Many further modifications and variations will suggest themselves to those versed in the art upon making reference to the foregoing illustrative embodiments, which are given by way of example only and which are not intended to limit the scope of the invention, that being determined solely by the appended claims. In particular the different features from different embodiments may be interchanged, where appropriate.

In the claims, the word comprising does not exclude other elements or steps, and the indefinite article "a" or "an' does not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be advantageously used.

Claims

CLAIMS1. A method of synchronizing the display of video data on at least two display devices, the same video data stream being provided by a source device to the display devices through a communication network, one of the display devices being provided with picture acquisition capabilities, the method comprising by a first display device comprising picture acquisition capabilities: -acquiring using the picture acquisition capabilities a distant picture corresponding to a picture displayed by a second display device; -identifying a local picture displayed by the first display device at the time of acquisition of the distant picture; -determining a time drift between the first and the second display devices using the local and the distant pictures; and -providing a command to modify a display timer in the data stream provided to at least one of the display devices based on the determined time drift.
2. The method according to claim 1, wherein the command to modify the display timer is provided to the source device.
3. The method according to claim 1, wherein the command to modify the display timer is provided to a display device.
4. The method according to any one claim 1 to 3, wherein determining a time drift between the first and the second display devices using the local and the distant pictures comprises: -determining points of interest in the distant picture; -determining points of interest in the local picture; and -determining the distance in term of pictures between the local picture and the distant picture in the sequence of pictures composing the video data stream.
5. The method according to claim 4, wherein the sequence of pictures composing the video data stream is composed of successive predefined patterns for an easy determination of the points of interest.
6. The method according to any one claim I to 5, wherein the time of acquisition of the distant picture is chosen at a predefined time of the display of local pictures.
7. The method according to claim 6, wherein the precision of the determined time drift is improved by iterating on: -providing to the source device a command to modify a display timer in the data stream provided to at least one of the display devices based on dichotomous time drifts; -acquiring using the picture acquisition capabilities a distant picture corresponding to a picture displayed by a second display device; -identifying a local picture displayed by the first display device at the time of acquisition of the distant picture; and -determining a time drift between the first and the second display devices using the local and the distant pictures.
8. The method according to claim 6 or 7, wherein the time of acquisition of the distant picture is chosen to be the middle of time of display of local pictures.
9. The method according to any one claim 1 to 8, wherein the first display device comprises the source device.
1O.The method according to any one claim 1 to 9, wherein the synchronization process is repeated periodically.
11.A first display device comprising: -picture acquisition capabilities; -a synchronization module for synchronizing the display of video data on said first display device and on a second display device, the same video data stream being provided by a source device to the display devices through a communication network; -an acquisition module for acquiring using the picture acquisition capabilities a distant picture corresponding to a picture displayed by a second display device; -an identifying module for identifying a local picture displayed by the first display device at the time of acquisition of the distant picture; -a time drift module for determining a time drift between the first and the second display devices using the local and the distant pictures; and -a command module for providing a command to modify a display timer in the data stream provided to at least one of the display devices based on the determined time drift.
12.The device according to claim 11, wherein the command module provides the command to modify the display timer to the source device.
13.The device according to claim 11, wherein the command module provides the command to modify the display timer to a display device.
14. The device according to any one claim 11 to 13, wherein the time drift module comprises: -a module for determining points of interest in the distant picture; -a module for determining points of interest in the local picture; and -a module for determining the distance in term of pictures between the local picture and the distant picture in the sequence of pictures composing the video data stream.
15.The device according to claim 14, wherein the sequence of pictures composing the video data stream is composed of successive predefined patterns for an easy determination of the points of interest.
16. The device according to any one claim 11 to 15, wherein the time of acquisition of the distant picture is chosen at a predefined time of the display of local pictures.
17.The device according to claim 16, wherein the device comprises for improving the precision of the determined time drift an iteration module for iterating on: -providing to the source device a command to modify a display timer in the data stream provided to at least one of the display devices based on dichotomous time drifts; -acquiring using the picture acquisition capabilities a distant picture corresponding to a picture displayed by a second display device; -identifying a local picture displayed by the first display device at the time of acquisition of the distant picture; and -determining a time drift between the first and the second display devices using the local and the distant pictures.
18. The device according to claim 16 or 17, wherein the time of acquisition of the distant picture is chosen to be the middle of time of display of local pictures.
19.The device according to any one claim 11 to 18, wherein the first display device comprises the source device.
20.The device according to any one claim 11 to 19, wherein the device comprises a module for repeating periodically the synchronization process.
21.A computer program product for a programmable apparatus, the computer program product comprising a sequence of instructions for implementing a method according to any one of claims 1 to 10, when loaded into and executed by the programmable apparatus.
22.A computer-readable storage medium storing instructions of a computer program for implementing a method according to any one of claims ito 10.
23.A method of synchronization substantially as hereinbefore described with reference to, and as shown in Figure 4 to 6.