JP2016096440A - Video image coded data transmission device, management device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video image coded data transmission device, a management device, and a program, capable of suppressing transmission data amount in a whole system, and reducing a delay.SOLUTION: A video image coded data transmission device connected to the other video image coded data transmission device through a network, comprises: coding means that codes video image data; timing notification means that notifies information for controlling an intra-flame transmission timing to the other video image coded data transmission device; timing information reception means that receives information on the intra-flame transmission timing in the other video image coded data transmission device from the other video image coded data transmission device; and transmission means that transmits video image coded data to an external data reception device. Timing control means transmits the intra-flame in a timing different from the other video image coded data transmission device.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a video encoded data transmission device, a management device, and a program.

  Currently, MR (Mixed Reality) systems that seamlessly integrate the real world and virtual space in real time have been used. One method for realizing the MR system is a video see-through method. As an example of the video see-through method, there is a technique in which a field of view of the HMD user is captured by a video camera attached to the HMD, and an image in which CG is superimposed on the captured image is observed by a display attached to the HMD. Here, HMD means Head Mounted Display, and CG means Computer Graphics.

  Since the processing for superimposing CG on the captured image has a high calculation processing cost, it is connected to an external device such as a PC, temporarily transmits the captured image to the external device, superimposes the CG on the external device, and sends it back to the HMD. Or display from. In such a case, in order for the HMD user to experience the MR space while freely moving, it is desirable to communicate with an external device that superimposes the HMD and CG by wireless transmission.

  However, in general, wireless communication has a narrower communication band than wired communication, and delay due to packet loss on wireless transmission is likely to occur. In addition, when a plurality of HMD users use the MR system at the same time, the wireless communication band is further tightened. As a method for stably experiencing the MR space in real time, there is a method of reducing the amount of communication data by compressing captured image data using a video encoding technique.

  As a video encoding method, H.264 is used. H.264 (ISO / IEC 14496-10 MPEG-4 Part10 Advanced Video Coding) and H.264. There are video coding standards such as H.265 (ISO / IEC 23008-2 HEVC). In these video encoding standards, a technique called inter-frame prediction that encodes a difference image between a predicted image of a frame at a different time and an input image is used in order to increase compression efficiency. In general, interframes encoded by interframe prediction have a smaller data size than intraframes encoded without using interframe prediction. On the other hand, when an error occurs in the frame data referred to by the inter frame, the error is propagated. Therefore, a method of regularly inserting an intra frame that does not use inter-frame prediction is generally used.

  In Patent Document 1, as a similar environment to an environment in which a plurality of HMD users simultaneously use an MR system, a video bitstream code amount is controlled from a communication band and an allowable delay time in a system that receives video data from a plurality of terminals. A method is disclosed.

JP 2007-336260 A

  However, in the MR system, since the real time MR space is required for the HMD user to experience a realistic MR space, the allowable delay time is very short. Therefore, the method of Patent Document 1 has a problem in that when the transmission timings of intra frames of a plurality of HMD users overlap, the amount of instantaneous transmission data in the entire system increases, causing a delay. It was.

  An object of the present invention is to suppress the amount of transmission data in the entire system and reduce delay.

In order to achieve the above object, a video encoded data transmitting apparatus according to the present invention comprises the following arrangement.
That is, a video encoded data transmitting apparatus connected to another video encoded data transmitting apparatus via a network, and encoding means for encoding video data;
Timing control means for controlling intra-frame transmission timing, which is transmission timing of intra-frames generated in encoding by the encoding means;
Timing notification means for notifying the other video encoded data transmission device of information related to transmission timing controlled by the timing control means;
Timing information receiving means for receiving information on intra frame transmission timing in the other video encoded data transmitting apparatus from the other video encoded data transmitting apparatus;
Transmission means for transmitting the video encoded data encoded by the encoding means to an external data receiving device,
The timing control unit performs control so as to transmit the intra frame at a timing different from that of the other video encoded data transmission device based on the information on the intra frame transmission timing received by the timing information reception unit. And

The management device of the present invention has the following configuration.
A management device connected to one or more video encoded data transmission devices via a network,
A transmission start request receiving means for receiving a transmission start request, which is a request to start transmission of video encoded data from the video encoded data transmitting device;
Transmission end request receiving means for receiving a transmission end request, which is a request to end transmission of video encoded data from the video encoded data transmitting device;
Timing management means for managing intra-frame transmission timing, which is intra-frame transmission timing generated at the time of encoding by the video encoded data transmitting device;
Timing calculating means for calculating the intra frame transmission timing of the video encoded data transmitting device;
Timing transmission means for transmitting the intra frame transmission timing to the video encoded data transmission device,
The timing calculation means includes
When the transmission start request receiving means receives the transmission start request,
The intra frame transmission timing of the encoded video data transmission apparatus that has transmitted the transmission start request is not overlapped with the intra frame transmission timing of the encoded video data transmission apparatus already managed by the timing management unit. Further, the intra frame transmission timing of each video encoded data transmission device is newly calculated,
When the transmission end request receiving means receives the transmission end request,
The intra-frame of the remaining encoded video data transmission apparatus, excluding the encoded video data transmission apparatus that has transmitted the transmission end request, which is already managed by the timing management unit. A new calculation of the intra-frame transmission timing of each video encoded data transmission device so that the transmission timing interval widens;
It is characterized by that.

  According to the present invention having the above configuration, the amount of data in the entire system can be suppressed and the delay can be reduced.

1 is a configuration diagram of an MR system according to Embodiment 1 of the present invention. It is a functional block diagram of HMD in connection with Example 1 of this invention. It is a sequence diagram which shows the flow of the synchronous message transmitted / received between HMD. It is a flowchart which shows the intra-frame transmission timing calculation method. It is a ring buffer which shows an intra-frame transmission timing arrangement | sequence. It is a functional block diagram of PC for CG superimposition concerning Example 2 of the present invention. It is a MR system block diagram concerning Example 3 of this invention. It is a functional block diagram of PC for MR system management concerning Example 3 of the present invention. It is a flowchart which shows the intra-frame transmission timing calculation method. It is a functional block diagram of PC for MR system management concerning Example 4 of the present invention. It is a flowchart which shows the intra frame transmission timing recalculation method. It is a ring buffer which shows an intra-frame transmission timing arrangement | sequence.

  Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

[Example 1]
FIG. 1 is a diagram showing a configuration example of an MR system according to an embodiment of the present invention.
In FIG. 1, three users A, B, and C wear HMD 101, HMD 102, and HMD 103, respectively, and use the MR system. The number of users need not be three. Each HMD is connected to an AP (Access Point) 104 via a wireless network, and transmits / receives packets such as video data and synchronization messages between the HMDs. Examples of the wireless transmission method between each HMD and the AP 104 include, but are not limited to, IEEE802.11n and IEEE802.11ac.

  The AP 104 is connected by wire to the router 105, and constructs a LAN (Local Area Network). The AP 104 and the router 105 may be integrated. Examples of the wired transmission method between the AP 104 and the router 105 include, but are not limited to, Gigabit Ethernet (registered trademark) and IEEE 1394. The PC 106, PC 107, and PC 108 corresponding to each HMD on a one-to-one basis are wired to the router 105 in the same LAN, and transmit and receive video data packets.

  Video data 109, video data 110, and video data 111 are real space video data captured by a camera attached to each HMD. Each PC superimposes CG on each received real space video data. The CG superimposed video data 112, the CG superimposed video data 113, and the CG superimposed video data 114 are transmitted from each PC to each HMD via the router 105 and the AP 104. Each CG superimposed video data is displayed on a display attached to each HMD. Each PC may not correspond to each HMD on a one-to-one basis, or may be shared by one PC.

  FIG. 2 is a functional block diagram showing functions of the HMD 101, the HMD 102, and the HMD 103 necessary for realizing the embodiment of the present invention in the MR system shown in FIG. The configuration shown here can be realized by a combination of hardware and software, and one functional block does not correspond to one hardware. The same applies to the following functional block diagrams.

The synchronization message transmission unit 201 transmits a message for achieving synchronization between the HMDs. There are two types of messages to send. The synchronous message transmission unit 201 corresponds to a timing notification unit.
One is a message for notifying the other user's HMD of the start of use when the HMD of each user starts using the MR system.
The other is a message for notifying synchronization information when a message for notifying the start of use is received from the HMD of another user.

A message for notifying the first other user of the start of use is transmitted to all other users using the same LAN by broadcast transmission.
A message for notifying the second other user of the synchronization information is transmitted to the user who has transmitted the use start notification message by unicast transmission.
The wireless interface 202 is a communication interface for wireless connection with the AP 104.
The synchronization message receiving unit 203 receives a synchronization message from another user. The synchronization message includes information on the intra frame transmission timing of another video encoded data transmission device. The synchronization message receiving unit 203 corresponds to a timing information receiving unit.

The synchronization processing unit 204 performs synchronization processing from the received synchronization message. A specific synchronization processing method will be described later.
The intra frame transmission timing calculation unit 205 calculates timing so that the intra frame transmission timing does not overlap with other users. A specific example of the calculation will be described later.
The imaging unit 206 is a camera attached to the HMD and images a real space image. The captured image encoding unit 207 encodes the captured real space video by an arbitrary encoding method according to the intra frame transmission timing calculated by the intra frame transmission timing calculation unit 205. Here, the arbitrary encoding method is, for example, H.264. H.264 (ISO / IEC 14496-10 MPEG-4 Part10 Advanced Video Coding) and H.264. 265H. H.265 (ISO / IEC 23008-2 HEVC) and the like are listed, but other methods may be used.

The captured image transmission unit 208 packetizes the encoded real space video data so as to conform to the communication protocol, and transmits the packet to the PC 106 that superimposes the CG via the wireless interface 202. “Encoded real space video data” corresponds to “video encoded data”, and “PC 106 for superimposing CG via wireless interface 202” corresponds to “external data receiving device”.
The CG superimposed image receiving unit 209 receives the CG superimposed video data on which the CG is superimposed on the PC 106 via the wireless interface 202. The CG superimposed image decoding unit 210 decodes the received CG superimposed video data using an arbitrary encoding method. The display unit 211 is a display attached to the HMD, and displays the decoded CG superimposed video.

FIG. 3 is a sequence diagram showing a flow of a synchronization message transmitted / received between the HMDs.
The HMD worn by the use start user holds a time T1 at which a message for notifying the start of use is broadcast to other users.
The HMD worn by the existing user holds the time t1 when the usage start message is received from the HMD worn by the usage start user.
In addition, the HMD worn by the existing user transmits a synchronization information message to the HMD worn by the user who starts use.
The information message for synchronization is
A time t2 at which the synchronization information message is transmitted (time t2 is a time measured by an HMD clock worn by an existing user);
the absolute time Δt of t2−t1,
Intra frame transmission timing information of HMD worn by existing users,
And a user ID that is unique within the MR system.

The HMD worn by the use start user acquires an absolute time ΔT (= T2−T1) from “time T1 at which the use start message transmission is transmitted” to “time T2 at which the synchronization information message is received”. Time T1 and time T2 are both times measured with an HMD clock worn by the user who started use.
The time (ΔT−Δt) obtained by subtracting Δt in the synchronization information message from ΔT is calculated as RTT (Round Trip Time), and the HMD of the HMD to be used by the user who starts use is set so that T2− (RTT / 2) = t2. Synchronize timers.
When there are a plurality of existing users, it may be synchronized with one representative user, for example, a user at an intermediate time within a plurality of synchronization times, or an average of a plurality of synchronization times may be taken. The method may be used.

The synchronization method described here is only an example of a method for realizing the embodiment of the present invention. The synchronization method applicable to the present invention is not limited to the synchronization method described here.
When synchronizing with the HMD worn by the existing user, the HMD worn by the user who has started using the user sets a unique user ID that does not overlap with the user ID of the existing user, and holds it internally. In addition, the HMD worn by the use start user calculates the intra frame transmission timing of the HMD worn by the use start user. The intra frame transmission timing of the HMD worn by the user who starts using is calculated so as not to overlap with the intra frame transmission timing of the HMD worn by the existing user.

FIG. 4 is a flowchart illustrating a method of calculating intra frame transmission timing.
In S401, after broadcasting the use start message to other users, it is determined whether or not a synchronization information message is returned from the existing user. As a determination method, if a message is not replied even after a predetermined timeout period has elapsed, it is determined that the message has not been received. When it is determined that no reception has been made, the process proceeds to S402.

In S402, 0 is set as the intra frame transmission timing, and the process ends.
The HMD internal timer of the first user is used as a common timer of the MR system, and the newly used user is set to synchronize with this common timer. When a synchronization information message is received from one or more existing users, the process proceeds to S403.

  In S403, the intra frame transmission timings of all users who have received the synchronization information message are sorted. Here, the timing is represented by a slot number of the ring buffer in which the number of slots is the period in which the intra frame is inserted (intra frame insertion period). The intraframe insertion period of each HMD is set in advance as a uniform period in the MR system. The “intra frame insertion period” corresponds to an “intra frame transmission period”.

FIG. 5 shows an example of a ring buffer in which one intra frame is inserted every 30 frames.
For example, it is assumed that there are an existing user A and an existing user B, and the intra frame transmission timings in the synchronization information message returned from each are 0 and 12.
Here, it is assumed that a new intra frame transmission timing is calculated for the new user C.
When the intra-frame transmission timings returned from the existing users are sorted in descending order and stored as elements of the arrays T [0] and T [1] having the same number as the existing users,
User B's intra frame transmission timing “12” is stored in T [0],
User A's intra frame transmission timing “0” is stored in T [1].

<Determining the maximum value of the interval of each intra frame transmission timing in the intra frame transmission period>
In S404, an array combination T [m], T [n] that obtains the maximum difference between the elements of the adjacent arrays of each sorted intra frame transmission timing is acquired. When the difference is a negative number, the difference is an intra frame insertion period, which is a value obtained by adding 30 in this embodiment.
The difference between the elements of adjacent arrays, T [0] and T [1], is T [0] −T [1] = 12−0 = 12.
The difference between the elements of adjacent arrays, T [1] and T [0] is T [1] −T [0] = 0−12 = −12. Since the difference has become a negative number, a value obtained by adding the intraframe insertion period 30, −12 + 30 = 18, is set as the difference.
Therefore, the maximum value of the difference is Max (18, 12) = 18.

<The middle of the maximum value of the intra frame transmission timing interval is set as a new intra frame transmission timing>
In S405, the middle of the maximum value of the intra-frame transmission timing interval is obtained. Specifically, a value obtained by adding one half of the maximum difference value to the smaller one of the two timings at which the difference becomes the maximum value Min (T [m], T [n]) + ΔT / 2
Is the intra frame transmission timing.

However, since the difference is a negative number, when the intra-frame insertion period 30 is added to the difference, the difference is set to T [m] when the maximum value Max (T [m] −T [n]) is calculated. Add the frame insertion period 30.
Therefore, in the example shown in FIG. 5, T [1] = 30.
Min (T [0], T [1]) + ΔT / 2
= Min (12,30) + ΔT / 2
Is the intra frame transmission timing.
Here, ΔT is the maximum difference 18 acquired in S404.
Also, Min (T [0], T [1]) = Min (12, 30) = 12.
Therefore, Min (T [m], T [n]) + ΔT / 2 = 12 + 18/2 = 12 + 9 = 21, and the intra-frame transmission timing of the newly used user C is 21.
When the intra frame transmission timing calculated here exceeds the cycle, (calculated value−cycle) is set as the intra frame transmission timing. ΔT / 2 is an integer rounded down after the decimal point.

Using the timing calculated by the intra frame transmission timing calculation unit 205, the encoding timing of the captured image is controlled. For example, when the frame rate is 30 fps and the calculation timing is 21, in the synchronized internal timer, 0.000 seconds is set to timing 0, and timing 21 is set to 0.033 × 21 = 0.593.
The generation of the intra frame may be waited until the intra frame transmission timing, and when the intra frame transmission timing is reached, the generation of the intra frame and the transmission of the intra frame may be performed substantially simultaneously. Alternatively, an intra frame is generated immediately, but transmission of the intra frame is waited for, and the generated intra frame is inserted when the calculated intra frame transmission timing elapses. Thereafter, the intra frame is generated and transmitted every period. Good.

  In the present embodiment, synchronization is achieved among a plurality of HMD users, and the instantaneous maximum transmission data amount in the entire system is suppressed by intentionally shifting the intra frame transmission timing. This makes it possible to effectively utilize the communication band and maximize the video quality in the MR system. In the present embodiment, the video see-through MR system has been described. However, the present invention is not limited to the MR system, and can be applied to an optical see-through MR system, for example. In this embodiment, the MR system is described as an example, but the present invention can also be applied to a surveillance camera system, a videophone system, or the like having a similar topology configuration. In this embodiment, the configuration within the same LAN has been described. However, the present invention can also be applied between different LANs or via a WAN.

[Example 2]
In the first embodiment, a description has been given of a mode in which a plurality of HMD users are synchronized and the intra frame transmission timing is shifted. In this embodiment, a mode in which the intra frame transmission timing is shifted not between HMD users but between a PC on which HMD and CG are superimposed will be described.
Since the configuration example of the MR system according to the embodiment of the present invention is the same as that of FIG. 1 described in the first embodiment, description thereof will be omitted, but in this embodiment, even one user can be applied.
The functional block diagram showing the functions of the HMD 101, HMD 102, and HMD 103 necessary for realizing the embodiment of the present invention in the MR system shown in FIG. 1 is the same as FIG. Omitted.

FIG. 6 is a functional block diagram showing functions of the user A PC 106 necessary for realizing the embodiment of the present invention in the MR system shown in FIG.
The synchronization message receiving unit 610 receives a use start message for notifying the start of use from the HMD 101 via the wired interface 601. When the use start message is received, the synchronization information message creating unit 611 determines the intra frame transmission timing at an arbitrary timing, and creates a message including the determined intra frame transmission timing and synchronization time information. Since the sequence and the synchronization time information are the same as those in FIG. 3 described in the first embodiment, description thereof is omitted.

  The synchronization message transmission unit 612 transmits the created synchronization information message to the HMD 101 via the wired interface 601. The captured image receiving unit 602 receives real space video data from the HMD 101 via the wired interface 601. The captured image decoding unit 603 decodes the received real space video. The feature point extraction unit 604 performs image analysis on the decoded real space video and extracts feature points. Since the extraction method is not an essential element of the present invention, the description thereof is omitted.

  The position / posture calculation unit 605 calculates the position / posture of the HMD 101 using the extracted feature points. The CG image superimposing unit 606 determines a location, a direction, and the like to superimpose the CG from the calculated position and orientation of the HMD 101, and superimposes the CG on the real space video using the CG data acquired from the content DB 607. The CG superimposed image encoding unit 608 encodes the CG superimposed video by an arbitrary encoding method. Intraframe transmission is performed at the timing determined by the synchronization information message creation unit 611.

  The CG superimposed image transmission unit 609 packetizes the encoded CG superimposed video data so as to conform to the communication protocol, and transmits the packetized data to the HMD 101 via the wired interface 601. The method for determining the intra-frame transmission timing in the HMD 101 is the same as that in FIG.

  In this embodiment, the instantaneous maximum transmission data including uplink and downlink in the wireless communication section between the HMD and the AP by synchronizing the HMD user and the CG superposition PC and intentionally shifting the intra frame transmission timing. Reduce the amount. This makes it possible to effectively utilize the communication band and maximize the video quality in the MR system. In this embodiment, the intra frame transmission timing is determined first on the PC side. However, the intra frame transmission timing may be determined first on the HMD side and the synchronization processing may be performed on the PC side. Further, the intra frame transmission timing control between a plurality of HMDs described in the first embodiment and the intra frame transmission timing control between the HMD and the PC described in the present embodiment may be used in combination.

[Example 3]
In the first embodiment, a description has been given of a mode in which a plurality of HMD users are synchronized and the intra frame transmission timing is shifted. In the present embodiment, a description will be given of a mode in which a PC that manages the entire MR system controls the intra-frame transmission timing of multiple HMDs.
FIG. 7 is a diagram showing a configuration example of an MR system according to the embodiment of the present invention.
In FIG. 7, the description regarding the common parts with the first embodiment is omitted. The MR system management PC 701 is a PC that manages and controls the intra frame transmission timing of each HMD. The MR system management PC 701 is wired to the router 105 in the same LAN.

FIG. 8 is a functional block diagram showing functions of the MR system management PC 701 necessary for realizing the embodiment of the present invention in the MR system shown in FIG.
The usage start message receiving unit 802 receives a usage start message from each HMD via the wired interface 801. When the use start message is received, the intra frame transmission timing calculation unit 803 calculates the intra frame transmission timing. The calculation method will be described later. The management DB 804 manages the intra frame transmission timing of all existing users. The management DB 804 corresponds to timing management means.

The synchronization information message transmission unit 805 packetizes the intra frame transmission timing calculated by the intra frame transmission timing calculation unit 803 and the synchronization information message including the synchronization time information so as to conform to the communication protocol. The synchronization information message transmission unit 805 corresponds to a timing transmission unit.
The packetized message is transmitted to the HMD via the wired interface 801.
The use end message receiving unit 806 receives a use end message from each HMD. The use end message receiving unit 806 corresponds to “stop notification receiving unit”. When the use end message is received, the intra frame transmission timing of the existing user managed by the management DB 804 is updated.

FIG. 9 is a flowchart showing an intra-frame transmission timing calculation method in intra-frame transmission timing calculation section 803 in FIG.
When a use start message is received from the new use HMD in S901, the process proceeds to S902. In S902, the management DB 804 is accessed to determine whether there is an existing user. If there is no existing user, the process proceeds to S903, and the intra frame transmission timing is set to zero. If there is an existing user, the process proceeds to S904. Since S904 and S905 are the same processes as S404 and S405 of FIG. 4 in the first embodiment, description thereof is omitted.

  An array for managing the intra frame transmission timing is acquired from the management DB 804. When the intra frame transmission timing is set in S903 or S905, the management DB is updated in S906. In S907, the set intra frame transmission timing and synchronization time information are transmitted to the HMD. The synchronization time information is the same as the information described in the sequence diagram of FIG.

  In this embodiment, the MR system management PC manages and controls the intra-frame transmission timings of a plurality of HMD users, and intentionally shifts the intra-frame transmission timings of the plurality of HMDs, thereby instantaneously transmitting the maximum transmission in the entire system. Reduce the amount of data. This makes it possible to effectively utilize the communication band and maximize the video quality in the MR system.

[Example 4]
In the first to third embodiments, the intra frame transmission timing of each HMD or each CG superposition PC is determined only at the start of use. However, in this embodiment, a mode in which the intra frame transmission timing is dynamically changed will be described. To do.
The configuration example of the MR system according to the embodiment of the present invention is the same as that of FIG.

FIG. 10 is a functional block diagram showing the functions of the MR system management PC 701 necessary for realizing the embodiment of the present invention in the MR system shown in FIG.
In FIG. 10, the description of the common parts with FIG. 8 shown in the third embodiment is omitted.
10 is different from FIG. 8 in that all HMD intraframe transmission timing recalculation section 1001 and intraframe insertion request reception section 1002 exist in FIG.
The all HMD intra frame transmission timing recalculation unit 1001 recalculates the intra frame transmission timing of all HMDs of the existing user. The all HMD intraframe transmission timing recalculation unit 1001 corresponds to a timing recontrol unit. The recalculation method and recalculation conditions will be described later.

  The intra-frame insertion request receiving unit 1002 receives a message requesting forcible intra-frame insertion from the HMD or CG superposition PC to the corresponding CG superposition PC or HMD outside the cycle timing. This message is transmitted / received when, for example, a video data packet being transmitted is lost due to a communication error, an error is propagated in inter-frame decoding, and the video frame needs to be refreshed.

FIG. 11 is a flowchart showing a method for recalculating intra frame transmission timings of all HMDs of existing users.
In S1101, it is determined whether the recalculation condition is satisfied. Examples of the recalculation conditions include the following conditions.
(1) When an intraframe insertion request message is received at the intraframe insertion request receiving unit 1002 (2) When a usage start message is received at the usage start message receiving unit 802 (the usage start message receiving unit 802 receives the transmission start request) (The usage start message corresponds to a transmission start request for encoded video data.)
(3) When a use end message is received by the use end message receiving unit 806 (the use end message receiving unit 806 corresponds to a stop notification receiving unit or a transmission end request receiving unit. The use end message is encoded video data. (This corresponds to a transmission stop notification or a transmission end request.)
(4) When the number of users exceeds the threshold value (5) When the difference between the minimum difference and the maximum difference between intra-frame transmission timings between the HMDs exceeds the threshold value (6) When a certain time has elapsed

  One of the above conditions may be used or combined. Another condition may be added. If it is determined in S1101 that the recalculation condition is satisfied, the process proceeds to S1102. In S1102, the number of users used is acquired from the management DB 804. In S1103, the fractional number of the value (= period / number of users) obtained by dividing the intra-frame insertion period predetermined in the MR system by the number of users (= number of encoded video data transmission devices) subject to timing recontrol. Is truncated and obtained as an integer. In S1104, the intra frame transmission timing of each user is reset.

  The resetting method will be described using a specific example shown in FIG. In FIG. 12A, there are five existing users (A to E), and the transmission timing of each intra frame is user A: 0, user B: 15, user C: 27, user D: 10, user. E: 4 is shown. The intra-frame transmission timing intervals are 4 between A and E, 6 between E and D, 5 between D and B, 12 between B and C, and 3 between C and A. . In this embodiment, the intra-frame insertion cycle is 30 and the number of users is 5, so (cycle / number of users) is 30/5 = 6. Here, the intra-frame transmission timing interval between the users is reset to 6.

  The state after resetting is shown in FIG. User A is 0, user B is 18, user C is 24, user D is 12, user E is 6. Note that the sort order of the user is re-established by taking into account the effects of resetting intra-frame transmission timing, for example, the effects of temporarily extending the intra-frame transmission interval, shortening the video quality, and increasing the amount of transmitted data. It is desirable to maintain the order before setting. After resetting the intra frame transmission timing of each user in S1104, the management DB 804 is updated in S1105. After the management DB 804 is updated, a message notifying the reset timing of the intra frame set for each user in S1106 is transmitted.

  In this embodiment, the MR system management PC manages and controls the intra-frame transmission timing of a plurality of HMD users, and dynamically resets the intra-frame transmission timing according to the situation, so that the instantaneous moment in the entire system can be obtained. Suppress the maximum transmission data volume. This makes it possible to effectively utilize the communication band and maximize the video quality in the MR system. In the present embodiment, the MR system management PC described in the third embodiment is used. However, the present invention is also applicable to a mode in which intraframe transmission timing is managed and controlled between the HMDs described in the first embodiment.

[Other Examples]
Although the embodiment has been described in detail above, the present invention can take an embodiment as a system, apparatus, method, program, recording medium (storage medium), or the like. Specifically, the present invention may be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to a device composed of a single device. good.

  The present invention also provides a software program that implements the functions of the above-described embodiments directly or remotely to a system or apparatus, and the system or apparatus computer reads out and executes the supplied program code. Achieved. The program in this case is a computer-readable program that corresponds to the flowchart shown in the drawing in the embodiment.

Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.
In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, or the like.

  Recording media for supplying the program include the following media. For example, floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD- R).

  As a program supply method, the following method is also possible. That is, the browser of the client computer is connected to a homepage on the Internet, and the computer program itself (or a compressed file including an automatic installation function) of the present invention is downloaded to a recording medium such as a hard disk. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. It is also possible to make it. That is, the user can execute the encrypted program by using the key information and install it on the computer.

Further, the functions of the above-described embodiments are realized by the computer executing the read program. Furthermore, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be realized by the processing.
Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, and then executed, so that the program of the above-described embodiment can be obtained. Function is realized. That is, based on the instructions of the program, the CPU provided in the function expansion board or function expansion unit can perform part or all of the actual processing.

101 HMD worn by user A
102 HMD worn by user B
103 HMD worn by user C
109 Real Space Video Data 110 Captured with HMD Weared by User A 110 Real Space Video Data Captured with HMD Weared by User B 111 Real Space Video Data Captured with HMD Weared by User C 112 User A PC CG video data 113 with CG superimposed on CG video data 114 with CG superimposed on PC for user B CG video data with CG superimposed on PC for user C

Claims (15)

A video encoded data transmission device connected to another video encoded data transmission device via a network,
Encoding means for encoding video data;
Timing control means for controlling intra-frame transmission timing, which is transmission timing of intra-frames generated in encoding by the encoding means;
Timing notification means for notifying the other video encoded data transmission device of information related to transmission timing controlled by the timing control means;
Timing information receiving means for receiving information on intra frame transmission timing in the other video encoded data transmitting apparatus from the other video encoded data transmitting apparatus;
Transmission means for transmitting the video encoded data encoded by the encoding means to an external data receiving device,
The timing control unit performs control so as to transmit the intra frame at a timing different from that of the other video encoded data transmission device based on the information on the intra frame transmission timing received by the timing information reception unit. A video encoded data transmitting apparatus. The timing control means includes
The maximum value of the interval of each intra-frame transmission timing in the intra-frame transmission period, which is the intra-frame transmission period from the intra-frame transmission timing of the other video encoded data transmission device received by the timing information receiving means. Seeking
2. The encoded video data transmission apparatus according to claim 1, wherein control is performed so that an intra frame of the own apparatus is transmitted in the middle of the maximum value of the interval of the intra frame transmission timing. Timing recontrol means for dynamically changing the intra frame transmission timing of the other video encoded data transmission device and the intra frame transmission timing of the own device, and
The timing re-control means has a value obtained by dividing an intra-frame transmission period, which is an intra-frame transmission period, by the number of video encoded data transmission apparatuses that are targets of timing re-control including its own apparatus. 2. The video according to claim 1, wherein the intra-frame transmission timing of each video encoded data transmission device is re-controlled so that the intra-frame transmission timing interval of each video encoded data transmission device is the same. Encoded data transmission device. Stop means for stopping transmission of video encoded data by the transmission means;
Stop notification means for notifying the other video encoded data transmission device of the transmission stop of the video encoded data when the stop means stops transmission of the video encoded data;
Stop notification receiving means for receiving a transmission stop notification, which is a notification of transmission stop of the video encoded data from the other video encoded data transmitting device,
4. The encoded video data according to claim 3, wherein the timing re-control unit re-controls the intra-frame transmission timing every time the transmission stop notification is received from the other encoded video data transmission apparatus. Transmitter device.   The timing notification unit notifies the video encoded data transmission device of information on the intra frame transmission timing every time an intra frame is transmitted at a timing different from the timing controlled by the timing control unit or the timing re-control unit. 5. The encoded video data transmission apparatus according to claim 3, wherein the encoded video data transmission apparatus is a video encoded data transmission apparatus.   The timing re-control unit re-controls the intra-frame transmission timing each time the timing information receiving unit receives information on the intra-frame transmission timing from the video encoded data transmission device. 6. The encoded video data transmission apparatus according to any one of items 1 to 5. A management device connected to one or more video encoded data transmission devices via a network,
A transmission start request receiving means for receiving a transmission start request, which is a request to start transmission of video encoded data from the video encoded data transmitting device;
Transmission end request receiving means for receiving a transmission end request, which is a request to end transmission of video encoded data from the video encoded data transmitting device;
Timing management means for managing intra-frame transmission timing, which is intra-frame transmission timing generated at the time of encoding by the video encoded data transmitting device;
Timing calculating means for calculating the intra frame transmission timing of the video encoded data transmitting device;
Timing transmission means for transmitting the intra frame transmission timing to the video encoded data transmission device,
The timing calculation means includes
When the transmission start request receiving means receives the transmission start request,
The intra frame transmission timing of the encoded video data transmission apparatus that has transmitted the transmission start request is not overlapped with the intra frame transmission timing of the encoded video data transmission apparatus already managed by the timing management unit. Further, the intra frame transmission timing of each video encoded data transmission device is newly calculated,
When the transmission end request receiving means receives the transmission end request,
The intra-frame of the remaining encoded video data transmission apparatus, excluding the encoded video data transmission apparatus that has transmitted the transmission end request, which is already managed by the timing management unit. A new calculation of the intra-frame transmission timing of each video encoded data transmission device so that the transmission timing interval widens;
A management device characterized by that.   The timing management unit newly sets the intra frame transmission timing newly calculated by the timing calculation unit when the transmission start request reception unit receives a transmission start request for encoded video data from the encoded video data transmission apparatus. The management device according to claim 7, wherein the management device is managed as follows.   The timing management means manages the intra frame transmission timing of the video encoded data transmitting apparatus when the transmission end request receiving means receives a video encoded data transmission end request from the video encoded data transmitting apparatus. The management apparatus according to claim 7, wherein the management apparatus is excluded from a target.   The timing calculation means calculates so as to transmit an intra frame in the middle of the maximum value of the interval of the intra frame transmission timings of the plurality of encoded video data transmission devices managed by the timing management means. The management apparatus according to any one of claims 7 to 9. Timing recalculation means for dynamically changing the intra frame transmission timing of the video encoded data transmission device,
The timing recalculation means is a value obtained by dividing the intra frame transmission period, which is an intra frame transmission period, by the number of the encoded video data transmission apparatuses managed by the timing management means. The intra-frame transmission timing of each encoded video data transmission apparatus is recalculated so as to be the interval of the intra-frame transmission timing of the encoded data transmission apparatus. The management apparatus as described in.   The said timing recalculation means recalculates the said intra-frame transmission timing of the said video encoding data transmission apparatus, whenever the transmission end request | requirement means receives a transmission end request | requirement. Management device. Intra frame insertion request receiving means for receiving an intra frame insertion request at a timing different from the timing calculated by the timing calculating means or the timing recalculating means,
12. The timing recalculation unit recalculates the intra frame transmission timing of each encoded video data transmission apparatus every time the intra frame insertion request reception unit receives an intra frame insertion request. 12. The management device according to 12.   The program for functioning a computer as each means which the video coding data transmission apparatus of any one of Claims 1 thru | or 6 has. The program for functioning a computer as each means which the management apparatus of any one of Claims 7 thru | or 13 has.

Images