JP2006325020A - Data processing apparatus and data processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To eliminate fluctuations when simultaneously receiving and reproducing video from a network camera via the Internet or the like.
Packet receiving means for simultaneously receiving a plurality of packetized video streams, stream dividing means for distributing received packets for each stream according to specific information in the packet, and storing the distributed stream data in units of image frames Image frame unit storage means, frame capture means for periodically retrieving stored image frames according to the frame rate of each stream, each packet of each moving picture stream is stored as image frame data, and the number of frames is set to a predetermined period And a frame capturing correction unit that detects a network fluctuation change amount from the counting results, calculates a correction timing of each stream based on the detected network fluctuation amount, and corrects the frame capturing unit.
[Selection] Figure 3

Description

  The present invention relates to a data processing device, a data processing method, a storage medium, and a program, and more particularly to a data processing device that can simultaneously receive and reproduce a plurality of moving image streams and the program.

  FIG. 1 is a diagram illustrating an example of a system configuration for providing a moving image distribution service configured by a conventional data processing apparatus. In this providing system, a personal computer (hereinafter referred to as “PC”) 1 can access various content servers 2-1 and 2-2 according to user instructions via the Internet 3. The PC 1 and the Internet 3 are connected by a line of a company providing Internet access called an Internet service provider (hereinafter referred to as “ISP”). The user starts up the browser software on the PC 1 and uses the Internet 3 and ISP from the WWW (World Wide Web) server that can distribute the WEB contents stored in the various content servers 2-1 and 2-2. It is accessed via the line and displayed in the browser on the display.

  Conventionally, with regard to the fluctuation control technique in a moving picture stream as described above, a technique of once buffering in a memory and absorbing fluctuation is common.

  As a first prior art example, claim 1 of Japanese Patent Laid-Open No. 8-139704 “Data Processing Device and Data Processing Method” includes an extracting means for extracting time information included in transmission data, and the extracted time information. And a storage means for storing the transmission data supplied to the extraction means, and a storage amount of the transmission data in the storage means. There is described a detecting means for detecting, and a reading control means for changing the reading rate of the transmission data by increasing or decreasing the storage amount corresponding to the detection result of the detecting means.

  As a second example of the prior art, claim 1 of JP-A-2000-22678 “Data transmission terminal device” is provided between networks, and stores a transmission buffer for storing transmission data and reception data. A data transmission terminal device for transmitting predetermined data to each other with a transmission clock different from the reception side transmission clock from the transmission buffer. When data is transmitted and accumulated in the reception buffer, a change amount of the accumulated data amount for each predetermined period is detected, and a correction clock corresponding to the change amount is obtained using a statistical method, The correction clock is corrected by the correction clock and is synchronized with the transmission clock. There.

  As a third prior art example, in claim 1 of Japanese Patent No. 3314700 “MPEG data transfer control circuit”, a buffer for sequentially storing received MPEG data, and MPEG data stored in the buffer are stored in advance. Compare the transfer rate control circuit for transferring to the decoder at a set transfer rate, the time information included in the MPEG data read from the buffer and the information indicated by the reference clock used for timing control of the decoder, and compare A time information comparison circuit for supplying an instruction to change the transfer rate to the transfer rate control circuit based on a result, wherein the transfer rate control circuit changes the transfer rate according to the instruction. Is described.

JP-A-8-139704 JP 2000-22678 A Japanese Patent No. 3314700

  However, in the first conventional example, the system clock that is the reference for processing the transmission data is generated based on the time information included in the transmission data, so that the influence on other systems controlled by the clock is great. Further, if the system clock of a circuit related to transmission data (audio / video) is separated in order to eliminate the influence, the cost increases and the circuit design becomes complicated, and the video is processed (processed). It becomes difficult.

  Further, in the second conventional example, the amount of change of the data amount of the predetermined data from the counterpart terminal device accumulated in the reception buffer is detected for each predetermined period, and the correction according to the amount of change is performed using a statistical method. A reception clock is obtained, and the reception-side transmission clock is corrected by this correction clock, so that synchronization with the transmission-side clock is achieved. Therefore, for example, when there are a plurality of data received from the partner terminal device and each data and transmission / reception are transmitted and received via different paths (for example, by radio frequency and channel), the partner terminal device Even if a transmission / reception buffer is prepared for each codec or a transmission / reception buffer is prepared for each codec, it is difficult to generate a correction clock from the amount of change in the reception buffer.

  In the third conventional example, in the MPEG data transfer control circuit that sequentially stores MPEG data transmitted through a non-guaranteed network such as a LAN or the Internet, the length of the MPEG data becomes variable. In the case where the order of the MPEG data itself is changed or an event occurs such that one packet data is dropped, it is difficult for the circuit to perform normal processing.

  The present invention has been made in view of such a situation, and simultaneously receives images from a plurality of network cameras 4-1, 4-2, 4-3 as shown in FIG. 2 via the Internet 3 or the like. In this case, it is possible to receive multiple video streaming contents through different routes such as the Internet and wireless, and minimize the influence of the clock on other circuit systems, eliminating network fluctuations. An object is to provide a data processing apparatus and a data processing method.

  In order to achieve the above object, in the present invention, a packet receiving means for simultaneously receiving a plurality of packetized moving image streams, and a stream for sorting packets received by the packet receiving means according to specific information in the packet and distributing the packets for each stream A dividing unit, an image frame unit storing unit that stores the stream data divided by the stream dividing unit in an image frame unit, and an image frame for each stream stored by the image frame unit storing unit, Frame capturing means periodically fetched according to the frame rate of the stream, and each packetized packet of each moving picture stream is stored as image frame data by the image frame unit storage means, and the number of frames of the stored image frame data Where Frame counting means that counts every period and the frame counting means of each video stream are combined to detect the network fluctuation change amount, and the correction timing of each stream is calculated based on the network fluctuation change amount, There is provided a data processing apparatus comprising: a frame capturing correction unit that corrects the frame capturing unit according to the correction timing.

  According to the present invention, a data processing apparatus that receives a plurality of moving picture streaming contents through different paths such as the Internet and wireless, and eliminates the fluctuation of the network by minimizing the influence of the clock on other circuit systems. In addition, there is an effect that a data processing method can be provided.

A data processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.
(First embodiment)
The data processing apparatus according to the first embodiment of the present invention will be described below with reference to FIGS. FIG. 3 is a block diagram illustrating a configuration example of the data processing apparatus according to the first embodiment.

  As shown in FIG. 3, the data processing apparatus according to the first embodiment performs separate streams of moving images (called “ES: Elementary stream” in MPEG-4) and audio that are not MUX (multiplexed or multiplexed). A multi-stream packet receiving unit 101 capable of receiving a plurality of packets simultaneously, a packet distributing unit 102 having a function of distributing a packet to a buffer for each stream, and an A stream frame for buffering A stream packets in units of frames A storage unit 103-1, a B stream frame storage unit 103-2 that buffers B stream packets in units of frames, an n stream frame storage unit 103-3 that buffers n stream packets in units of frames, and A Stream frame storage A stream frame counting unit 104-1 that recognizes the frame delimiter of the A stream stored in 103-1, and counts the number of frames, and recognizes the frame delimiter of the B stream stored in the B stream frame storage unit 103-2 The B stream frame counting unit 104-2 that counts the number of frames and the n stream frame counting unit 104-that recognizes the frame delimiter of the n streams stored in the n stream frame storage unit 103-3 and counts the number of frames. 3, the periodic frame number management unit 105-1 of the A stream frame, the periodic frame number management unit 105-2 of the B stream frame, the periodic frame number management unit 105-3 of the n stream frame, and the respective stream frame counts Information from the sections 104-1, 104-2, 104-3 Fluctuation management unit 109 that calculates network fluctuations and manages fluctuations of each stream in an integrated manner, Codec-A (Codec A) 110-1 that is an A stream dedicated codec, and Codec- that is a B stream dedicated codec B (codec B) 110-2, Codec-n (codec n) 110-3, which is a dedicated codec for n streams, and frame control for controlling the capture timing of each codec 110-1, 110-2, 110-3 Part 113.

In the following description, it is assumed that the moving image recording recording method is the MPEG-4 recording method unless otherwise specified.
Next, the operation of the data processing apparatus according to the first embodiment will be described.
FIG. 4 is a flowchart showing the operation of the data processing apparatus according to the first embodiment.

  Here, the multi-stream packet receiving unit 101 is means for receiving a plurality of moving image / audio stream packets from a plurality of cameras 4-1, 4-2, 4-3 and the like as shown in FIG. In the present embodiment, the moving images and audio from the cameras 4-1, 4-2, 4-3 are received as separate streams that are not multiplexed (not multiplexed). The control of stream selection received from each of the cameras 4-1, 4-2, 4-3 is performed by controlling the position information such as URL (Uniform Resource Locator) by another layer protocol (for example, HyperText Transfer Protocol (HTTP)). For example, acquisition and retrieval by acquiring location information by SIP (Session Initiation Protocol) or the like.

Each moving image and audio stream is delivered by a transmission protocol such as RTP (Real-time Transport Protocol). Furthermore, control of stream reception itself (session establishment, playback start, pause, playback restart, playback stop, etc.) is controlled by RTSP (Real Time Streaming Protocol) or the like.
In the first embodiment, it is assumed that information on the type and frame rate of each received stream is obtained in advance using the protocol.

  The various mixed packets received by the multi-stream packet receiving unit 101 in step S1000 are separated into individual streams by the packet sorting unit 102, and stored in separate buffers for each stream (S1010). For example, a label used for a label switch is used as the information used for separating the individual streams. It is assumed that the label is added in units of streams before the stream is transmitted from each camera 4-1, 4-2, 4-3.

  Further, in the first embodiment, each stream is divided into three streams of A stream, B stream, and n stream, respectively, and an A stream frame storage unit 103-1, which is a buffer, and B stream frame storage. Stored in the unit 103-2 and the n-stream frame storage unit 103-3. Here again, each frame unit is attached with a label similar to that described above so that each frame can be distinguished, and the change in the label can be recognized as a frame break. Buffering in units of frames is possible (S1020). Note that the same label may be used as the label for each frame as long as the streams are different.

  In addition, as a fluctuation control process, each of the periodic frame number management units 105-1, 105-2, and 105-3 operates a separate process for counting the number of arrived frames per certain period (S1050). The periodic frame number management units 105-1, 105-2, and 105-3 reach the corresponding stream frame storage units 103-1, 103-2, and 103-3 at regular intervals (S1051). Are acquired from the stream frame counting units 104-1, 104-2, and 104-3 and stored in the memory as the number of periodic frames (S1052). Note that the arrival frame count processing routine per certain period is repeated until the end flag is set (S1053).

  Each stream frame counting unit 104-1, 104-2, 104-3 recognizes the frame break due to the label change (S1030), counts the number of arrived frames (S1040), and information about the frame break and each period The fluctuation frame management unit 109 is notified of the number of periodic frames and the like acquired from the frame number management units 105-1, 105-2, and 105-3.

  The fluctuation overall management unit 109 that has received information such as frame break information and the number of periodic frames from each stream frame counting unit 104-1, 104-2, 104-3 is based on the number of periodic frames for each stream. The total network fluctuation is calculated, and then the frame arrival fluctuation correction timing for each stream is determined (S1060) and notified to the frame controller 113 together with the frame arrival timing for each stream.

  The frame control unit 113 that has received the frame arrival fluctuation correction timing for each stream and the frame arrival timing for each stream considers the frame rate in each stream and if correction is necessary (S1070), the frame arrival fluctuation A frame acquisition instruction is issued to the codecs 110-1, 110-2, 110-3 for each stream according to the correction timing and the frame arrival timing (S1080). On the other hand, if correction is not necessary, a frame acquisition instruction is issued to the codecs 110-1, 110-2, and 110-3 for each stream at the frame arrival timing or the frame rate timing.

  The codecs 110-1, 110-2, and 110-3 for each stream that have received the frame acquisition instruction from the frame control unit 113 correspond to one frame from each stream frame storage unit 103-1, 103-2, and 103-3. Is obtained (S1090), decoded (S1100), and passed to the next process.

  As described above, according to the first embodiment, it is possible to realize hardware by controlling using a label at a moving image stream and frame level, and at a higher speed and minimizing an in-device delay. There is an effect that can be done.

(Second Embodiment)
Next, a second embodiment of the present invention will be described.
FIG. 5 is a block diagram illustrating a configuration example of a data processing device according to the second embodiment.

  As shown in FIG. 5, the data processing apparatus according to the second embodiment has separate video and audio streams that are not MUX (multiplexed, multiplexed) (referred to as ES: Elementary stream in MPEG-4) and audio. A multi-stream packet receiving unit 201 capable of receiving a plurality of packets simultaneously, a packet distribution unit 202 having a function of distributing a packet to a buffer for each stream, a packet order control according to each stream, and data in the packet Consistency Check & Correction Unit 203-x that repairs errors that occurred on the network of No. 1 by FEC (Forward Error Correction) or the like, video stream No. that buffers the video stream packet of x in frame units. x frame storage unit 204-x; No. for buffering the audio stream packet of x. x audio stream storage unit 205-x and video stream No. No. stored in the x frame storage unit 204-x. The video stream No. that recognizes the frame break of the x stream and counts the number of frames x frame number counting unit 206-x; x No. for counting the number of packets stored in the audio stream storage unit 205-x at a constant period. x audio packet number counting unit 207-x and moving picture stream No. a moving image cycle frame number management unit 208-x that samples and counts the number of frames counted by the x frame number counting unit 206-x at a constant period and records the number of the periodic frames; x frame number counting unit 206-x and No. 1 Prepared by configuring a fluctuation overall management unit 210 that manages network fluctuations based on data notified from the x audio packet number counting unit 207-x, and a hard codec corresponding to the video and audio codecs that have been received. The stream control unit 211 that manages the combination of various storage units by switching the switch 213 and manages the decoding timing in the codec module 212-x, and the codec module that can cope with various video and audio decoding by configuration 212-x and the video stream No. x frame storage unit 204-x or No. The switch 213 connects the x audio stream storage unit 205-x and the codec module 212-x. In the above description, “x” added to the reference sign is a subscript, and x is a natural number.

Next, the operation of the data processing apparatus according to the second embodiment will be described.
6 and 7 are flowcharts showing the operation of the data processing apparatus according to the second embodiment.

  Here, the multi-stream packet receiving unit 201 is means for receiving a plurality of moving image / audio stream packets from a plurality of cameras 4-1, 4-2, 4-3, and the like as shown in FIG. In the present embodiment, the moving images and audio from the cameras 4-1, 4-2, 4-3 are received as separate streams that are not multiplexed (not multiplexed). In the same manner as in the first embodiment, control of stream selection received from each of the cameras 4-1, 4-2, 4-3 is performed by URL (Uniform using a protocol of another layer (for example, HyperText Transfer Protocol (HTTP)). (Location acquisition such as Resource Locator) or location information by SIP (Session Initiation Protocol) or the like).

  Each moving image and audio stream is delivered by a transmission protocol such as RTP (Real-time Transport Protocol). Furthermore, control of stream reception itself (session establishment, playback start, pause, playback restart, playback stop, etc.) is controlled by RTSP (Real Time Streaming Protocol) or the like.

  In the second embodiment, information such as the type and frame rate of each received stream is obtained in advance using the protocol, and the codec type of each of video and audio is determined by the stream type information. At the same time, the codec module 212-x is configured using the stream control unit 211, and the necessary number of codec modules 212-x for the moving image codec and the audio codec are prepared.

  Further, a storage unit as a buffer is prepared for each video and audio codec type based on the stream type information and the like. The number of the codec module 212-x configured for the corresponding video codec matches the number of the x frame storage unit 204-x, and The switch 213 is switched using the stream control unit 211 so that the number of the x audio stream storage unit 205-x matches the number of the codec module 212-x configured in the corresponding audio codec. Specifically, the video stream No. x-frame storage unit 204-x and No. The switch 213 is switched so that the x audio stream storage unit 205-x and the codec module 212-x have the correspondence shown in FIG.

  The various mixed packets received by the multi-stream packet receiving unit 201 in step S2000 are separated into individual streams by the packet distribution unit 202, and in particular, passed to separate processing for each of the moving image and audio streams (S2010). . For example, a label used for a label switch is used as the information used for separating the individual streams. The label is added to each stream before the stream is transmitted from each of the cameras 4-1, 4-2 and 4-3. Each has a different label to distinguish the frames.

  Packets separated from the video distribution unit 202 into each moving image and audio stream are processed by the consistency check & correction unit 203-x according to the characteristics of each stream, such as changing the order of packets and restoring packet data. And stored in a buffer for each stream (S2020 and S2200). Also, in the video stream, the same and different labels as described above are added to each video frame so that each video frame can be distinguished, and the change in the label is recognized as a break of the video frame. This is possible, and buffering in units of frames is possible by using the label (S2030). Note that the same label may be used as the label for each moving image frame as long as the moving image stream is different.

  Next, as the fluctuation control process, in each moving image periodic frame number management unit 208-1, 208-2, an arrival frame number counting routine per certain period is operating separately (S2060). The moving image cycle frame number management units 208-1 and 208-2 each correspond to each moving image stream No. every predetermined cycle (S 2061). The number of frames of each stream that has reached the x frame storage units 204-1 and 204-2 is assigned to each video stream No. It is acquired from the x frame number counting units 206-1 and 206-2 and stored in the memory as the number of periodic frames (S2062). Note that the arrival frame count processing routine per certain period is repeated until the end flag is set (S2063).

  In addition, each No. x voice packet number counting units 207-1 and 207-2 correspond to the corresponding Nos. x audio stream storage units 205-1 and 205-2 (S2210) The audio packet sequential counting process (S2220) for counting up the number of audio packets every time an audio packet is stored, and every certain period A voice packet cycle sampling notification process (S2230) for recording the number of packets counted by the voice packet sequential counting process and notifying the fluctuation overall management unit 210 is executed.

  A frame break due to the label change is recognized (S2040). The x frame number counting units 206-1 and 206-2 count the total number of arrived frames (S2050), and information on frame breaks and the number of periodic frames acquired from the moving image periodic frame number management units 208-1 and 208-2. And the like are notified to the fluctuation management unit 210.

  Each video stream No. x frame number counting units 206-1 and 206-2, information on frame breaks, information such as the number of periodic frames, etc. The fluctuation supervision managing unit 210 that has received the information such as the voice packet cycle sampling value from the x voice packet number counting units 207-1 and 207-2 analyzes the various information of the related video stream and voice stream, and calculates the total Network fluctuations are calculated, and then the frame arrival fluctuation correction timing for each associated video and audio stream pair is determined. The fluctuation correction timing is calculated together with the frame arrival timing for each video stream, and the fluctuation correction timing for each audio stream. Only the stream control unit 211 is notified (S2070).

  The stream control unit 211 that has received the frame arrival timing and fluctuation correction timing in each video stream and the fluctuation correction timing in each audio stream first corrects each video stream in consideration of the frame rate in each video stream. Is required (S2080), a frame acquisition instruction is issued to the codec module 212-x configured in the codec corresponding to each moving picture stream based on the fluctuation correction timing and the frame arrival timing (S2090). Further, if correction is necessary for each audio stream based on the fluctuation correction timing in each audio stream (S2090), the codec configured to the codec corresponding to each audio stream based on the fluctuation correction timing. The sound acquisition timing of the module 212-x is corrected.

The codec module 212-x configured with the codec corresponding to each moving image stream that has received the frame acquisition instruction from the stream control unit 211 receives the moving image stream No. Data for one frame is acquired from the x frame storage units 204-1 and 204-2 (S2100), and after decoding is performed (S2110), it is transferred to the next process.
In addition, the codec module 212-x configured to the codec corresponding to each audio stream, which has received the correction information from the stream control unit 211, has each No. 1 in the corrected cycle. A certain amount of data is acquired from the x audio stream storage units 205-1 and 205-2 (S2260), and after decoding is performed (S2270), it is passed to the next process.

  As described above, according to the second embodiment, there is an effect that fluctuation and delay control can be performed even in a configuration in which a codec module can be reused without having a fixed codec. Also, by installing an error recovery function such as FEC for each stream, it is possible to control fluctuations and delays of each stream with a minimum in-device delay such as, for example, video packet repair does not affect audio packet delivery delay. There is also an effect that.

(Other embodiments of the present invention)
The present invention is not limited to each of the above embodiments, and the following modifications are possible.
In the first and second embodiments, the packet distribution units 102 and 202 perform packet distribution using the label of the label switch. However, the packet distribution unit 102 and 202 may perform packet distribution according to an address itself having stream identification information.
In the first and second embodiments, the moving picture recording format of the moving picture stream may be other than the MPEG-4 recording method, such as Motion-JPEG, MPEG2, H264, DivX, XVID, Windows (registered trademark) Media9, or the like. May be.

  In the first and second embodiments, the fluctuation supervision management units 109 and 210 calculate network fluctuations based on the number of arrivals of moving image frames and the number of arrivals of voice packets by periodic processing. A statistical technique is used for this processing. Calculation may be performed by adding the expected fluctuation as a coefficient. Further, the fluctuation supervision management units 109 and 210 calculate the network fluctuation by notifying the periodic frame number at the frame break of the video stream, but the present invention is not limited to this timing, and the frame break notice and the periodic frame number notice are calculated. These timings may be divided and notified at each timing, and in this case, the network fluctuation may be calculated at the frame rate timing or the like.

  In the second embodiment, the video stream No. In the x frame storage unit 204-x, when a buffer overflow of a stream is likely to occur in the storage unit as the buffer, a long storage unit is dynamically prepared separately from the current storage unit, and the packet distribution unit 202 The distribution destination may be changed, the switch 213 may be switched to the target codec module so that the fluctuation control does not stop, the consistency check & correction unit 203-x and the video stream Nos. Of multiple sizes. An x frame storage unit may be provided to change the distribution destination of the packet distribution unit 202 so that the fluctuation control does not stop.

Also, a software program for realizing the functions of the above-described embodiment for a computer in an apparatus or a system connected to the various devices so that the various devices are operated to realize the functions of the above-described embodiments. What was implemented by supplying the code and operating the various devices according to a program stored in the computer (CPU or MPU) of the system or apparatus is also included in the scope of the present invention.
In this case, the program code of the software itself realizes the functions of the above-described embodiments, and the program code itself constitutes the present invention. Further, means for supplying the program code to the computer, for example, a recording medium storing the program code constitutes the present invention. As a recording medium for storing the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.
Further, by executing the program code supplied by the computer, not only the functions of the above-described embodiments are realized, but also the OS (operating system) or other application software in which the program code is running on the computer, etc. It goes without saying that the program code is also included in the embodiment of the present invention even when the functions of the above-described embodiment are realized in cooperation with the embodiment.
Further, after the supplied program code is stored in the memory provided in the function expansion board of the computer or the function expansion unit connected to the computer, the CPU provided in the function expansion board or function expansion unit based on the instruction of the program code Needless to say, the present invention includes a case where the functions of the above-described embodiment are realized by performing part or all of the actual processing.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

It is a figure which shows the structure of the provision system of the conventional moving image delivery service. It is a figure which shows the structure of the provision system of the moving image delivery service which can apply this invention. It is a block diagram which shows the structural example of the data processor by 1st Embodiment. It is a flowchart which shows operation | movement of the data processor by 1st Embodiment. It is a block diagram which shows the structural example of the data processor by 2nd Embodiment. It is a flowchart which shows operation | movement of the data processor by 2nd Embodiment. It is a flowchart which shows operation | movement of the data processor by 2nd Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Personal computer 2 ... Server 3 ... Internet 4-x ... Movie camera 101,102 ... Multi-stream packet receiving part 102,202 ... Packet distribution part 103-x ... x stream frame storage part 104-x ... x stream frame number count 105-x, 208-x ... x periodic frame number management unit 109, 210 ... fluctuation overall management unit 110-x ... codec-x
113: Frame control unit 203-x: Consistency check & correction unit 204-x: Stream No. x frame storage unit 205-x. x audio stream storage unit 206-x. x frame number counting section 207-x. x voice packet number counting unit 211 ... stream control unit 212-x ... codec module 213 ... switch

Claims (17)

A packet receiving means for simultaneously receiving a plurality of packetized video streams;
Stream dividing means for determining the specific information in the packet and distributing the packets received by the packet receiving means for each stream;
Image frame unit storage means for storing the stream data divided by the stream dividing means in image frame units;
Frame fetching means for periodically fetching image frames for each stream stored by the image frame unit storage means according to the frame rate of each stream;
Each packetized packet of each video stream is stored as image frame data by the image frame unit storage means, and a frame counting means for counting the number of frames of the stored image frame data every predetermined period;
Collecting the counting results of the frame counting means of each video stream, detecting the network fluctuation change amount, calculating the correction timing of each stream based on the network fluctuation change amount, and correcting the frame capturing means based on the correction timing A data processing apparatus comprising: a frame capturing correction unit that performs the processing.   2. The data processing apparatus according to claim 1, wherein a label is used as the specific information for distributing the packet received by the packet receiving means for each stream.   The data processing apparatus according to claim 1, wherein the packetized moving image stream is a single moving image stream that is not multiplexed with audio.   2. The data processing apparatus according to claim 1, wherein the packetized moving picture stream is a single moving picture stream that is not multiplexed with audio and is an MPEG-ES (ES: Elementary stream).   2. The data processing apparatus according to claim 1, wherein the frame capturing means is a video stream decoder or codec. A packet receiving means for simultaneously receiving a plurality of packetized video streams;
Stream dividing means for determining the specific information in the packet and distributing the packets received by the packet receiving means for each stream;
Image frame unit storage means for determining the specific information in the packet and storing the stream data divided by the stream dividing means in image frame units;
Frame decoding means for periodically extracting and decoding image frames for each stream stored by the image frame unit storage means according to the frame rate of each stream;
Each packetized packet of each video stream is stored as image frame data by the image frame unit storage means, and a frame counting means for counting the number of frames of the stored image frame data every predetermined period;
Counting result notifying means for notifying the next processing of the counting result counted by each frame counting means at the timing when one frame is aligned by the frame unit storing means;
A frame for detecting a network fluctuation change amount by collecting the count results of the respective count result notification means, calculating a correction timing for each stream based on the network fluctuation change amount, and correcting the frame decoding means based on the correction timing. A data processing apparatus comprising: a decoding correction unit.   7. The data processing apparatus according to claim 6, wherein a label is used as the specific information for distributing the packet received by the packet receiving means for each stream.   7. The data processing apparatus according to claim 6, wherein the packetized moving image stream is a single moving image stream that is not multiplexed with audio.   7. The data processing apparatus according to claim 6, wherein the packetized moving image stream is a single moving image stream that is not multiplexed with audio and is an MPEG-ES (ES: Elementary stream).   7. The data processing apparatus according to claim 6, wherein the frame capturing means is a video stream decoder or codec. A packet receiving step for simultaneously receiving a plurality of packetized video streams;
A stream distribution step of determining specific information in the packet and distributing the packet received by the packet reception step for each stream; and
An image frame unit storage step for determining the specific information in the packet and storing the stream data distributed in the stream distribution step in a memory in image frame units;
A frame capturing step of periodically taking out the image frame for each stream stored in the image frame unit storing step according to the frame rate of each stream;
A packet counting step in which each packetized packet of each moving image stream is stored in a memory as image frame data in the image frame unit storing step, and the number of frames of the stored image frame data is counted every predetermined period;
The network fluctuation change amount is calculated by collecting the counting results of the frame counting steps of each video stream, the correction timing of each stream is calculated based on the network fluctuation change amount, and the frame capturing step is corrected based on the correction timing. And a frame capturing correction step.   12. The data processing method according to claim 11, wherein a label is used as the specific information for distributing the packet received by the packet receiving step for each stream.   12. The data processing method according to claim 11, wherein the packetized moving image stream is a single moving image stream that is not multiplexed with audio.   12. The data processing method according to claim 11, wherein the packetized moving picture stream is a single moving picture stream that is not multiplexed with audio and is an MPEG-ES (ES: Elementary stream).   12. The data processing method according to claim 11, wherein the frame capturing step also serves as a video stream decoder or codec step. A packet receiving step for simultaneously receiving a plurality of packetized video streams;
A stream distribution step of determining specific information in the packet and distributing the packet received by the packet reception step for each stream; and
An image frame unit storage step for determining the specific information in the packet and storing the stream data distributed in the stream distribution step in a memory in image frame units;
A frame capturing step of periodically taking out image frames for each stream stored in the image frame unit storing step according to a frame rate of each stream;
A packet counting step in which each packetized packet of each moving image stream is stored in a memory as image frame data in the image frame unit storing step, and the number of frames of the stored image frame data is counted every predetermined period;
The network fluctuation change amount is calculated by collecting the counting results of the frame counting steps of each video stream, the correction timing of each stream is calculated based on the network fluctuation change amount, and the frame capturing step is corrected based on the correction timing. A program for causing a computer to execute a frame capturing correction step.   A computer-readable storage medium storing the program according to claim 16.

Images