JP2005229587A - Multiplexing conversion device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a multiplexing conversion device capable of preventing the display of moving image data that cannot be correctly decoded for a user.
A conversion unit 103 of a multiplexing system conversion apparatus determines a packet loss of a TS packet, and when a packet loss is detected, performs error processing and outputs packet data after error processing. The determination unit 202 includes a box creation unit 206 that creates moov including error information and mdat based on the input AU data and the display time in the PES header.
[Selection] Figure 2

Description

  The present invention relates to a multiplexing system conversion apparatus that converts a multiplexing system of encoded media data such as moving images and voices and outputs the converted data.

  In recent years, devices and services that handle encoded multimedia data such as moving images and voices have become widespread as storage media and communication networks have increased in capacity and transmission technology has advanced. For example, in the broadcasting field, broadcasting of digitally encoded media data has started instead of conventional analog broadcasting. The current digital broadcasting is intended only for fixed reception, but in the future broadcasting for mobiles such as mobile phones is also planned. Also in the communication field, an environment for handling multimedia data on both fixed terminals and mobile terminals has been established, such as the launch of a video distribution service for third-generation mobile phones. In view of these backgrounds, content data received via broadcasting or the Internet is recorded on a memory card such as an SD (Secure Digital) card or an optical disc such as a DVD-RAM (Digital Versatile Disk-Rewritable), and it is used between devices. It is expected that usage methods such as sharing content data will spread.

  When media data is broadcast, stored, or distributed via a network, header information and media data necessary for reproducing the media data are multiplexed. In multiplexing, standard multiplexing schemes are standardized for storage devices such as broadcast and DVD, and for mobile units. First, in digital broadcasting and DVD, the MPEG-2 (Moving Picture Expert Group) system standard standardized by ISO / IEC JTC1 / SC29 / WG11 (International Standardization Organization / International Engineering Consortium) is used. In mobile terminals, TS26.234 (Transparent end), which was established as a wireless video distribution standard by the 3GPP (Third Generation Partnership Project), an international standardization organization aimed at standardizing third-generation mobile communication systems. -to-end packet switched streaming service) adopts the MP4 file format standardized by ISO / IEC JTC1 / SC29 / WG 11.

  In addition, since MPEG-4 AVC (Advanced Video Coding) has been standardized as a successor to the currently popular MPEG-2 Visual and MPEG-4 Visual as a moving picture encoding system, MPEG-4 AVC will be used in the future. It is expected that encoded video data is multiplexed, broadcasted, stored, or distributed according to the MPEG-2 system standard or MP4 file format (hereinafter referred to as MP4).

  The outline of the encoded data multiplexing method in the MPEG-2 system and MP4 will be described below. In the MPEG-2 system and MP4, since an access unit (AU) is used as a basic unit when handling encoded data, the structure of the AU will be described first. AU is a unit including encoded data for one picture, and AU data in MPEG-4 AVC has a structure shown in FIG. In MPEG-4 AVC, in addition to data essential for picture decoding, auxiliary information called SEI (Supplemental Enhancement Information) that is not essential for decoding, AU boundary information, and the like can be included in AU data. Are stored in a NAL (Network Adaptation Layer) unit. In the MPEG-2 system, a NAL unit indicating the start of an AU called “Access Unit Delimiter” is always added to the head of the AU. The NAL unit is composed of a header and a payload as shown in FIG. 24A, the header size is 1 byte, and a field indicating the type of data stored in the payload (hereinafter referred to as a NAL unit type). included. The NAL unit type has a value defined for each type of data such as slice and SEI, and refers to the NAL unit type when acquiring the type of data stored in the NAL unit. As shown in FIGS. 24B and 24C, in addition to slice data for one picture, the AU stores NAL units such as header information and SEI. The NAL unit stores NAL unit data. Since there is no information for identifying the boundary, the boundary information can be added to the head of each NAL unit when storing the AU. The boundary information includes a method of adding a start code prefix indicated by 3 bytes of 0x000001 as shown in FIG. 24B (hereinafter referred to as a byte stream format), and the size of the NAL unit as shown in FIG. There are two types of methods (hereinafter referred to as NAL size format). Note that it is stipulated that one or more zero_bytes (one byte whose value is 0x00) is added before the start code prefix for the head NAL unit of the AU and a NAL unit having a specific NAL unit type value. ing. In the MPEG-2 system, the byte stream format is used, and in MP4, the NAL size format is used.

  Next, the slice and header information will be described in detail. The slices are divided into two types: IDR (Instantaneous Decoder Refresh) slices and other slices. An IDR slice is slice data that is intra-coded, and header information such as SPS (Sequence Parameter Set) described later can be switched only in the IDR slice. When an IDR slice is included in a picture, all other slices in the same picture are also IDR slices. Therefore, an AU including an IDR slice is hereinafter referred to as an IDR AU. Also, the unit composed of the AU immediately before the next IDR AU is called a sequence from the IDR AU, and when decoding the AU slice data, only the AU in the sequence is referenced, so that random access is made in sequence units. can do. Next, there are two types of header information, SPS and PPS (Picture Parameter Set), SPS is fixed header information in sequence units, and PPS is header information that can be switched in picture units. There can be a plurality of SPS and PPS, and each SPS or PPS is distinguished by an index number. Further, one SPS or PPS is stored in one NAL unit. The index numbers of SPS and PPS that each picture refers to are obtained as follows. First, the PPS index number referenced by the picture is indicated in the header portion of the slice data. Next, since the PPS indicates the index number of the SPS referred to by the PPS, the index number of the SPS referenced by the picture is obtained by analyzing the PPS referenced by the picture. SPS and PPS referred to by a picture are indispensable when decoding slice data of the picture.

  Next, a method for multiplexing AU data by the MPEG-2 system in broadcasting will be described. In the MPEG-2 system, encoded data is first multiplexed into a PES (Packetized Elementary Stream) packet, and further the PES packet is multiplexed into a TS (Transport Stream) packet. FIGS. 25A and 25B show the structures of the PES packet and the TS packet, respectively. Access unit (AU) data is stored in the payload of the PES packet. (1) to (3) in FIG. 25 (a) show an example of storing AU data in the payload of the PES packet. As shown in (1) and (2), one or more AUs are grouped together. The AU data may be divided and stored as shown in (3). Further, the stuffing data can be included in the payload separately from the AU data. The header of the PES packet starts with a start code prefix of 3 bytes of 0x000001 and a 4-byte start code composed of a 1-byte stream ID. The stream ID is an identification number indicating the type of encoded data included in the payload data of the PES packet. In MPEG-4 AVC, the stream ID can take any value from 0xE0 to 0xEF. The header can store the decoding time and display time of the first AU starting in the payload, but these time information is not necessarily stored in all PES packets, and PES in which time information is not stored is stored. There are also packets. When AU time information whose decoding time or display time is not indicated by the header of the PES packet is required, AU data is analyzed, and a difference value of the decoding time or display time from the immediately preceding AU is acquired. The start position of the PES packet is detected by searching for a 4-byte start code in the payload data of the TS packet. On the other hand, as shown in FIG. 25B, the data of the PES packet is divided and stored in the payload of the TS packet. The TS packet is a fixed-length packet having a size of 188 bytes, and includes a 4-byte header, an adaptation field, and payload data. Note that the adaptation field exists only when a specific flag in the header is set. The header includes an identification number called PID indicating the type of data transmitted by the TS packet and a counter called continuity_counter. The continuity_counter is a 4-bit field. In a TS packet with the same PID, the continuity_counter increases by 1 in the order of transmission and circulates when the maximum value is reached. The correspondence between the PID of the TS packet and the type of data transmitted by the TS packet is provided by program information separately transmitted by the TS packet. For this reason, when a TS packet is received, first, the PID of the TS packet is acquired, and the packet is distributed according to the value of the PID. For example, when the program information acquired at the start of reception indicates that MPEG-4 AVC data is transmitted by a TS packet with a PID of 32, by acquiring the TS packet with a PID of 32, MPEG-4 AVC AU data can be acquired. Here, when a gap occurs in the continuity_counter value of the received TS packet, it indicates that a packet loss has occurred in the transmission path. Further, when separating AU data from TS packets, PES packets are separated from payload data of TS packets, and AU data is separated from the separated PES packets.

Finally, a method for multiplexing AU data in MP4 will be described. In MP4, header information and media data in units of samples are managed in units of objects called boxes. Here, a sample is a basic unit for handling media data in MP4, and one sample corresponds to 1 AU. Each sample is assigned a sample number so as to be in ascending order in decoding time order, and the sample number increases by one for each sample. FIG. 26 (a) shows the box structure, which is composed of the following fields.
size: Size of the entire Box including the size field
type: A box identifier, usually represented by four alphabetic characters. The field length is 4 bytes, and when a Box is searched in the MP4 file, it is determined by determining whether or not the continuous 4 bytes of data matches the identifier of the type field.
version: Box version number
flags: Flag information data set for each box: header information and media data are stored.

  In addition, since version and flags are not essential, these fields do not exist depending on the Box. Hereinafter, the identifier of the type field is used to refer to Box. For example, a Box whose type is 'moov' is called moov. The Box structure in the MP4 file is shown in FIG. The MP4 file is composed of ftyp, moov, mdat, or moof, and ftyp is arranged at the head of the file. ftyp includes information for identifying the MP4 file, and media data is stored in mdat. Each media data included in mdat is called a track, and each track is identified by a track ID. Next, header information about samples included in each track of mdat is stored in moov. In moov, as shown in FIG. 27 (a), Boxes are arranged hierarchically, and header information is stored in separate traks for each media track such as audio and video. Even in trak, Boxes are arranged in a hierarchical manner, and sample size, decoding time, display start time, information on randomly accessible samples, and the like are stored in each Box in stbl (FIG. 27 (b)). . Randomly accessible samples are called sync samples, and a list of sample numbers of sync samples is indicated by stss in stbl. In the above description, the header information of all the samples in the track is stored in moov. However, the track can be divided and fragmented, and header information can be stored in fragment units. The header information for the unit obtained by dividing the track is indicated by moof. In the example of FIG. 28, the header information of the sample stored in mdat # 1 can be stored in moof # 1.

Consider a case in which broadcast data received at a mobile terminal is transmitted from a mobile phone as an attachment to an e-mail. 3GPP stipulates that media data is multiplexed by MP4 when handling moving images, voices, and the like in electronic mail. For this reason, it is necessary to convert the multiplexing method from TS to MP4 at the time of e-mail transmission. The operation when a packet sequence of a TS packet multiplexed with MPEG-4 AVC encoded data is converted into an MP4 file in a conventional multiplexing system conversion apparatus will be described below (see, for example, Patent Document 1). . In the converted MP4, fragments are not used. FIG. 29 is a flowchart showing the conversion operation. In step 101, the data of the PES packet is separated by detecting the start position of the PES packet from the payload of the input TS packet. Next, in step 102, the start position of the AU is detected from the payload data of the separated PES packet. Steps 101 and 102 are repeated until the start position of the AU is detected (loop A), and when detected, AU data for one AU is separated. In step 103, display time information indicated in the PES header is acquired, and in step 104, moov and mdat are created. The processing from step 101 to step 104 (loop B) is repeated until the processing of the final TS packet is completed. When all the AUs of media data are processed, the loop B is terminated, and in step 105, the data of moov and mdat To complete the creation of the MP4 file. FIG. 30 is a flowchart showing details of the processing in step 104. In step 201, AU data is analyzed, and header information necessary for creating a Box in moov is obtained. In step 202, the storage format of the NAL unit constituting the AU is converted from the byte stream format to the NAL size format. In step 203, Box data in the moov is created based on the AU header information acquired in step 201, and in step 204, mdat data is created. The Box for storing the header information for each sample, such as the sample size and decoding time information, is completed when the processing of all AUs in the track is completed.
JP 2003-114845 A

  When receiving broadcast TS packets, packet loss may occur depending on the weather and reception conditions. In particular, packet loss may occur frequently in the case of receiving while moving by a mobile terminal such as a mobile phone. By checking whether the continuity_counter value is not discontinuous in the TS packet, it can be detected that AU data is missing due to packet loss, but MP4 cannot indicate missing sample data. Therefore, when playing an MP4 file, there is a first problem that sample data that cannot be decoded normally due to missing data is decoded, and as a result, low-quality moving image data is displayed. It was.

  Furthermore, when converting the multiplexing method from TS to MP4, it is necessary to convert the storage format of the NAL unit from the byte stream format to the NAL size format. In the byte stream format, the boundary of the NAL unit is detected by detecting the start code added to the head of the NAL unit. However, when the boundary part is lost due to packet loss, the boundary cannot be detected. The data cannot be separated accurately. Conventionally, when the NAL unit data cannot be accurately separated, a method for storing the NAL unit data as an MP4 sample has not been defined. Therefore, it is not possible to determine the data to be stored in the converted NAL unit. There was a problem.

  Accordingly, the present invention has been made in view of the above circumstances, and an object of the present invention is to provide a multiplexing system conversion apparatus that can prevent moving image data that cannot be correctly decoded from being displayed to a user. And

  In order to achieve the above object, a multiplexing method conversion apparatus according to the present invention performs packet multiplexing of encoded media data packet-multiplexed by the first method by a second method different from the first method. An output multiplexing method conversion apparatus, wherein packet loss determination means for determining presence / absence of packet loss in encoded media data packet-multiplexed by the first method, and the packet loss determination means have packet loss. And determining the missing information generated by the missing information generating means based on the position where the data is missing in the encoded media data, and the missing information generated by the missing information generating means. 2 for storing the encoded media data in the packet of the second scheme and packet-multiplexing the encoded media data by the second scheme. Characterized in that it comprises a means.

  Thus, for example, in the second format file that is MP4, without decoding the sample data, for example, a sample with missing data due to the packet loss of the TS packet of the first format that is the TS of the MPEG-2 system. Since the information is obtained, for example, the decoding or display operation of the sample can be determined in a system layer higher than the audio or video decoding means.

  The multiplexing method conversion apparatus may further include a packet separation unit that separates a header and a payload from a packet of encoded media data packet-multiplexed by the first method.

  The multiplexing method conversion apparatus further includes a frame separation unit that detects a frame boundary of the encoded media data from the payload separated by the packet separation unit and separates the frame, and the missing information generation unit When the packet loss determination means determines that there is a packet loss, the multiplexing means generates missing information regarding data loss based on the position where the data in the encoded media data is lost, May store the missing information in units of frames generated by the missing information generation means in the packet of the second scheme. Here, the missing information generating means may generate, as the missing information, information indicating a section in which one or more frames in which no data essential for decoding is missing are consecutive in decoding order.

  As a result, in the file of the second method, for example, MP4, since the section in which the sample can be correctly decoded can be specified in the system layer, the reproduction quality by decoding the sample with missing data and outputting a distorted display image Can be prevented.

  In addition, the multiplexing method conversion apparatus further includes a creation unit that creates a table for registering a randomly accessible frame, and the multiplexing unit includes a frame in which the encoded media data is randomly accessible. Is registered in the table, it is determined whether or not data essential for decoding is missing in the frame based on the position where the data in the encoded media data is missing, and is essential for decoding in the frame. If data is missing, the frame need not be registered in the table as a randomly accessible frame.

  As a result, in the second method file, which is MP4, for example, it is guaranteed that the sync sample can be correctly decoded, so that it is possible to prevent the reproduction quality from being deteriorated during random access.

  The multiplexing system conversion device may further include conversion means for converting the storage format of the encoded media data. Here, the storage format is used in a narrower sense than the multiplexing method, for example, a frame data storage method.

  Further, the frame is composed of one or more subframe units, and the converting means adds the size information of the subframe to the head of the subframe unit and converts the storage format when the encoding medium is converted. The subframe data stored in the packet of the second scheme and the size of the subframe data may be determined based on the position where the data is missing in the data.

  Thereby, for example, in the file of the second method that is MP4, AU data of MPEG-4 AVC can be converted into the NAL size format even when data is lost due to packet loss.

  In addition, the frame is composed of one or more subframe units, and the converting means inserts the subframe unit indicating data loss at a position where the data is missing in the encoded media data, and then performs the encoding. Media data may be packet-multiplexed by the second method.

  As a result, for example, in the second format file, which is MP4, by detecting the NAL unit for notifying the packet loss information inserted in the sample data, the decoding means can also identify the missing position of the AU data. Appropriate error concealment processing can be performed.

  The missing information generation means may generate information indicating whether or not data essential for decoding in a predetermined unit is missing as the missing information.

  As a result, in the file of the second method, which is MP4, for example, a sample lacking data such as SPS (Sequence Parameter Set) or PPS (Picture Parameter Set) essential for decoding can be specified in the system layer. It is possible to prevent deterioration in reproduction quality caused by decoding a sample with missing data and outputting a distorted display image.

  The demultiplexing apparatus according to the present invention is a demultiplexing apparatus that decodes multiplexed data in which encoded media data is packet-multiplexed, and is a missing unit that extracts missing information related to missing data from the multiplexed data. Information extracting means, missing information analyzing means for determining data to be decoded in the encoded media data based on the missing information extracted by the missing information extracting means, and decoding to be performed by the missing information analyzing means And decoding means for decoding the data.

  As a result, for multiplexed data such as MP4, for example, the decoding and display operation of the sample can be determined with reference to the Box indicating the missing information of the sample. It is possible to prevent a reduction in reproduction quality due to the output of the display image, and it is possible to obtain the same output result even when using decoding display means having different error concealment methods.

  The demultiplexing apparatus may further include missing information notifying means for notifying a user of reproduction quality based on the missing information extracted by the missing information extracting means.

  Thus, for example, with multiplexed data that is MP4, it is possible to notify the user of the reproduction quality such as the missing rate (missing data / all data) of the sample data, for example.

  Furthermore, the present invention can be realized not only as such a multiplexing method conversion device and a demultiplexing device, but also by using characteristic means included in such a multiplexing method conversion device and a demultiplexing device as steps. It can also be realized as a multiplexing method conversion method and a demultiplexing method including, or as a program for causing a computer to execute these steps. Needless to say, such a program can be distributed via a recording medium such as a CD-ROM or a transmission medium such as the Internet.

  As is clear from the above description, according to the multiplexing method conversion apparatus of the present invention, it is possible to show a reduction in reproduction quality by showing moving image data that cannot be correctly decoded.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(Embodiment 1)
First, a multiplexing scheme conversion apparatus according to Embodiment 1 of the present invention will be described with reference to FIGS. Here, it is assumed that a TS packet multiplexed with MPEG-4 AVC encoded video data is input, AU data is separated from the TS packet, multiplexed into MP4, and output. The encoded data to be packet-multiplexed is not limited to MPEG-4 AVC, but MPEG-4 Visual or H.264. It may be H.263, video encoded data such as VC-1 being standardized by SMPTE (Society of Motion Picture and Television Engineers), or audio encoded data such as MPEG-4 AAC or AMR. Furthermore, the input TS packet may include not only a TS packet for transmitting encoded video data but also audio or program information. In this case, first, a TS packet including program information is acquired, and the PID of the TS packet for transmitting video and audio encoded data is determined. Then, based on the value of PID, video and audio TS packets are sorted and processed.

  FIG. 1 is a diagram showing an overall configuration of a multiplexing method conversion apparatus. This apparatus inputs TS packet data and stores it in the memory 102, and also outputs a data I / O unit 101 that outputs the MP4 file after the multiplexing method conversion, acquires TS packet data from the memory 102, and acquires AU data, And the PES packet header are separated and analyzed, the analysis result is stored in the memory 102, the separated AU data is converted into MP4 using the analysis result, and the conversion processing result is output to the memory 102 It consists of. Here, the memory 102 has a storage area for input / output data and a work area for temporarily storing a conversion processing result and AU data, and the size of each area can be made variable. In addition, the reconstruction unit 103 is a CPU here, but may be dedicated hardware. Hereinafter, the operation of the conversion unit 103 will be described in detail.

  FIG. 2 is a block diagram illustrating a configuration of the conversion unit 103. The conversion unit 103 includes a TS payload separation unit 201, a packet loss determination unit 202, a PES separation analysis unit 203, an AU separation unit 204, a PES header analysis unit 205, a Box creation unit 206, and a connection unit 207. The TS payload separation unit 201 separates the header and payload of the TS packet and outputs them to the packet loss determination unit 202. The packet loss determination unit 202 determines the packet loss of the TS packet. When the packet loss is detected, the packet loss determination unit 202 performs error processing and outputs the payload data after error processing to the PES separation analysis unit. When determining the packet loss, the continuity_counter value is acquired from the TS header and compared with the continuity_counter value of the TS packet input immediately before. When the value is discontinuous, it is determined that the packet has been lost. Here, since the maximum value of continuity_counter is 15, when the continuity_counter value of the immediately preceding TS packet is 15 and the continuity_counter value of the current TS packet is 0, it is considered that the continuity_counter has circulated and is not determined as a packet loss. . When a packet loss is detected, the NAL unit type of 25 and the payload is empty as information for packet loss notification is inserted at the beginning of the payload data of the current TS packet as a byte stream format. . Note that the NAL unit type of the NAL unit to be inserted can take any value between 24 and 31 unless the NAL unit type is used for other purposes. Here, 0 and NAL unit types of 24 to 31 are defined as areas that can be used arbitrarily by the application in the AVC standard. As the packet loss notification information, a code pattern that does not appear in the AVC stream may be inserted, or packet loss information may be managed separately from the stream. Next, the PES separation analysis unit searches for the start code (0x000001E0 or more and 0x000001EF or less) of the PES packet in MPEG-4 AVC from the payload data of the TS packet in which the NAL unit for packet loss notification is inserted. To detect the start position of the PES packet. Here, when a field indicating the size of the PES packet exists in the header of the PES packet, the payload may be separated using the size. After the PES packet is separated, the header and the payload of the PES packet are separated, the header data is output to the PES header analysis unit 205, and the payload data is output to the AU separation unit 204. Note that the stuffing data in the PES packet is not included in the payload data. The AU separation unit 204 detects the AU boundary from the payload data of the PES packet, separates the AU data, and outputs the separated AU data to the Box creation unit 206. Here, when the packet loss notification NAL unit is inserted by the packet loss determination unit, the AU data output to the Box creation unit 206 also includes the packet loss notification NAL unit. The PES header analysis unit 205 determines whether the input PES header data includes the display time of the first AU starting in the PES payload. If included, the PES header analysis unit 205 sends the AU display time to the Box creation unit 206. Output. Next, the Box creation unit 206 creates moov and mdat including packet loss information based on the input AU data and the display time in the PES header, and outputs the moov and mdat to the connection unit 207. The Box creation unit 206 may create moof and mfra in addition to moov and mdat. Finally, the connecting unit 207 connects moov and mdat and outputs the data as MP4 file data.

  FIG. 3 is a flowchart showing the operation of the conversion unit 103. First, in step 301, it is determined whether or not a TS packet loss has occurred. When the occurrence of a packet loss is detected, a packet loss notification NAL unit is inserted at the packet loss detection position. Subsequently, in step 302, the header and the payload of the PES packet are separated, and in step 303, the AU boundary is detected from the payload of the separated PES packet, and the AU data is separated. As shown in loop A of FIG. 3, the operations from step 301 to step 303 are repeated until 1 AU worth of data can be separated. After separating the data for one AU, in step 304, the header of the PES packet is analyzed to determine whether or not the display time of the head AU starting in the PES payload is included. To get. Note that the processing in step 304 may be performed by processing in loop A. Next, in step 305, Box data in the MP4 file is created based on the AU data separated in loop A and the display time information acquired in step 304. Loop B is repeated until processing of all AU data included in the input TS packet is completed, and when processing of all AU data is completed, moov and mdat are completed. And mdat are connected to output an MP4 file. Here, the loop B includes the processing of the loop A, step 304, and step 305. Note that ftyp is created in step 305 and added to moov for output, or added to moov in step 306. An MP4 file using fragments may be output. In this case, when the moof and mdat for one fragment are determined in step 305, the moof and mdat are output and the loop B is exited. In step 306, the output moof and mdat are sequentially connected. Furthermore, when using a fragment, the header information of all the samples of the media track may be stored in moof, and the header information of the sample at the beginning of the track may be stored in moov.

  The configuration and operation of the Box creation unit 205 will be described in detail. In the following description, when the term “NAL unit” is used, it refers to the NAL unit included in the payload of the TS packet, and does not include the NAL unit for packet loss notification. FIG. 4 is a block diagram showing the configuration of the Box creation unit 205. The Box creation unit 205 includes an AU analysis unit 301, an error information generation unit 302, an AU format conversion unit 303, and a Box data determination unit 304. First, the AU analysis unit 301 analyzes AU data, obtains information necessary for creating a Box that constitutes a conventional MP4 file, such as whether it is an AU display time or an AU IDR, and analyzes the result. The data is output to the Box data determination unit 304. Here, the AU size is not included in the analysis result output by the AU analysis unit 301. In addition, the Box constituting the conventional MP4 file does not include the Box indicating the error information (missing information) or the identification information for indicating the error information in mdat.

  In the following, an explanation will be given of the operation when SPS or PPS data to be referred to when decoding AU time information or AU data is missing due to packet loss. For AUs whose decoding time or display time cannot be acquired from the header information of the PES packet, the time information is acquired by analyzing the data in the AU. Specifically, a parameter indicating the display order of pictures called Picture Order Count, or SEI such as Picture Timing SEI and Buffering Period SEI is analyzed. When information indicating such time information is missing due to packet loss, the AU decoding time or display time is calculated by a predetermined method. When the information of a descriptor transmitted by a TS packet having a PID different from that of SPS or encoded media data indicates that the frame rate is fixed, the decoding time or display based on the frame rate Time may be determined. A descriptor is auxiliary information of encoded media data defined in the MPEG-2 system standard. Next, in MP4, SPS and PPS are stored in a sample entry in a sample description box ('stsd') or a parameter set track. Furthermore, the sample in the track of the encoded data of MPEG-4 AVC and the sample entry storing the SPS and PPS to which the sample refers, or the sample in the parameter set track are associated one-to-one. Therefore, when SPS or PPS data to be referred to when decoding a sample is lost, the same SPS or PPS as that of the immediately preceding sample is referred to. A preset default SPS or PPS may be used.

  The error information generation unit 302 searches for packet loss notification NAL units from the AU data, and when detecting a packet loss notification NAL unit, creates error box data indicating error information. Further, the NAL unit data to be stored in mdat is determined based on the presence or absence of packet loss in the data of the NAL units constituting the AU, and the size of the AU data configured from the NAL unit data determined to be stored as sample data in mdat Is output to the Box data determination unit 304 together with Box data indicating error information. In the AU format conversion unit 303, the NAL unit data in the AU determined to be stored as sample data in the error information generation unit is converted from the byte stream format to the NAL size format, and the converted AU data is converted into the Box determination unit 304. Output to. Here, the converted AU data output to the Box determination unit 304 becomes sample data in MP4.

  FIG. 5 is a flowchart showing the operation of the Box creation unit 205. First, in step 401, information necessary for creating a Box constituting a conventional MP4 file is acquired from the input AU data. In step 402, the NAL unit for packet loss notification is detected from the AU data to create box data indicating error information. In step 403, NAL unit data to be stored as sample data in the MP4 file is determined. To do. In step 404, the NAL unit data determined to be stored as sample data in step 403 is converted into a NAL size format, and sample data composed of the converted NAL unit data is created. In step 405, moov data including a box indicating error information created in step 402 is created. In step 406, mdat data for storing the sample data created in step 404 is created. FIG. 6 is a flowchart showing the operation of step 405. In step 404, first, in step 501, a box constituting a conventional MP4 file is created, and in step 502, a box for storing error information is created.

  Next, a procedure for creating Box data indicating error information in Step 402 will be described. FIG. 7 is a flowchart showing the operation of step 402. Here, the term “sample” is used to describe the Box data creation method in the MP4 file, but one sample is the same as one AU. First, in step 601, it is searched whether the AU data includes a packet loss notification NAL unit. At the time of search, the start position of the NAL unit is detected by checking the start code prefix (3 bytes consisting of 0x000001) of the byte stream format at the byte alignment position in order from the head data of the AU. Subsequently, the NAL unit type of the detected NAL unit is acquired, and if the value of the NAL unit type is a predetermined value, the NAL unit for packet loss notification is determined. After checking the NAL unit type up to the last NAL unit of the AU, it is determined in step 602 whether a packet loss notification NAL unit is detected in the sample. When the packet loss notification NAL unit is detected, the processing of step 603 is performed. When the packet loss notification NAL unit is not detected, the box data creation processing is terminated. In step 603, the sample number of the sample is added to a table showing a list of samples including errors. Hereinafter, the operation from step 601 to step 603 will be described with reference to FIG. FIG. 8A shows sample data in AVC encoded data arranged in decoding order, and the numbers in the figure indicate sample numbers. In this example, it is assumed that NAL units for packet loss notification are detected in the NAL unit data in the samples in the samples whose sample numbers are 7, 9, and 15. The procedure for creating a box indicating error information for samples with sample numbers 1 to 15 is as follows. Note that the name of the box indicating error information is Error Sample Box. First, since the NAL unit for packet loss notification is not detected in step 601 for the first to sixth samples, it is determined in step 602 that there is no packet loss notification NAL unit in the sample, and Error Sample Box The process is terminated without updating the table data. For the seventh sample, since a NAL unit for packet loss notification is detected in step 601, it is determined in step 602 that the table data in the Error Sample Box is updated, and an update process is performed in step 603. In step 603, in order to indicate that packet loss is included in the seventh sample, an entry in the table is added and sample number 7 is stored. Similar processing is performed for subsequent samples, and sample numbers 9 and 15 are added to the table. As a result, as shown in FIG. 8B, 7, 9, 15 are stored in the Error Sample Box table as sample numbers including packet loss. FIG. 8C shows an example of syntax of the Error Sample Box, where error_sample_number indicates the sample number of the sample in which the NAL unit data in the sample is missing due to packet loss. The error sample box is placed directly under stbl in trak in moov, and is placed directly under track fragment run box ('traf') in moof, but it can also be placed directly under other boxes. Good. Here, the Error Sample Box is created only when there is a sample with missing data in the media track, but even when there is no sample with missing data, the Error Sample Box whose entry is empty May be created. Note that when placing in the traf, error_sample_number indicates the number when the samples are counted in order, with the first sample number in the traf being 1. For example, when 10 samples are included in traf and the data of the third and fifth samples are missing due to packet loss, two entries with error_sample_number of 3 and 5 are created. To do. Alternatively, in the track fragment run box ('trun'), a flag indicating that data in the sample is missing due to packet loss is added to tr_flags, which is a field indicating flag information about the header information of each sample. It is good as well.

  Furthermore, in the above description, it is indicated that the data in the sample is missing due to the packet loss. However, it may be indicated whether the data is missing in the specific NAL unit in the sample. For example, in MPEG-4 AVC, it indicates whether or not data is missing in the data of slices or SPS or PPS NAL units. This is because a correct decoding result can be obtained if the slice and the parameter set referred to by the slice are prepared. More specifically, the NAL unit of the parameter set may indicate whether SPS or PPS data necessary for decoding the slice data in the sample is missing. Further, information for specifying the NAL unit in which data is missing may be stored as a syntax as shown in FIG. In the figure, error_nal_unit_number is a number for identifying the NAL unit in which data is missing. For example, when the second and fourth NAL units in the sample are lost, error_nal_unit_number is 2 and 4 Two entries are created. The NAL unit type may be indicated in the entry. Further, error_nal_unit_number is a video packet in MPEG-4 Visual or H.264. The slice in H.263 may be specified.

  In addition to packet loss, when information about bit errors can be acquired, samples that contain bit errors in sample data, or information for specifying NAL units may be stored.

  Next, in steps 403 and 404, NAL unit data to be stored as sample data in the MP4 file is determined, and the operation when converting the storage format of the NAL unit to the NAL size format is described with reference to the flowchart of FIG. To explain. First, in step 701, valid data is determined and separated in each NAL unit in the AU. Here, when a NAL unit for packet loss notification is detected, the data valid in the NAL unit for which the start position of the NAL unit was detected immediately before is the data up to immediately before the NAL unit for packet loss notification. To do. After the NAL unit for packet loss notification, the data until the next start position of the NAL unit is detected becomes invalid. This is because the NAL unit to which the invalid part data belongs may not be specified, or This is because even if it can be specified, the missing data size is unknown and cannot be used at the time of decoding. If no NAL unit for packet loss notification is detected, all NAL unit data is regarded as valid data. The NAL unit data separated as valid data does not include zero_byte and the start code prefix. Next, in step 702, the size of valid data separated in step 701 is acquired. In step 703, NAL unit data including the valid data is converted into a NAL size format based on the size acquired in step 703. Here, in the NAL size format, the size of the field indicating the size of the NAL unit is variable between 1 and 4 bytes, but the field size is given in advance. Next, in step 704, it is determined whether or not the data of the NAL unit is missing. If it is missing, the NAL unit for error notification is inserted as a NAL size format. The processing from step 701 to step 705 (loop A) is repeated until the processing of the final NAL unit of the sample is completed. Note that the size of the sample is determined when the size of the final NAL unit of the sample is determined.

  Whether or not sample data is missing is indicated only in moov or a box indicating error information stored in moof, and sample data stored in mdat includes a NAL unit for packet loss notification. May not be inserted. At this time, step 704 and step 705 are unnecessary.

  Further, all NAL unit data included in the payload of the TS packet may be stored without discarding the invalidated data. Further, when the size of data lost due to packet loss can be specified, a special NAL unit (hereinafter referred to as a NAL unit for packet loss compensation) for stuffing the lost part may be inserted. In the NAL unit for complementing packet loss, a NAL unit type that is an Unspecified area and is different from that for packet loss notification is used. Also, error information may be indicated only in the NAL unit for error notification, and the Error Sample Box may not be created.

  In addition, when packet loss information in the NAL unit data in the AU is provided separately instead of the packet loss notification NAL unit, in step 701, valid data in the NAL unit is obtained based on the separately indicated information. decide.

  FIGS. 10 to 12 show specific examples when valid NAL unit data is determined and converted to the NAL size format. In each figure, SCP, Nal_len, and ERR_NALU indicate the start code prefix in the byte stream format, the field size indicating the size of the NAL unit in the NAL size format, and the NAL unit for error notification, respectively. FIG. 10 shows an example in which the loss of NAL unit data due to packet loss does not cross the boundary of NAL units. FIG. 10A shows the AU data before conversion. The AU is composed of four NAL units from NALU # 1 to NALU # 4, and packet loss occurs in the third NAL unit. NALU # 3 is originally composed of data in the areas indicated as reception 1, lost, and reception 2 in the figure, but since the lost area is missing due to packet loss, immediately after the data in the reception 1 part. ERR_NALU has been inserted. At this time, only the data of the reception 1 portion is determined as valid data in NALU # 3, and therefore, only the data of the reception 1 portion is stored as the mdat sample data. Since the data size of the received 1 part is size3_1, when the data of NALU # 3 is converted to the NAL size format, the Nal_len field value is size3_1. Next, immediately after NALU # 3, ERR_NALU in NAL size format is inserted. FIG. 10B shows the structure of AU data output after conversion. Note that the data structure of the AU when storing all the NAL unit data included in the payload of the TS packet without inserting the NAL unit for error notification is as shown in FIG. The size of 3 is the sum of size3_1 and size3_2 (the size of the reception 2 part).

  FIG. 11 shows an example in which the loss of NAL unit data due to packet loss crosses the boundary of NAL units. FIG. 11A shows AU data before conversion. The AU is composed of four NAL units from NALU # 1 to NALU # 4, and data is missing across the NAL units from the terminal part of NALU # 2 to the head part of NALU # 3 (in the figure). And the data in the area shown as lost). At this time, what is valid in NALU # 2 is the data of the reception 1 portion in the figure. Next, since the NAL unit start position could not be detected for NALU # 3, the data immediately after ERROR_NALU to the start position of NAUL # 4 becomes invalid. As a result, the data of NALU # 3 becomes sample data. Is not stored. That is, the data of the reception 2 part in the figure becomes invalid. FIG. 11B shows the structure of AU data output after conversion.

  FIG. 12 shows an example in which the loss of NAL unit data due to packet loss crosses the AU boundary. FIG. 12A shows AU data before conversion. The three AU data from AU # 1 to AU # 3 are composed of NALU # 1 to NALU # 4, NALU # 5 to NALU # 8, and NALU # 9 to NALU # 12, respectively. The data is missing from AU # 2 from the middle to the beginning of NALU # 9 (data in the area shown as lost in the figure). Valid data in AU # 1 to AU # 3 is as follows. In AU # 1, all the data from NALU # 1 to NALU # 3 and the data for nal4_size acquired in NALU # 4 are valid. Since all AU # 2 data is lost, there is no valid data. Therefore, in the MP4 file, the sample for storing AU # 3 is stored after the sample for storing AU # 1. At this time, the sum of the reproduction time lengths of AU # 1 and AU # 2 is stored as the reproduction time length of the sample storing AU # 1. Finally, in AU # 3, data from NALU # 10 to NALU # 12 is valid. Here, when it is possible to specify that NALU # 10 is not the head NAL unit in AU # 3, ERR_NALU may be inserted at the head of NALU # 10. For example, since the MPEG-2 system standard stipulates that an AU boundary determination NAL unit called an Access Unit Delimiter is inserted at the beginning of an AU, NALU # 10 must be the NAL unit of the Access Unit Delimiter. For example, it can be determined that the head NAL unit of AU # 3 has been lost. FIG. 12B shows the structure of AU data output after conversion. FIG. 12C shows an example of the structure of AU data output when using a NAL unit for packet loss compensation. The output data of AU # 1 and AU # 3 are the same as in FIG. 12B, but in this example, the data of AU # 2 is also output. The AU # 2 data includes only a packet loss compensation NAL unit, and whether the packet loss compensation NAL unit size is the encoding bit rate of MPEG-4 AVC data and a fixed frame rate. It can be estimated based on the information of whether or not. In this way, by inserting a NAL unit for packet loss compensation, when video data is encoded at a fixed bit rate, sample data in MP4 can be obtained even when data is lost due to packet loss. It is possible to store the original fixed bit rate as much as possible. In addition, when the stream has a fixed frame rate, the sample playback time length is made constant by creating a sample composed of NAL units for packet loss compensation. As a result, the sample playback time length is stored. The table size to be reduced can be reduced.

  In addition, when AU slice data is missing, the same data as the previous AU is stored in decoding order or display order, or error concealment processing is performed in units of slices, etc. It may be configured and recorded. Further, information indicating that the AU data is reconfigured may be separately indicated.

  In addition, you may store the information regarding a packet loss with a SEI message (Supplemental Enhancement Information). For example, an SEI message for storing user data can be used.

  It should be noted that error information such as the ratio of data lost due to packet loss, the ratio of slices in which data is lost among the slices in the AU, or the frequency of occurrence of bit errors, the NAL unit payload for error notification, and Error Sample You may store in Box. In addition, when describing error information in the NAL unit type and size of the NAL unit for error notification or in the payload of the NAL unit for error notification, a box indicating the type of error information to be described is included in moov or moof. May be stored.

  Each of the above operations can also be applied when a plurality of AUs are stored as one sample by using the frme_count field in the sample entry in the sample description box (stsd).

  Further, a method for creating a box for error notification or inserting a NAL unit for error notification into sample data includes encoding media data multiplexed by an RTP (Real-time Transmission Protocol) packet into an MP4 file. It can also be applied when converting. In the RTP packet, the packet loss can be determined by examining the continuity of the sequence numbers added to the header of the RTP packet.

  In this embodiment, error information such as Error Sample Box is added to each track, but the present invention is not limited to this. For example, error information may be added in units of MP4 files. In this case, the error information can be arranged directly under moov. Further, when using a fragment, it may be arranged in the moof of the final fragment or a box in mfra. As a result, the user is notified of information such as the missing rate of sample data (missing data / total data) and the ratio of the time length during which both audio and video can be played out of the total playback time length as the entire MP4 file during playback. It becomes possible.

  In addition, for example, in a playlist in which a plurality of items for which a predetermined playback section is specified is provided in a list format and the items are played back in order, error information may be added for each item. This playlist is a list in which, for example, audio data is used as an item, and a song selected by the user is described in the order of selection. Here, each playback section may be a different section in the same file, a different file, or a specific section of a different file. In this case, at the time of reproduction, it is possible to acquire information such as the missing rate of sample data for each item simply by referring to the playlist.

(Embodiment 2)
A multiplexing method conversion apparatus according to Embodiment 2 of the present invention will be described with reference to FIG. 13 and FIG. The configuration of the multiplexing system conversion apparatus is the same as that of the multiplexing system conversion apparatus according to the first embodiment, and only the operation at the time of creating stss that is a table of sync samples in the Box data determination unit 304 is different. Only the stss creation method will be described.

  In the multiplexing system conversion apparatus of the first embodiment, IDR AU is used as a sync sample regardless of whether data in the sample is missing. However, since the sync sample becomes a sample to start decoding at random access, if the sync sample cannot be decoded normally, a normal decoding result may not be obtained for the subsequent samples in the sequence. was there. Therefore, in the multiplexing system conversion apparatus according to the present embodiment, only the IDR AU in which no data is missing is used as a sync sample, so that decoding is not disturbed at least in the sync sample when decoding starts from the sync sample. The result can be obtained.

  FIG. 13 is a flowchart showing an operation when creating stss in the Box data determination unit 304. First, in step 801, it is determined whether the NAL unit of the IDR slice is included in the sample data. Here, whether or not the NAL unit is an IDR slice is identified by the NAL unit type. If the NAL unit of the IDR slice is not included, the process ends. If included, the process of step 802 is performed. In step 802, it is determined based on the error information for each sample acquired by the error information generation unit 302 whether the sample data includes a NAL unit whose data is lost due to packet loss. When it is determined that the sample data is missing, the process is terminated. When it is determined that the sample data is not missing, in step 803, the sample data is displayed in the stss table indicating a list of sync samples. Add a sample number. As described above, stss is created by repeating the processing from step 801 to step 803 up to the final sample in the track. It should be noted that the sync sample is determined so that the difference value of the reproduction start time between sync samples is as constant as possible without including all the samples including the NAL unit of the IDR slice and lacking data as sync samples. Also good. In addition, in the AU including the IDR slice, if no data is missing in the slice or the SPS or PPS NAL unit, it may be a sync sample. More specifically, the SPS or PPS data necessary for decoding the slice data in the sample may be set as the sync sample only when the sample or the decoding time can be acquired from the previous sample.

  FIG. 14 shows a specific example when stss is created by the operation shown in the flowchart of FIG. FIG. 14A shows samples arranged in the order of sample numbers. Samples with sample numbers 1, 11, 21, and 31 include NAL units of IDR slices. Data in the data is missing. At this time, in step 801, the first, eleventh, twenty-first, and thirty-first samples are set as sync sample candidates. However, in step 802, the twenty-first sample is determined not to be a sync sample. In step 803, the first, eleventh, and thirty-first samples are added to the stss table (FIG. 14B).

A randomly accessible sample registered in mfra can be determined in the same manner.
Note that stss is created by the above method and creating an Error Sample Box can be selected independently. For example, stss is created by the above method, and the Error Sample Box does not have to be created.

(Embodiment 3)
A multiplexing scheme conversion apparatus according to Embodiment 3 of the present invention will be described with reference to FIGS. The configuration of the multiplexing system conversion apparatus is the same as that of the multiplexing system conversion apparatus according to Embodiment 1, and the error information generation unit 302 creates a box indicating the error information of the sample by the operation of step 402 in FIG. Since only the operation at this time is different, the operation of this part will be described. In this multiplexing system conversion apparatus, when decoding is started from a sync sample, a box indicating whether or not up to what number of samples after the sync sample can be correctly decoded is created. When playing back an MP4 file created by the multiplexing system conversion apparatus, by referring to this Box, it is possible to determine a sample to be displayed after identifying a sample that is guaranteed to be correctly decoded.

  FIG. 15 is a flowchart showing an operation when creating a box indicating error information. err_free_flag is a flag indicating that no data is missing in the samples from the sample at which the sequence starts to the sample. The initial value is 0. In Step 901, it is determined whether or not the NAL unit of the IDR slice is included in the sample. When included, err_free_flag is set to 1 in Step 902. In step 903, it is determined whether or not err_free_flag is 1. When it is 1, the processing of step 904 is performed, and when it is 0, the subsequent processing steps are skipped and the processing of the next sample is performed. In step 904, it is determined whether data of the NAL unit constituting the sample is missing. If no slice or SPS / PPS NAL unit data is missing, it may be determined in step 904 that no data is missing. If there is a NAL unit with missing data, in step 905, the sample number of the immediately preceding sample is added to the table showing the list of the last samples in the error-free section. Here, the error-free section is a section in which it is guaranteed that no data is lost in the samples of the section arranged in the decoding order. In random access, decoding is started from sync samples, and since the sync sample is a sample including the NAL unit of the IDR slice, the start sample in the error-free section is a sync sample. Note that, when data is missing in a sample including an NAL unit of an IDR slice, an error free section for the sample is not defined. Therefore, step 905 is skipped. As described above, the processing from step 901 to step 905 is repeated until the processing of the final sample of the track is completed. Here, guaranteeing that no data loss will occur means that no error was detected from the continuity_counter value of the TS packet, and that no data loss has occurred at all. It is not guaranteed. When packet loss can be detected more accurately by packet loss information other than continuity_counter, an error may be detected using the information.

  If an error free section can be indicated, information other than the final sample may be used, such as indicating the number of samples included in the error free section.

  FIG. 16 shows a specific example when a box for storing error information is created by the operation shown in the flowchart of FIG. FIG. 16 (a) shows two consecutive sequences each consisting of 10 samples. Sequence # 1 starts with the first sample and the 7th and 9th sample data is missing, and sequence # 2 starts with the 11th sample and the 16th sample data is missing doing. The procedure for determining the error-free section will be described below. First, err_free_flag is set to 1 in the first sample. Up to the sixth sample, since it is determined in step 904 that sample data is not missing, the processing in step 905 and step 906 is skipped, and the value of err_free_flag is held at 1. Next, since it is determined in step 904 that the sample data is missing in the seventh sample, in step 905, an entry is added to the table storing the final sample in the error-free section, and the previous sample is added. A sample number indicating the sixth sample which is a sample is stored. Until the 10th sample, the value of err_free_flag remains zero. Subsequently, in the eleventh sample, err_free_flag is set to 1 again, and the 15th sample is added to the table as the final sample of the error-free section in the same manner as in sequence # 1. FIG. 16B shows a sync sample table. FIG. 16B shows a table of the last sample in the error free section. From the two tables of FIG. 16B and FIG. 16C, the error free section in sequence # 1 is a sample with sample numbers 1 to 6, and the error free section in sequence # 2 is a sample number. Are 11 to 15 samples. FIG. 16D is a syntax example of an Error Free End Sample Box that is a box indicating a list of final samples in the error-free section, and end_sample_number indicates the sample number of the last sample in the error-free section. The Error Free End Sample Box is placed directly under stbl in trak in moov, and placed directly under the track fragment run box ('traf') in moof, but placed directly under other boxes. Also good.

  In step 901, it is determined whether or not the sample includes the NAL unit of the IDR slice, and err_free_flag is set. However, when the sample including the NAL unit of the IDR slice is not necessarily the sync sample Cannot specify an error-free section using the list of sync samples shown in stss and the Error Free End Sample Box. Therefore, in step 901, it is determined whether or not the sample is a sync sample. If the sample is a sync sample, err_free_flag may be set. Moreover, it is good also as showing the head sample instead of the last sample of an error free area. For example, in FIG. 16A, when the data of the 6th to 10th samples of sequence # 1 are completely prepared, the 6th sample is obtained by setting the top sample of the error free section to 6. To the 11th sample which is the first sample of the sequence 2 can be shown to be an error free section.

  In MP4, a Gradual Decoder Refresh point can be designated in addition to a sync sample as a randomly accessible sample. The Gradual Decoder Refresh point refers to a sample in which a correct decoding result cannot be obtained for the sample, but a correct decoding result can be obtained by decoding a specified number of subsequent samples in the decoding order. Alternatively, in order to obtain a correct decoding result in a certain sample, it is possible to indicate a decoding start sample when it is necessary to start decoding from the number of previous samples specified in the decoding order. For example, when decoding starts from the Mth sample in decoding order, the correct decoding result is obtained from the Nth (N> M) sample, and in the MP4 file, the difference between N and M Information indicating the value is shown. Therefore, a sample corresponding to the Gradual Decoder Refresh point may be used as the first sample in the error-free section.

  In addition, it is good also as showing both the head sample and the last sample of an error free area so that an error free area can be specified without making it match | combine with a sync sample.

  In the above description, a list of the last samples in the error-free section is shown. Hereinafter, an example will be described with reference to FIG. FIG. 17 (a) shows six consecutive sequences each consisting of ten samples. Here, for the four sequences of sequence # 1, sequence # 3, sequence # 4, and sequence # 6, all the sample data in the sequence is completely aligned. In sequence # 2 and sequence # 5, Suppose that samples with missing data are included. FIG. 17 (c) is a list of head samples in a sequence in which all sample data in the sequence are completely prepared. FIG. 17B is a list of sync samples when the first sample of each sequence is a sync sample. By referring to the two tables in FIG. 17B and FIG. A sequence in which data loss occurs is a sequence that starts from sync samples with sample numbers 11 and 41, and a sequence that does not occur starts from sync samples with sample numbers 1, 21, 31, and 51 It turns out that it is a sequence. In addition, when the sample including the NAL unit of the IDR slice is not necessarily a sync sample, it may indicate whether or not the sample data between two consecutive sync samples is missing. FIG. 17D shows a syntax example of the Error Free Sequence Box that shows a list of the top samples of sequences in which no data is missing in the sequence. The sequence_start_sample_number in the syntax indicates the sample number of the first sample in the sequence. In addition to the first sample number of the sequence, an index such as Good, Medium, or Bad may be indicated as a parameter indicating the quality of the reproduced image when the sequence is reproduced according to the ratio of missing data in the sequence. Good. For example, the quality can be determined according to the ratio of missing data, the frequency of intra-coded AUs, the ratio of intra macroblocks, and the like. Furthermore, information for indicating samples that can be correctly decoded in the sequence may be stored. Specifically, when the sample is decoded, the reference relationship between the samples and the slice data, SPS, or PPS NAL unit in each sample indicate whether or not the data is missing. Since it is known whether or not data is missing, it can be determined whether or not the sample can be correctly decoded. The error free sequence box is placed directly under stbl in trak in moov, and is placed directly under track fragment run box ('traf') in moof, but is placed directly under other boxes. May be.

  In addition, the Error Sample Box, Error Free End Sample Box, and Error Free Sequence Box described in Embodiments 1 to 3 may be used in combination. For example, by combining the Error Sample Box and the Error Free End Sample Box, it is possible to identify which sample data is missing in the samples not included in the error free section and determine the sample to be decoded.

  In addition, the playback quality of a track may be determined based on the percentage of samples that can be correctly decoded or the data is missing, and a box storing information indicating the playback quality may be placed directly under the trak, or may be audio or video. Alternatively, a Box indicating the reproduction quality when the text data track is reproduced at the same time may be arranged immediately below the moov.

  In each of the above embodiments, the part for converting from the byte stream format to the NAL size format and the process of adding information indicating the lack of data may be performed separately. For example, data missing information can be added to NAL unit data transmitted in a format used for moving image distribution over the Internet, such as a NAL size format or RTP (Real Time Transmission Protocol).

  The AU format converter 303 can also be used when multiplexing MPEG-4 AVC encoded data. For example, in order to simplify the encoding process of MPEG-4 AVC, the format of the encoded data can be unified to the byte stream format. At this time, the encoded data output from the encoding unit is TS, MP4, etc. The processing of the AU format conversion unit 303 can be applied when multiplexing is performed using multiplexing systems having different encoded data storage formats. Here, since there is no error in the stream at the time of encoding, the process of compensating for data loss can be omitted. In the AU format conversion unit 303, the byte stream format is converted to the NAL size format. However, when the output at the time of encoding is unified to the NAL size format, the NAL size format may be converted to the byte stream format. Good.

(Embodiment 4)
A demultiplexing apparatus according to Embodiment 4 of the present invention will be described with reference to FIGS. This demultiplexer inputs the MP4 file created by the multiplexing method converter according to the first to third embodiments, separates the AU data, decodes and displays it. The decoding or display operation is determined by referring to the error information shown in the Error Sample Box or the like. For this reason, when there are many samples that cannot be decoded correctly in the sequence, operations such as skipping to the next sequence can be determined from the information of the file format level. It is possible to prevent a reduction in reproduction quality due to display. Also, by determining the decoding and display operations based on the file format level information, the same output result can be obtained even when a decoding unit or a display unit with different error concealment processing is used. In the MP4 file input to the demultiplexer, it is assumed that a box indicating error information (missing information) is always added when data missing in a sample is indicated. In the following, the operation when reproducing an MPEG-4 AVC track will be described. However, the reproduced track is not limited to MPEG-4 AVC. It may be video encoded data such as H.263 or MPEG-4 Visual, or audio encoded data such as MPEG-4 AAC or AMR, or a video and audio track may be reproduced simultaneously.

  FIG. 18 is a block diagram illustrating a configuration of the demultiplexer. This apparatus is composed of a header separation unit 401, a header memory 402, an mdat memory 403, a sample acquisition unit 404, an error information analysis unit 405, and a decoding / playback unit 406, and plays back using a playback start time input from the outside as a trigger. Start processing. When starting from the beginning of the file, the playback start time is 0 seconds, and when starting from the position 10 seconds from the beginning, it is 10 seconds. However, when decoding / playback is started from a sync sample, there may not be a sync sample having a playback start time that matches the input playback start time. In this case, there is no time before the input playback start time. Alternatively, decoding / reproduction is started from a sync sample whose reproduction start time is closest later. Note that playback may be started from a sample indicated as a Gradual Decoder Refresh point instead of a sync sample.

  The header separation unit 401 separates the header part including moov and moof from the MP4 file and the data part composed of mdat, outputs the header part data to the header memory 402, and outputs the mdat data to the mdat memory 403. . The sample acquisition unit 404 determines a sync sample to start decoding based on the playback start time input from the outside. For the subsequent samples, the sample number of the sync sample is output to the error information analysis unit 405 when the header information about the sync sample is acquired. In the error information analysis unit 405, error information in the sequence starting from the sync sample number input from the sample acquisition unit 404 is acquired from the Error Sample Box, Error Free End Sample Box, or Error Free Sequence Box, analyzed, After determining the samples to be decoded in the sequence, a list of samples determined to be decoded is output to the sample acquisition unit 404. Further, the error information analysis unit 405 outputs the acquired missing information to the missing information notification unit 406c of the decoding display unit 406. Further, when missing information for each NAL unit of a slice can be acquired, information indicating a slice to be decoded in a sample may be output. Note that the samples to be decoded may be determined for each predetermined number of samples without being determined for each sync sample. The sample acquisition unit 404 analyzes moov or moof, acquires the data of the sample determined to be decoded by the error information analysis unit, and the display time, and outputs them to the decoding display unit 406. Here, moov or moof is acquired from the header memory 402, and sample data is acquired from the mdat memory 403. When the decoding time and the display time are different, both may be output to the decoding display unit 406.

  Finally, the decoding display unit 406 includes a decoding unit 406a, a sample display unit 406b, and a missing information notification unit 406c. The decoding unit 406a decodes the sample data. The sample display unit 406b displays the decoded sample data according to the display time. The missing information notification unit 406c notifies the user of information such as the missing rate (missing data / all data) of the sample data based on the missing information, and inquires whether or not to play the MP4 file. The missing information notification unit 406c outputs a reproduction instruction signal to the sample acquisition unit 404 when receiving a reproduction permission signal when the user permits reproduction based on information such as the missing rate of sample data. The error information analysis unit 405 outputs the error information such as the missing data or the error free section information to the decoding display unit 406 without determining the sample to be decoded. The sample or slice data to be decoded may be determined based on the input error information or the information indicated by the error notification NAL unit inserted in the sample data. At this time, the list of decoded samples is not output to the sample acquisition unit 404.

  Next, an operation when playing back an MP4 file in the demultiplexer configured as described above will be described. FIG. 19 is a flowchart showing an operation when reproducing an MP4 file.

  The missing information notification unit 406c notifies the user of the missing information in MP4 file or track unit acquired from the header information (moov) by the error information analysis unit 405, and determines whether or not to reproduce the MP4 file to the user. An inquiry is made (step 1001). Next, it is determined whether or not the user has instructed reproduction in response to the inquiry (step 1002). As a result, if reproduction is instructed (yes in step 1002), the missing information notification unit 406c instructs the reproduction by outputting a reproduction instruction signal to the sample acquisition unit 404 (step 1004). Playback is started from the input playback start time, and the sample is decoded and played back until the last sample or the end of playback is instructed by the user (step 1005). On the other hand, if reproduction is not instructed (no in step 1002), the user is notified that the MP4 file is not reproduced (step 1003).

  Next, a procedure for reproducing a sample in step 1005 will be described. FIG. 20 is a flowchart showing the operation of Step 1005. First, in step 1101, a sync sample for starting decoding is determined based on the input reproduction start time. After the processing of step 1101, the processing of loop B is performed. Loop B is a loop process in which the processes from step 1102 to step 1106 are repeated, and is repeated each time a new sync sample is detected during decoding. In step 1102, error information such as whether or not data is missing is obtained from the error sample box, error free end sample box, or error free sequence box until the next sync sample. In step 1103, based on the error information acquired in step 1102, samples to be decoded in the sequence are determined, and a list of sample numbers of samples to be decoded is stored in a table. After the end of step 1103, in the process of loop A, the samples determined to be decoded are sequentially decoded and displayed by the table created in step 1103. Loop A is composed of the processing from step 1104 to step 1106, and is repeated until the processing of the final sample to be decoded at the time of reproduction ends. In step 1104, sample data and sample display time are acquired by analyzing moov or moof. In step 1105, the sample data acquired in step 1104 is decoded. In step 1106, the sample data decoded in step 1105 is displayed based on the display time acquired in step 1104.

  Hereinafter, in step 1102, a procedure for determining a sample to be decoded with reference to each box of the Error Sample Box, Error Free End Sample Box, and Error Free Sequence Box will be described in order.

  First, when referring to the Error Sample Box, it is determined that a sample indicating that data is missing is not decoded. Note that when data loss information in units of NAL units of slices can be acquired, whether to decode in units of slice NAL units in a sample may be determined. In addition, when it is indicated whether or not the data of the slice NAL unit referred to when decoding the slice NAL unit in the sample is missing, only when the data of the slice NAL unit of the reference destination is completely prepared The data of the slice NAL unit may be decoded. For example, when the coding type of the slice NAL unit constituting each sample changes periodically, if the periodic structure is known, the data of the sample referred to by the slice NAL unit can be specified. As an example, it is assumed that the period of the coding type is 15 samples, and the coding type of the slice NAL unit in each sample in the period is IBBPBBPBBPBBPBB in decoding order. Here, I is a sample composed of an intra slice or IDR slice, B is a sample composed of a bi-prediction slice, and P is a sample composed of a unidirectional prediction slice. At this time, if the B sample refers to two samples other than the B sample immediately preceding in the decoding order, and the P sample refers to a sample other than the B sample immediately preceding in the decoding order, the slice data of the B sample and the P sample is The sample to be referenced can be specified.

  Next, when determining the samples to be decoded with reference to the Error Free End Sample Box, it is determined that the samples not included in the error free section are not decoded.

  Finally, when determining a sample to be decoded with reference to the Error Free Sequence Box, if a sample with missing data is included between two consecutive sync samples, only the first sync sample is decoded.

  When two or more Error Sample Boxes, Error Free End Sample Boxes, and Error Free Sequence Boxes are used at the same time, the method for determining the sample to be decoded is changed according to the Box used. Also good. For example, when the Error Sample Box and the Error Free End Sample Box are used at the same time, samples that can be correctly decoded are indicated by the information in the Error Sample Box even for samples that are not included in the error free section. At this time, samples outside the error-free interval may be decoded. The same applies when the Error Sample Box and Error Free Sequence Box are used simultaneously.

  In the present embodiment, the missing information notification unit 406c makes an inquiry to the user as to whether or not to reproduce the MP4 file. However, the present invention is not limited to this. For example, the missing information notification unit 406c may simply notify the user of information such as the missing rate of sample data without inquiring the user whether or not to play the MP4 file. The missing information notification unit 406c may determine information such as the missing rate of sample data with a predetermined threshold, and output a reproduction instruction signal to the sample acquisition unit 404 when the condition is satisfied. Furthermore, the missing information notification unit 406c determines information such as the missing rate of sample data with a predetermined threshold, and inquires of the user whether or not to play the MP4 file only when the condition is not satisfied. It doesn't matter.

  The multiplexing apparatus can also play back conventional MP4 files.

(Embodiment 5)
A demultiplexing apparatus according to Embodiment 5 of the present invention will be described with reference to FIG. This demultiplexer inputs the MP4 file created by the multiplexing method converter according to the first to third embodiments, separates the AU data, decodes and displays it. The decoding or display operation is determined by referring to the error information indicated in the error notification NAL unit inserted in the sample data. In the MP4 file input to the demultiplexer, a box indicating error information may not be added. In the following, the operation when reproducing an MPEG-4 AVC track will be described. However, the reproduced track is not limited to MPEG-4 AVC. It may be video encoded data such as H.263 or MPEG-4 Visual, or audio encoded data such as MPEG-4 AAC or AMR, or a video and audio track may be reproduced simultaneously.

  The configuration of the present demultiplexing apparatus is the same as that of the demultiplexing apparatus according to the fourth embodiment shown in FIG. 18 except for the error information analysis unit 405, and thus the description thereof is omitted. FIG. 21 is a flowchart showing the operation of the present demultiplexer. First, in step 1101, a sync sample for starting decoding is determined based on the input reproduction start time. After the process of step 1101, the process of loop A is performed. Loop A is a loop process in which the processes from step 1102 to step 1005 are repeated, and is repeated until the process of the final sample to be decoded at the time of reproduction ends. In step 1102, sample data to be decoded is acquired. Henceforth, the process from step 1103 to step 1105 becomes a process in the decoding display part 406. FIG. In step 1103, in the sample data acquired in step 1102, an error notification NAL unit or sample is specified, and when it is indicated that the sample cannot be correctly decoded, the sample is not decoded. Subsequently, in step 1104, the data of the slice NAL unit determined to be decoded in step 1103 is decoded, and in step 1105, the decoded sample data is displayed based on the display time.

(Embodiment 6)
Here, a system using the multiplexing method conversion apparatus and the demultiplexing apparatus shown in the first to fifth embodiments will be described.

  FIG. 22 is a block diagram showing an overall configuration of a system that realizes a content distribution service by broadcasting and communication. First, a case where broadcast data is received will be described. A cellular phone ex105 or a disk recorder ex104 such as a DVD recorder receives a TS packet sequence in which digitized encoded media data is multiplexed. In the mobile phone ex105, the received TS packet sequence is converted into MP4 by the multiplexing system conversion device according to the present invention, and then recorded on the SD card ex106. The recorded MP4 file can be viewed on a mobile phone ex105, a disk recorder ex104, or a personal computer (not shown) equipped with the demultiplexer according to the present invention. In addition, the MP4 file is attached to an e-mail and transmitted from the mobile phone ex105 via the radio base station ex107 to another mobile phone ex108 equipped with the demultiplexing device according to the present invention. You can also watch the file. Furthermore, instead of mail attachment, download may be performed from the mobile phone ex105 to the mobile phone ex108 or pseudo-streaming distribution using a protocol such as HTTP (Hyper Text Transport Protocol) and TCP (Transmission Control Protocol).

  Also in the disk recorder ex104, the received TS packet sequence can be converted into MP4 by the multiplexing system conversion apparatus according to the present invention and recorded on an optical disk such as an SD card or a DVD, or a hard disk. The recorded MP4 file may be downloaded to a mobile phone or a personal computer (not shown) or pseudo-streamed. Note that a TS packet sequence may be recorded on the SD card and converted to MP4 at the time of distribution.

  When receiving the TS packet sequence distributed from the content server ex102 via the Internet at the mobile phone ex105 or the disc recorder ex104, the MP4 file can be used in the same manner as when the broadcast data is received.

(Embodiment 7)
A program for realizing the multiplexing method conversion method in the multiplexing method conversion device and the demultiplexing method in the demultiplexing device described in each of the above embodiments is recorded on a storage medium such as a flexible disk. As a result, the processing described in each of the above embodiments can be easily performed in an independent computer system.

  FIG. 23 is a diagram illustrating a method of performing the multiplexing method conversion method in the multiplexing method conversion device and the demultiplexing method in the demultiplexing device according to each embodiment described above using a program recorded on a recording medium such as a flexible disk. It is explanatory drawing in the case of implementing by a system.

  FIG. 23B shows an external appearance, a cross-sectional structure, and a flexible disk when viewed from the front of the flexible disk, and FIG. 23A shows an example of a physical format of the flexible disk that is a recording medium body. The flexible disk FD is built in the case F, and on the surface of the disk, a plurality of tracks Tr are formed concentrically from the outer periphery toward the inner periphery, and each track is divided into 16 sectors Se in the angular direction. ing. Therefore, in the flexible disk storing the program, the program is recorded in an area allocated on the flexible disk FD.

  FIG. 23C shows a configuration for recording and reproducing the program on the flexible disk FD. When the above-described program for realizing the multiplexing method conversion method in the multiplexing method conversion device and the demultiplexing method in the demultiplexing device is recorded on the flexible disk FD, the program is transferred from the computer system Cs via the flexible disk drive. Write. In addition, when a multiplexing system conversion method in the multiplexing system conversion apparatus of each of the above embodiments and a demultiplexing method in the demultiplexing apparatus are constructed in a computer system by a program in the flexible disk, a flexible disk drive is used. Read the program from the flexible disk and transfer it to the computer system.

  In the above description, a flexible disk is used as the recording medium, but the same can be done using an optical disk. Further, the recording medium is not limited to this, and any recording medium such as an IC card or a ROM cassette capable of recording a program can be similarly implemented.

  The multiplexing system conversion apparatus according to the present invention can be applied to all devices that convert and record or transmit encoded data such as MPEG-4 AVC transmitted by a TS to MP4, particularly digital broadcasting. This is effective for receiving mobile phones.

It is a block diagram which shows the whole structure of the multiplexing system converter which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the conversion part 103 in the multiplexing system converter which concerns on Embodiment 1 of this invention. It is a flowchart which shows operation | movement of the multiplexing system converter which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the Box data determination part in the multiplexing system converter which concerns on Embodiment 1 of this invention. It is a flowchart which shows the operation | movement of the Box creation part 205 in the multiplexing system converter which concerns on Embodiment 1 of this invention. It is a flowchart which shows the production | generation operation | movement of the header part of MP4 in the Box production | generation part 205 of the multiplexing system converter concerning Embodiment 1 of this invention. It is a flowchart which shows the production | generation operation | movement of Error Sample Box in the multiplexing system converter which concerns on Embodiment 1 of this invention. It is an example of the production | generation operation | movement of Error Sample Box in the multiplexing system converter which concerns on Embodiment 1 of this invention. It is a flowchart which shows the determination method of the sample data in the multiplexing system converter which concerns on Embodiment 1 of this invention. In the multiplexing system conversion apparatus according to the first embodiment of the present invention, it is an example of sample data conversion when the packet loss does not cross the NAL unit. In the multiplexing system conversion apparatus according to the first embodiment of the present invention, it is an example of sample data conversion when the packet loss crosses the NAL unit. 5 is an example of sample data conversion when a packet loss crosses an AU boundary in the multiplexing system conversion apparatus according to Embodiment 1 of the present invention. It is a flowchart which shows the preparation method of the sync sample box in the multiplexing system converter which concerns on Embodiment 2 of this invention. It is a creation example of the sync sample box in the multiplexing system converter which concerns on Embodiment 2 of this invention. It is a flowchart which shows the production | generation operation | movement of Error Free End Sample Box in the multiplexing system converter which concerns on Embodiment 3 of this invention. It is an example of the production | generation operation | movement of Error Free End Sample Box in the multiplexing system converter which concerns on Embodiment 3 of this invention. It is an example of the production | generation operation | movement of Error Free Sequence Box in the multiplexing system converter which concerns on Embodiment 3 of this invention. It is a block diagram which shows the structure of the demultiplexing apparatus which concerns on Embodiment 4 of this invention. It is a flowchart which shows the operation | movement at the time of reproducing | regenerating MP4 file in the demultiplexing apparatus which concerns on Embodiment 4 of this invention. It is a flowchart which shows operation | movement of the demultiplexing apparatus which concerns on Embodiment 4 of this invention. It is a flowchart figure which shows operation | movement of the demultiplexing apparatus which concerns on Embodiment 5 of this invention. It is a figure which shows the usage example of the multiplexing system converter in this Embodiment 6. FIG. It is explanatory drawing about the storage medium for storing the program for implement | achieving the multiplexing system conversion method in the multiplexing system converter of each said embodiment, and the demultiplexing method in a demultiplexing apparatus by a computer system. . It is a figure which shows the data structure of AU in MPEG-4 AVC. It is a figure which shows the data structure of a PES packet and TS packet. It is a figure which shows the Box structure of MP4. It is a figure which shows the hierarchical structure of moov in MP4. It is a figure which shows the usage method of moof in MP4. It is a flowchart which shows operation | movement of the conventional multiplexing system converter. It is a flowchart which shows the creation operation of the MP4 file in the conventional multiplexing system converter.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Data I / O part 102 Memory 103 Conversion part 201 TS payload separation part 202 Packet loss determination part 203 PES separation analysis part 204 AU separation part 205 PES header analysis part 206 Box creation part 207 Connection part 301 AU analysis part 302 Error information generation Unit 303 AU format conversion unit 304 box data determination unit 401 header separation unit 402 header memory 403 mdat memory 404 sample acquisition unit 405 error information analysis unit 406 decoding reproduction unit 406a decoding unit 406b sample display unit 406c missing information notification unit

Claims (17)

A multiplexing method conversion apparatus for multiplexing and outputting encoded media data packet-multiplexed by a first method according to a second method different from the first method,
Packet loss determination means for determining presence or absence of packet loss in the encoded media data packet-multiplexed by the first method;
When the packet loss determination means determines that there is a packet loss, based on the position where the data in the encoded media data is lost, missing information generation means for generating missing information regarding data loss;
Multiplex means for storing the missing information generated by the missing information generating means in the packet of the second scheme and for multiplexing the encoded media data by the second scheme. Multiplexing system conversion device. The multiplexing method converter further includes:
2. The multiplexing method conversion apparatus according to claim 1, further comprising: a packet separation unit that separates a header and a payload from a packet of encoded media data packet-multiplexed by the first method. The multiplexing system conversion apparatus according to claim 2, wherein the packet loss determination unit determines whether or not there is a packet loss based on the header separated by the packet separation unit. The multiplexing method converter further includes:
Frame separation means for detecting a frame boundary of the encoded media data from the payload separated by the packet separation means and separating frames;
The missing information generating means generates missing information on data loss in units of frames based on a position where the data in the encoded media data is lost when the packet loss determining means determines that there is a packet loss. ,
The multiplexing method conversion apparatus according to claim 2, wherein the multiplexing means stores the missing information in units of frames generated by the missing information generating means in the packet of the second method. 5. The multiplexing according to claim 4, wherein the missing information generation unit generates information indicating a section in which one or more frames in which decoding essential data is not lost in a decoding order are continued as the missing information. Method conversion device. 6. The multiplexing method conversion apparatus according to claim 5, wherein the missing information generation unit generates information of a head frame of the section as information indicating the section. 6. The multiplexing method conversion apparatus according to claim 5, wherein the missing information generation unit generates information of a last frame of the section as information indicating the section. The multiplexing method converter further includes:
A creation means for creating a table for registering a frame that can be randomly accessed is provided.
The multiplexing means, when registering a random accessible frame of the encoded media data in the table, data essential for decoding in the frame based on the position where the data in the encoded media data is missing The frame is not registered in the table as a randomly accessible frame if data essential for decoding is missing in the frame. Multiplexing system converter. The multiplexing method converter further includes:
5. The multiplexing method conversion apparatus according to claim 4, further comprising conversion means for converting a storage format of the encoded media data. The frame is composed of one or more subframe units;
The converting means adds the size information of the subframe to the head of each subframe and converts the storage format based on the position where the data is missing in the encoded media data. The multiplexing system conversion apparatus according to claim 9, wherein the subframe data stored in the system packet and the size of the subframe data are determined. The frame is composed of one or more subframe units;
The converting means inserts a subframe unit indicating data loss at a position where data in the encoded media data is lost, and packet-multiplexes the encoded media data by the second method. The multiplexing system conversion apparatus according to claim 9, wherein: The multiplexing system conversion apparatus according to claim 1, wherein the missing information generating unit generates, as the missing information, information indicating whether or not data essential for decoding in a predetermined unit is missing. A demultiplexer that decodes multiplexed data in which encoded media data is packet-multiplexed,
Missing information extracting means for extracting missing information related to missing data from the multiplexed data;
Missing information analyzing means for determining data to be decoded in the encoded media data based on the missing information extracted by the missing information extracting means;
And a decoding means for decoding the data determined to be decoded by the missing information analyzing means. The demultiplexer further comprises:
The demultiplexing apparatus according to claim 13, further comprising missing information notifying means for notifying a user of reproduction quality based on the missing information extracted by the missing information extracting means. The demultiplexing apparatus according to claim 14, wherein the missing information notification unit receives a user instruction based on the notification. A multiplexing method conversion method for multiplexing and outputting encoded media data packet-multiplexed according to a first method by packet multiplexing according to a second method different from the first method,
A packet loss determination step for determining the presence or absence of a packet loss in the encoded media data packet-multiplexed by the first method;
A missing information generating step for generating missing information regarding data loss based on a position where data is missing in the encoded media data when it is determined that there is a packet loss in the packet loss determining step;
Storing the missing information generated in the missing information generating step in the packet of the second scheme, and a multiplexing step of multiplexing the encoded media data by the second scheme. Multiplexing method conversion method. A program for multiplexing and outputting encoded media data packet-multiplexed according to a first method by packet multiplexing according to a second method different from the first method,
A packet loss determination step for determining the presence or absence of a packet loss in the encoded media data packet-multiplexed by the first method;
A missing information generating step for generating missing information regarding data loss based on a position where data is missing in the encoded media data when it is determined that there is a packet loss in the packet loss determining step;
Storing the missing information generated in the missing information generating step in the packet of the second scheme, and causing the computer to execute a multiplexing step of packet multiplexing the encoded media data by the second scheme. A program characterized by

Images