JP2002369152A - Image processing apparatus, image processing method, image processing program, and computer-readable storage medium storing image processing program - Google Patents

Abstract

(57) [Summary] To decode encoded data subjected to object encoding, audio and video are reproduced without a positional difference. An image processing apparatus for decoding and reproducing encoded audio data and encoded video data, comprising:
Video decoding units 4 and 5 for decoding video encoded data to generate video data, and audio decoding units 6 and 7 for decoding audio encoded data and generating audio data.
7, a position determining unit 9 for obtaining the position of the video object based on the input video encoded data, and the audio decoded by the audio decoding units 6 and 7 based on the position information determined by the position determining unit 9. And a mixer 10 for controlling the reproduction of the left and right channels of the data by the sound reproducing devices 11 and 12.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and method for inputting and decoding encoded image data composed of audio encoded data and video encoded data. The present invention relates to an image processing apparatus that inputs and decodes and reproduces image data, an image processing method, an image processing program, and a computer-readable storage medium that stores the image processing program.

[0002]

2. Description of the Related Art In recent years, MPE has been
The G (Moving Picture Experts Group) -4 coding scheme is being studied and its international standardization is underway. Conventional MP
In moving picture coding represented by EG-2, the coding unit is a frame or a field, whereas the MP is used to realize reuse and editing of contents.
In EG-4, video data and audio data are encoded as objects. Furthermore, the objects included in the video data are also encoded independently, and each can be treated as an object. The details are described in detail in, for example, "All about MPEG-4" edited by Seiichi Miki (Industrial Research Council), International Standard ISO / IEC14496-2, and the like.

According to the MPEG-4 encoding method, in order to handle an object to be encoded in units of objects (objects), the shape of an object in an image must be known at the time of encoding and decoding. In addition, in order to represent a glass-like object through which an object behind can be seen, information such as the degree of transparency of the object is required. Such information on the shape of the object and the transparency of the object is collectively referred to as shape information. The encoding of the shape information is called shape encoding.
The encoded data of the core profile or higher can handle this arbitrary shape. In this method, the inside and outside of an object are distinguished by the definition of the shape, and the inside of the object is processed by texture coding including motion compensation and DCT transform coding as in MPEG-1 and MPEG-2.

[0004] The profile defines a tool set for realizing an assumed application.

In order to describe the arrangement of objects and the like, BIFS (Binary)
Format for Scene) coding is employed. This is encoded data for describing a scene, and describes the size of the entire screen, the arrangement of objects, the timing of reproduction, and the like.

[0006] A functional configuration of an apparatus for reproducing moving image data encoded by MPEG-4 will be described below with reference to FIG.

In FIG. 14, reference numeral 1001 denotes MPEG-4
An encoded data input unit inputs encoded data encoded by MPEG-4. Reference numeral 1002 denotes a separator which separates coded data related to the system, coded data related to the video object, and coded data related to the audio object from the multiplexed MPEG-4 coded data. A system decoding unit 1003 decodes encoded data related to the system. 1004,1
Reference numeral 005 denotes a video decoding unit that decodes each video object. Reference numerals 1006 and 1007 denote audio decoding units for decoding audio objects corresponding to the left and right channels, respectively. An image synthesizing unit 1008 controls and synthesizes video objects decoded by the video decoding units 1004 and 1005 based on the output result of the system decoding unit 1003. Reference numeral 1009 denotes a mixer that synthesizes audio data decoded by the audio decoding units 1006 and 1007. Reference numeral 1010 denotes a display device (monitor) for displaying a synthesized image, and reference numerals 1011 and 1012 denote acoustic devices for reproducing a synthesized sound, which are arranged on the left and right sides to realize a stereo effect.

Now, the MPEG-4 encoded data input from the encoded data input unit 1001 is
Are separated into respective encoded data and input to the respective decoding units. FIG. 2 shows a configuration example of the object to be reproduced.

In FIG. 2, reference numeral 1100 denotes the entire screen. This screen is a female object (VO1) 1101.
And a male object (VO2) 1102. Then, the voice of the female object 1101 becomes a female audio object, and the male object 1101
The second voice is a male audio object.

In FIG. 14, the system decoding unit 1003
The decoder decodes the BIFS encoded data, and decodes the size of the screen 1100, the arrangement of the objects, and the synchronization between the audio object and the video object for video. Here, the arrangement of each object is represented by the positional relationship between the upper left corner of each object and the upper left corner of the screen.

The video decoding unit 1004 decodes image data of the female object 1101, and the video decoding unit 1005 decodes image data of the male object 1102. The audio decoding unit 1006 converts the audio object of the female object 1101 into an audio decoding unit 1007
Decodes the audio object of the male object 1102, respectively. The image synthesizing unit 1008 selects the female object 1101 and the male object 1 according to the position of the object output from the system decoding unit 1003.
102 is arranged. Similarly, mixer 1009 also outputs female object 1101 and male object 11 according to the position of the object output from system decoding section 1003.
02 is mixed, and the sound volume balance is adjusted so that the audio data can be output from the left and right audio devices 1011 and 1012. The monitor 1010 displays the output of the image synthesizing unit 1008, and outputs the audio devices 1011 and 101.
2 reproduces audio data.

[0012]

A video object is defined as a VideoObject in the MPEG-4 encoding method, but is defined to have a size including an encoding target from the beginning to the end of a frame. The encoding target is represented by a circumscribed rectangle including the encoding target for each frame, and its position and size are defined in each frame. Its position is
The main scanning direction is vop_horizontal_mc_spatial_ref code,
The sub-scanning direction is represented by a vop_vertical_mc_spatial_ref code, and the size is vop_wi in the main scanning direction.
The sub-scanning direction is represented by a vop_height code by the dth code.

When combining objects, the position of the object handled by the system code is represented by the positional relationship between the upper left corner and the screen. However, even if the object to be encoded moves within the object, the reproduction of the audio data is limited to the position handled by the system code, so that the positional relationship between the reproduction of the audio object and the video object is displaced, giving a sense of incongruity.

FIGS. 13A and 13B are diagrams showing an example.

FIG. 13 (a) shows the first frame, and FIG. 13 (b) shows the last frame when FIG. 13 (b) shows the last frame.
Is moved from the left end to the right end of the screen. Here, reference numeral 1210 denotes an object, and a thin rectangle 1211 denotes a VOP (Video Object Plane).
Represents the circumscribed rectangle of. Conventionally, even if a woman moves on the screen in this way, there is an uncomfortable feeling that the voice of the woman does not move with the movement. In particular, MPEG-
In the four-encoding, the reuse of the content is considered, and the rearrangement of the video object frequently occurs, so that it is necessary to remove the discomfort.

SUMMARY OF THE INVENTION The present invention has been made in view of the above conventional example, and has an image processing apparatus and an image processing apparatus for reproducing audio and video with no positional difference when decoding object-encoded data. It is an object to provide a method, an image processing program, and a computer-readable storage medium storing the image processing program.

It is another object of the present invention to provide an image processing apparatus, an image processing method, and an image processing method that eliminate a sense of incongruity in video and audio reproduction by changing the position of audio generation associated with the movement of a video object. An object of the present invention is to provide a computer-readable storage medium storing a program and an image processing program.

Another object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing method that eliminate a sense of incongruity in video and audio reproduction by changing the audio reproduction volume associated with the movement of a video object. An object of the present invention is to provide a computer-readable storage medium storing a program and an image processing program.

[0019]

In order to achieve the above object, an image processing apparatus according to the present invention has the following arrangement.
That is, an image processing apparatus that decodes and reproduces encoded audio data and encoded video data, decodes the encoded audio data and generates audio data, and decodes the encoded video data. Video decoding means for generating video data, sound source position obtaining means for obtaining a position of a video object based on the video encoded data, and audio decoding based on the position information obtained by the sound source position obtaining means. Audio reproduction control means for controlling reproduction of the audio data decoded by the means.

In order to achieve the above object, an image processing apparatus according to the present invention has the following arrangement. That is, an image processing apparatus that decodes and reproduces encoded audio data and encoded video data, decodes the encoded audio data and generates audio data, and decodes the encoded video data. Video decoding means for generating video data, sound source distance calculation means for obtaining a distance to an audio source included in the video based on the video encoded data, and distance information calculated by the sound source distance calculation means. Audio playback control means for controlling the playback volume of the audio data decoded by the audio decoding means based on the audio data.

In order to achieve the above object, the image processing method of the present invention comprises the following steps. That is, an image processing method in an image processing apparatus that decodes and reproduces encoded audio data and encoded video data, comprising: an audio decoding step of decoding encoded audio data to generate audio data; A video decoding step of decoding encoded data to generate video data, a sound source position obtaining step of determining a position of a video object based on the video encoded data, and based on the position information obtained in the sound source position obtaining step. And an audio reproduction control step of controlling reproduction of the audio data decoded in the audio decoding step.

In order to achieve the above object, the image processing method of the present invention comprises the following steps. That is, an image processing method in an image processing apparatus that decodes and reproduces encoded audio data and encoded video data, comprising: an audio decoding step of decoding encoded audio data to generate audio data; A video decoding step of decoding encoded data to generate video data, and a sound source distance calculating step of determining a distance to a sound source of audio included in the video based on the video encoded data,
An audio playback control step of controlling a playback volume of audio data decoded in the audio decoding step based on the distance information calculated in the sound source distance calculation step;
It is characterized by having.

[0023]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

[Embodiment 1] FIG. 1 is a block diagram showing a configuration of a moving image reproducing apparatus according to Embodiment 1 of the present invention. In the present embodiment, MPEG-4 encoded moving image data is input, and video and audio included in the encoded data are reproduced.

In FIG. 1, reference numeral 1 denotes an MPEG-4 encoded data input section for inputting MPEG-4 encoded data.
Reference numeral 2 denotes a separator that separates the MPEG-4 encoded data input from the MPEG-4 encoded data input unit 1 and inputs the separated data to the subsequent units. Reference numeral 3 denotes a system decoding unit, which is a BIF encoded by the MPEG-4 encoding method separated by the separator 2.
S-coded data is input and decoded. Reference numerals 4 and 5 denote video decoding units, which decode the video objects separated by the separator 2 on a frame (VOP) basis. Reference numerals 6 and 7 denote audio decoding units, which decode the audio objects separated by the separator 2 in unit time.

Here, a case where two objects are reproduced will be described. Here, an example of the object shown in FIG. 2 will be described.

Reference numeral 8 denotes an image synthesizing unit, which is a video decoding unit 4,
The video data decoded and reproduced in step 5 is synthesized according to the output of the system decoding unit 3. Reference numeral 9 denotes a position determination unit which determines the position of each video object as a sound source based on the outputs of the system decoding unit 3 and the video decoding units 4 and 5. Reference numeral 10 denotes a mixer for mixing audio data related to two objects in accordance with an output of the position determination unit 9. Reference numerals 11 and 12 denote audio devices which are constituted by speakers or the like and reproduce sound based on the audio signals. A display device (monitor) 13 displays an image.

The operation of the above configuration will be described.

The MPEG-4 encoded data input unit 1
An elementary stream conforming to the PEG-4 encoding method is input. The separator 2 separates the elementary stream input from the MPEG-4 encoded data input unit 1 into BIFS encoded data relating to the system, video encoded data of each object, and audio encoded data. Among them, the BIFS encoded data is transmitted to the system decoding unit 3 by the object 110 in FIG.
The encoded data of the video related to 1 is sent to the video decoding unit 4.
The encoded video data of the object 1102 is input to the video decoding unit 5, the encoded audio data of the object 1101 is input to the audio decoding unit 6, and the encoded audio data of the object 1102 is input to the audio decoding unit 7.

The system decoding section 3 decodes the BIFS code and outputs the pix described in the DecSpecificInfo descriptor.
Set the screen size according to elWidth and pixelHeight. The screen size at this time is the size in the main scanning direction (Im
age_x) and the size in the sub-scanning direction (Image_y).
Further, the system decoding unit 3 decodes the position of each object. Here, the position of the object 1101 in the main scanning direction is (VO1_loc_x), and the position of the object 1101 in the sub-scanning direction is (VO1_loc_y).
And the position of the object 1102 in the main scanning direction is (VO
2_loc_x), and the position in the sub-scanning direction is (VO2_loc_y).

In the video decoding unit 4, the object 110
1, the VOP size and the relative position of the VOP (the position of the VOP in the main scanning direction (VOP1_loc_x) and the position in the sub-scanning direction (VOP1_loc_y) of the object 1101) are obtained. Is decrypted. In the video decoding unit 5, similarly, the object 1102
, And decodes the encoded data of each VOP,
Relative position of VOP (VO related to object 1102)
The position of the P in the main scanning direction (VOP2_loc_x) and the position of the P in the sub-scanning direction (VOP2_loc_y) are obtained, and the image data is decoded.

FIG. 3 is a block diagram showing a detailed configuration of the video decoding unit 4 (5).

In FIG. 3, an MPEG-4 video data input section 51 inputs the MPEG-4 video encoded data separated by the separator 2. The separator 52 separates the encoded data of the header, the shape information, and the texture from the inputted MPEG-4 video encoded data and inputs the encoded data to the subsequent units. The header decoding unit 53
The position and size of P are decoded, and information necessary for decoding the video encoded data is set in each unit. The shape decoding unit 54 inputs and decodes the encoded data of the shape information separated by the separator 52. The texture decoding unit 55 decodes encoded data relating to the texture of the object to reproduce image data.

In such a configuration, the MPEG-4 video data input section 51 inputs video encoded data conforming to the MPEG-4 encoding system. The separator 52 includes:
From this input video encoded data, a Visual Object
Sequence layer, Visual Object layer, Video Object
It is separated into encoded data of each header of the Layer layer and Video Object Plane layer, encoded data relating to shape information, and encoded data relating to texture. The encoded data of the header is input to the header decoding unit 53, the encoded data of the shape is input to the shape decoding unit 54, and the encoded data of the texture is input to the texture decoding unit 55.

The header decoding section 53 decodes information necessary for decoding the VOP, such as the size and position of the VOP, and sets information necessary for decoding the video object in each section. Among them, information on the size and position of the VOP is output to the position determining unit 9. The shape decoding unit 54 outputs
The binary image data representing the shape information is decoded. When representing the inside of the object, the pixel value is “1” and the value outside the object is “0”. The result of the decoding is output to the image synthesizing unit 8. Texture decoding unit 5
5 decodes the texture of the object of each VOP to obtain image data, and the image data is also output to the image synthesizing unit 8.

The above operation is performed in the video decoding units 4 and 5 in the same manner except that the object of the video encoded data to be processed is different.

Returning again to FIG. 1, the audio decoding unit 6
Decodes audio data to be reproduced at the time of the video display interval for the object 1101 in FIG.
Similarly, the audio decoding unit 7 outputs the object 1 in FIG.
The audio data to be reproduced at the time of the video display interval related to 102 is decoded.

The position judging unit 9 receives information on the position of the object from the system decoding unit 3, the position of the VOP of the object 1101 from the video decoding unit 4, and the information on the position of the VOP of the object 1102 from the video decoding unit 5. From this information, parameters for performing audio mixing are determined. Here, the position of the sound source of the object 1101 is represented by (VO1_loc_x + VOP1_loc_x).
Further, since the size of the entire screen in the main scanning direction is Image_x, the mixing parameter P1 relating to the audio of the object 1101 is as follows: P1 = (VO1_loc_x + VOP1_loc_x) / Image_x (1) Similarly, the mixing parameter P2 relating to the audio of the object 1102 is as follows: P2 = (VO2_loc_x + VOP2_loc_x) / Image_x (2) These results are input to the mixer 10.

The mixer 10 mixes the decoded data from the audio decoding units 6 and 7 in accordance with the mixing parameters P1 and P2. Here, the size of audio data related to the object 1101 is conceptually A1, and the size of audio data related to the object 1102 is conceptually A2. Also, when audio data having a size of A1 is reproduced, the size reproduced by the left audio device 11 in stereo reproduction is conceptually A1L, and the size reproduced in the right audio device 12 is conceptually A1R. Is defined. Similarly, if the size is A2
A2L and A2R are also defined for the audio data.

First, a method of calculating A1L and A1R will be described. Here, the sound device 11 is finally
The loudness of the left sound coming out of the sound is conceptually defined as AL, and the loudness of the right sound is conceptually defined as AR.

First, the left and right balance of the object 1101 is calculated, and A1R and A1L are obtained. This is obtained by the following equation.

A1L = A1 × P1 (3) A1R = A1 × (1-P1) (4) Similarly, for A2L and A2R, A2L = A2 × P2 (5) A2R = A2 × (1-P2) (6) Accordingly, the loudnesses AR and AL of the sounds reproduced by the audio devices 11 and 12 are represented by the following equations, respectively.

AR = A1R + A2R (7) AL = A1L + A2L (8) In this way, the data mixed by the mixer 10 is input to the audio devices 11 and 12, respectively. Will be played. Further, the display device 13 displays the decoded video data output from the image synthesizing unit 8, and
Reference numeral 12 reproduces the audio decoded by the audio decoding units 6 and 7.

FIGS. 4 (a) and 4 (b) are views showing an example of the result of the synthesis in this manner.

FIG. 4A shows the first frame and corresponds to the example of FIG. FIG. 4B shows the state of the last frame. In the first frame of FIG. 4A, the female voice of the object 1101 is mainly the sound device 1 on the left side.
1 and mainly heard from the left side of the screen, in the last frame 1100 of FIG. 4B, the female voice of the object 1103 is mainly played back from the right acoustic device 12 and mainly You can play it as you hear it from the right side of the screen.

The processing up to the reproduction will be described with reference to FIG.

FIG. 5 is a flowchart showing a moving image reproducing process in the moving image reproducing apparatus according to the first embodiment.

First, in step S101, MPEG-4
MPEG-4 encoded data is input from the encoded data input unit 1. Next, in step S102, the system coded data separated by the separator 2 is decoded by the system decoding unit 3 to obtain position information of each object. Next, the process proceeds to step S103, in which "1" is substituted for a variable n for counting the number of frames, and initialization is performed. Next, step S1
04, by comparing the value of the variable n with the number of frames, it is determined whether or not the reproduction of the moving image has been completed. If it is the last frame, the process ends. Proceed and continue the process.

In step S105, the coded data of one frame (n-th frame) (VOP), which is a video object, is decoded, and information on the position and size of the VOP, shape information, and texture data are obtained. This is obtained by decoding the system encoded data separated by the separator 2 by the system decoding unit 3 and decoding one frame of the n-th video object by the video decoding units 4 and 5. Next, the process proceeds to step S106, where audio data for a time equivalent to the interval of one frame is decoded from the encoded data of the audio object of the n-th frame. Next, the process proceeds to step S107, and a mixing parameter in the mixer 10 is obtained by using the above equation (1) or (2). Next, step S108
Then, the image data of the video objects decoded by the video decoding units 4 and 5 are synthesized by the image synthesizing unit 8 based on the shape information and the texture information from the system decoding unit 3, and according to the decoding result of the system code. Place it on the screen. At the same time, the audio data decoded by the audio decoding units 6 and 7 is reproduced according to the above equations (3) to (8). Thus, in step S109, the image and audio are reproduced and displayed / output, and then in step S11
The process proceeds to 0, "1" is added to the variable n, and step S104 is repeated.
To determine whether the process has been completed.

By such a series of processing, the position of the object can be determined more faithfully, and the audio can be moved and reproduced in accordance with the position. it can. Also, since the audio playback position is updated in frame units, it is possible to realize smooth audio playback / movement accompanying video display.

In the first embodiment, the input encoded data is MPEG-4 encoded data. However, the present invention is not limited to this. For example, the JPEG2000 encoding method may be used for each frame. Of course, data encoded with the function of the ROI may be used.

The present invention is not limited to the processing procedure in the first embodiment. For example, step S105 and step S106 may be performed in parallel, and other processing may be performed in parallel. If so, of course, it does not matter if parallel processing is performed. Further, in the present embodiment, the case where there are two objects has been described. However, even in the case where there is one object, it can be realized in exactly the same way.
Even in the case of three or more, it is possible to easily cope with the situation by increasing the number of decoding units.

[Second Embodiment] As the second embodiment of the present invention, a case where the video decoding unit 4 or 5 has a different configuration using the configuration of the moving picture processing apparatus shown in FIG. 1 will be described.

FIG. 6 is a block diagram showing a configuration of video decoding section 4 (5) according to Embodiment 2 of the present invention. still,
The same components as those in the first embodiment (FIG. 3) are denoted by the same reference numerals, and detailed description thereof will be omitted.

In FIG. 6, reference numeral 151 denotes a shape center-of-gravity calculating unit which calculates the position of the center of gravity of the object from the shape information decoded by the shape decoding unit 54. Reference numeral 152 denotes an object position determining unit which determines the position of the object from the decoding result of the system code decoded by the header decoding unit 53 and the position information of the center of gravity calculated by the shape center of gravity calculating unit 151.

Here, as in the first embodiment, MPEG
The −4 video data input unit 51 inputs the video encoded data compliant with the MPEG-4 encoding scheme separated by the separator 2. The separator 52 separates the input video encoded data into encoded data of each header, encoded data relating to shape information, and encoded data relating to texture. Here, the encoded data of the header is transmitted to the header decoding unit 5.
3, the encoded data on the shape is sent to the shape decoding unit 54,
The encoded data relating to the texture is sent to the texture decoding unit 5.
5, respectively.

The header decoding unit 53 decodes information essential for decoding the VOP, such as the size and position of the VOP, sets information necessary for the decoding in each unit, and stores information on the size and position of the VOP in the object position. Judgment unit 1
52. The shape decoding unit 54 decodes binary image data representing the shape information of each VOP. The result decoded in this way is output to the image synthesizing unit 8 and also to the shape centroid calculating unit 151. Shape center of gravity calculation unit 15
In step 1, the center of gravity (coordinates in the main scanning direction (O_x) and coordinates in the sub-scanning direction (O_y)) of the area whose shape information is “1” is obtained, and the position of the center of gravity is input to the object position determination unit 152.

The object position judging unit 152 reads the position (VOP1_loc_x) of the VOP relating to the object in the main scanning direction from the output of the header decoding unit 53, and sets it as a new object position (VOP1_loc_p_x). This is obtained by the following equation (9).

VOP1_loc_p_x = VOP1_loc_x + O_x (9) The texture decoding unit 55 decodes the texture of the object of each VOP to obtain image data. The decoded image data is also output to the image synthesizing unit 8.

The position determining unit 9 determines the position (VOP1_loc_p_x) of the new object by using (VOP1_l
oc_x) to calculate the mixing parameter P1.

By such a series of processing, the position of each object can be determined more faithfully even when the VOP has an arbitrary shape having no circumscribed rectangle. In addition, since the position at which the corresponding audio is generated can be changed according to the position of the video object, the audio can be reproduced without discomfort with the video. In addition, since the audio playback position is updated on a frame basis, it is possible to reflect the smooth movement of the audio playback position according to the movement of the video object.

[Third Embodiment] FIG. 7 is a block diagram showing a configuration of a moving picture reproducing apparatus according to a third embodiment of the present invention. The same components as those in the first embodiment are given the same reference numerals, and detailed description thereof will be omitted. In the third embodiment, video and audio are reproduced by inputting MPEG-4 encoded data. Third Embodiment
In the following, the configuration of the object shown in FIG. 2 will be described as an example. Here, a case where a male video object approaches will be described.

FIGS. 8A and 8B are diagrams showing this state. FIG. 8A shows the first frame, and FIG.
Is the last frame. Here, the male object 1102 to be encoded has moved to the front in the final frame. Here, if the loudness of the man does not change even if the man moves in this way, a person looking at this screen will feel uncomfortable.

In FIG. 7, reference numerals 201 and 202 denote video decoding units, which decode the video objects separated by the separator 2 on a frame (VOP) basis. Reference numerals 203 and 204 denote volume control units, which are video decoding units 201 and 20 respectively.
2, the volume of the audio data decoded by the audio decoding units 6 and 7 is adjusted.

The operation of the moving picture reproducing apparatus having the above configuration will be described below.

As in the first embodiment, the MPEG-
The 4-coded data input unit 1 inputs an elementary stream conforming to the MPEG-4 coding method. The separator 2 separates the elementary streams, of which the BIFS encoded data is transmitted to the system decoding unit 3, the video encoded data relating to the object 1101 is transmitted to the video decoding unit 201, and the video encoded data relating to the object 1102 is transmitted to the system decoding unit 201. The encoded video data of the object 1101 is input to the audio decoding unit 6, and the encoded audio data of the object 1102 is input to the audio decoding unit 7.

The system decoding section 3 decodes the BIFS code and sets the size of the screen. Also, the position of each object is decoded. The video decoding unit 201 decodes the encoded data of each VOP of the object 1101,
And the relative position of the VOP are obtained to decode the image data. The video decoding unit 202 decodes the encoded data of each VOP of the object 1102, and
The relative position of the OP is obtained and the image data is decoded.

FIG. 9 is a block diagram showing a configuration of video decoding section 201 (202) according to the third embodiment.
In FIG. 9, the same components as those in FIG. 3 in the first embodiment are given the same reference numerals, and detailed description thereof will be omitted.

In FIG. 9, reference numeral 251 denotes a shape size calculation unit which calculates the size of an object from the shape information decoded by the shape decoding unit 54. An object position determination unit 252 determines the distance of the object from the output of the header decoding unit 53 and the shape information calculated by the shape size calculation unit 251.

Here, as in the first embodiment, M
The PEG-4 video data input unit 51 inputs video coded data compliant with the MPEG-4 coding scheme separated by the separator 2. The separator 52 separates the input encoded video data into encoded data of each header, encoded data relating to shape information, and encoded data relating to texture. The encoded data of the header is input to the header decoding unit 53, the encoded data of the shape is input to the shape decoding unit 54, and the encoded data of the texture is input to the texture decoding unit 55.

The header decoding unit 53 decodes information essential for decoding the VOP, such as the size and position of the VOP, sets information necessary for the decoding in each unit, and stores information on the size and position of the VOP in the position. Output to the determination unit 9.
The shape decoding unit 54 decodes the shape information of each VOP and outputs it to the image synthesizing unit 8,
The shape information is also supplied to 51. The shape size calculation unit 251 counts the number of pixels whose shape information is “1”, obtains the size Sn of the shape, and inputs the size Sn to the object distance determination unit 252.

The object position determining unit 252 calculates the size Sn in the current frame and the size Sn-1 of the previous frame.
And the distance parameter D1 is obtained according to the following equation (10).

D1 = (Sn / Sn−1) × α (10) The distance parameter D1 thus obtained is compared with a threshold value Td, and if it is smaller than the threshold value Td, D1 = 1 is set.
This distance parameter D1 is output to the volume controller 203. Here, α is a predetermined value. Further, the texture decoding unit 55 decodes the texture of the object of each VOP and obtains the image data. This image data is also output to the image synthesizing unit 8.

The video decoding unit 2 according to the third embodiment
02, the same processing is performed, the size Sn in the current frame is compared with the size Sn-1 one frame before, and the distance parameter D2 is obtained according to the following equation.

D2 = (Sn / Sn−1) × α (11) The distance parameter D2 obtained in this way is compared with a threshold value Td. If the distance parameter D2 is smaller than the threshold value Td, D2 = 1 is set.
This distance parameter D2 is output to the volume controller 204.

Returning to FIG. 7, the audio decoding unit 6
Decodes audio data to be reproduced at the time of the video display interval for the object 1101 in FIG.
Similarly, the audio decoding unit 7 decodes audio data to be reproduced at the time of the video display interval for the object 1102. The volume control unit 203 receives the distance parameter D1 of the object from the video decoding unit 201 and the audio data of the size A1 from the audio decoding unit 6 as described above. Then, the volume control unit 203 calculates a volume control parameter Mn from the input distance parameter D1 and the volume control parameter Mn-1 one frame before according to the following equation (12). The volume control parameter M1 of the first frame is set to "1".

Mn = D1 × Mn−1 (12) A1 is adjusted using the volume control parameter Mn,
The adjustment value A1m is obtained from the following equation (13).

A1m = A1 × Mn (13) The audio data thus adjusted is reproduced by the audio devices 11 and 12 in the same manner as in the first embodiment.

FIGS. 10 (a) and 10 (b) are diagrams showing the result of combining and displaying in this manner.

FIG. 10A shows the first frame.
FIG. 10B shows the state of the last frame. In the first frame, the male voice of the object 1102 is reproduced in a low volume, and in the last frame, the object 1112
As shown by 02a, since the man is approaching, his voice is reproduced loudly.

The processing up to the reproduction will be described with reference to FIG.

FIG. 11 is a flowchart showing a moving picture reproducing process in the moving picture reproducing apparatus according to Embodiment 3 of the present invention.

First, in step S201, MPEG-4
MPEG-4 encoded data is input from the encoded data input unit 1, and in step S202, system encoded data separated by the separator 2 is decoded by the system decoder 3 to obtain position information of the object. Next, proceeding to step S203, a variable n for counting the number of frames is set.
Is initialized by substituting "1" into Next, step S20
Then, the process proceeds to 4 to compare the value of the variable n with the number of frames to determine whether or not the reproduction of the image has been completed. However, if not, the process proceeds to step S205 and the process is continued.

In step S205, the video decoding unit 20
1 (202), the encoded data of one frame (VOP) in the video object is decoded, and information on the position and size of the VOP, shape information, and texture data are obtained. Next, the process proceeds to step S206, where audio data for a time equivalent to an interval of one frame is decoded from the encoded data of the audio object. Next, the process proceeds to step S207, where the size is calculated from the shape information of the video object. Next, in step S208, the above (1)
A volume control parameter corresponding to the video object is obtained by using equation (2).

Next, the process proceeds to step S209, where the image data of the video object is synthesized based on the shape information and the texture information, and is arranged and displayed on the screen according to the decoding result of the system code in the system decoding unit 3. At the same time, the audio data to be reproduced is reproduced according to the above-mentioned equation (13). Thus, step S21
0, the image and the audio are reproduced / displayed. Then, the process proceeds to step S211 to add "1" to the variable n, and then to step S20.
4 to determine whether the processing is completed.

By such processing, the reproduction volume of audio data related thereto is adjusted based on the distance estimated from the size of the video object, whereby video and audio with perspective can be reproduced.

In the third embodiment, the input is M
Although PEG-4 encoded data is used, the present invention is not limited to this. For example, JPEG20 for each frame
Of course, it is also possible to use data or the like coded with the ROI function in the 00 coding method. Further, the present invention is not limited to the processing procedure described in the third embodiment.
For example, step S205 and step S206 may be processed in parallel, and if there is another place that can be processed in parallel, parallel processing may be performed.

Further, in the third embodiment, the case where the number of objects is two has been described. However, even when the number of objects is one, it is completely the same. Even when the number of objects becomes three or more, the number of decoding units is increased. We can easily cope.

In the third embodiment, the volume control parameter Mn is calculated by comparing the size of the previous frame with the size of the current frame. However, the present invention is not limited to this. pixelWidth x pixelHeight)
By comparing with (14), it may be obtained as in equation (14).

Mn = Sn / (pixelWidth × pixelHeight) (14) Further, instead of the screen size (pixelWidth × pixelHeight), the maximum value of the VOP size may be used.

In the third embodiment, the size of the object is obtained from the shape information. However, the decoding result of the vop_eidth code and the vop_height code representing the size of the VOP may be used. Further, in the third embodiment, a function of adjusting the volume is provided separately for the sake of explanation, but the same processing may be performed by mixer 10 as a matter of course.

[Embodiment 4] FIG. 12 is a block diagram showing a configuration of a moving picture reproducing apparatus according to Embodiment 4 of the present invention. The same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.
In the fourth embodiment, MPEG-4 encoded video data and audio encoded data are input, and video and audio are reproduced.

In FIG. 12, reference numeral 501 denotes an MPEG-4 encoded data input section for inputting video encoded data and audio encoded data. Reference numeral 502 denotes a separator, which separates the encoded data input from the encoded data input unit 501 and inputs the separated data to the subsequent units. An image synthesis unit 503 synthesizes an image with a predetermined image size (Image_x, Image_y). Reference numeral 504 denotes a position determination unit which generates a sound source (video object) based on the thread information input from the video decoding units 4 and 5.
Is determined.

The operation of the above configuration will be described below.

Here, as in the first embodiment, M
The PEG-4 encoded data input unit 501 is an MPEG-4 encoded data input unit.
Video encoded data and audio encoded data conforming to the encoding method are input. The separator 502 transmits the encoded video data of the object 1101 of FIG. 2 to the video decoding unit 4, the encoded video data of the object 1102 to the video decoding unit 5, and the object 11
The audio coded data relating to the object 1102 is input to the audio decoding unit 7, and the audio coded data relating to the object 1102 is input to the audio decoding unit 7.

In the video decoding unit 4, the object 110
1, the encoded data of each VOP is decoded, and the size of the VOP and the relative position of the VOP are obtained to decode the image data.
In addition, the video decoding unit 5 outputs each VO of the object 1102.
The encoded data of P is decoded, and the size of the VOP, VO
The relative position of P is obtained and the image data is decoded.

[0097] The audio decoding unit 6
The audio data to be played back at the time of the video display interval related to 01 is decoded. Similarly, the audio decoding unit 7
Decodes audio data to be reproduced at the time of the video display interval for the object 1102. The position determining unit 504 receives the object 1101 from the video decoding unit 4.
, And the VOP position information of the object 1102 from the video decoding unit 5.
In this case, it is assumed that the position of each object overlaps the origin of the screen with the upper left end overlapping. Position determination unit 50
4 determines parameters P1 and P2 for performing audio mixing from the position information. Here, the position of the sound source of the object 1101 is represented by (VOP1_loc_x). Here, the size of the entire screen in the main scanning direction is (Image_
x), the mixing parameter P1 relating to the audio of the object 1101 is as follows: P1 = VOP1_loc_x / Image_x (15) Similarly, the mixing parameter P2 relating to the audio of the object 1102 is represented by P2 = VOP2_loc_x / Image_x (16). These results are input to the mixer 10.

As in the first embodiment, the mixer 10 mixes the audio data input from the audio decoding units 6 and 7 according to the mixing parameters P1 and P2. The audio data thus mixed is input to the audio devices 11 and 12, respectively. The display device 13 displays the output from the image synthesizing unit 503, and the sound devices 11 and 12 reproduce sound.

By such processing, the position of the object can be determined more faithfully even without using the BIFS code relating to the system, and the audio can be reproduced without discomfort due to the movement of the video object.
In addition, since the position as the source of the related audio is updated in accordance with the movement of the video in units of frames, it is possible to realize the smooth movement of the sound source in accordance with the movement of the video.

Further, in the above embodiment, the acoustic device 11,
Although the present invention is not limited to this, the sound reproduction can be controlled by the position of the object in the sub-scanning direction if the sound devices 11 and 12 are also provided above and below. Is clear.

The present invention relates to a plurality of devices (for example, a host computer, an interface device, a video camera,
The present invention may be applied to a system including a video cassette recorder, a display, and the like, or may be applied to a device including one device (for example, a video camera, a video cassette recorder, and the like).

It is also an object of the present invention to provide a storage medium (or a storage medium) on which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or an apparatus, and a computer of the system or the apparatus. (Or C
PU and MPU) can also be achieved by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention. When the computer executes the readout program codes, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instructions of the program codes. This also includes a case where some or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing.

Further, after the program code read from the storage medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, based on the instruction of the program code, The case where the CPU of the function expansion card or the function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing. Although each embodiment has been described with reference to a single object in order to simplify the present invention, it is apparent that the same processing is performed for each object so as to correspond to a plurality of objects.

As described above, according to the present embodiment, when decoding object encoded data including a video object and an audio object such as MPEG-4, the position of the video object is taken into consideration.
By decoding and playing back the audio data related thereto, it is possible to play back a moving image that does not feel uncomfortable with the movement of the video.

Further, when a new image is created by combining these objects, there is an effect that a perspective and a three-dimensional effect can be easily obtained without defining a three-dimensional space.

[0106]

As described above, according to the present invention, when decoding the object-encoded data, audio and video can be reproduced without a positional difference.

Further, according to the present invention, it is possible to change the audio generation position related to the movement of the video object in accordance with the movement of the video object, thereby eliminating a sense of incongruity in video and audio reproduction.

Further, according to the present invention, it is possible to change the audio reproduction volume associated with the movement of the video object, thereby eliminating a sense of incongruity in video and audio reproduction.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a functional configuration of a moving image playback device according to Embodiment 1 of the present invention.

FIG. 2 is a diagram illustrating an example of combining objects according to the first embodiment;

FIG. 3 is a block diagram illustrating a configuration of a video decoding unit according to Embodiment 1 of the present invention.

FIG. 4 is a diagram illustrating an example of combining objects according to the first embodiment of the present invention.

FIG. 5 is a flowchart illustrating a decoding process in the video playback device according to Embodiment 1 of the present invention;

FIG. 6 is a block diagram showing a functional configuration of a video decoding unit according to Embodiment 2 of the present invention.

FIG. 7 is a block diagram showing a functional configuration of a moving image reproducing apparatus according to Embodiment 3 of the present invention.

FIG. 8 is a diagram illustrating an example of an object according to Embodiment 3 of the present invention.

FIG. 9 is a block diagram showing a functional configuration of a video decoding unit according to Embodiment 3 of the present invention.

FIG. 10 is a diagram illustrating an example of combining objects according to the third embodiment of the present invention.

FIG. 11 is a flowchart illustrating a decoding process in the video playback device according to Embodiment 3 of the present invention.

FIG. 12 is a block diagram showing a functional configuration of a moving image reproducing apparatus according to Embodiment 4 of the present invention.

FIG. 13 is a diagram illustrating a display example of an object.

FIG. 14 is a block diagram showing a configuration of a conventional moving picture reproducing apparatus for MPEG-4 encoded data.

Claims (18)

[Claims] 1. An image processing apparatus for decoding and reproducing encoded audio data and encoded video data, comprising: audio decoding means for decoding audio encoded data to generate audio data; Video decoding means for decoding data to generate video data, sound source position obtaining means for determining the position of a video object based on the video encoded data, based on the position information obtained by the sound source position obtaining means, An image processing apparatus, comprising: audio playback control means for controlling playback of audio data decoded by the audio decoding means. 2. A sound source position acquiring unit, comprising: shape information extracting unit for extracting shape information of a video object from the video encoded data; and a position of the video object from the shape information extracted by the shape information extracting unit. The image processing apparatus according to claim 1, further comprising: a determination unit configured to determine the position of the sound source by determining a position of the sound source. 3. The audio reproduction control means, based on a ratio between a position of the video object and a screen size obtained by the sound source position obtaining means, adjusts a reproduction volume balance of left and right channels of the decoded audio data. The image processing apparatus according to claim 1, wherein the adjustment is performed. 4. The image processing apparatus according to claim 1, wherein the audio reproduction control unit adjusts a reproduction volume balance between left and right channels of the audio data decoded by the audio decoding unit. 5. An image processing apparatus for decoding and reproducing encoded audio data and encoded video data, comprising: audio decoding means for decoding the encoded audio data to generate audio data; Video decoding means for decoding data to generate video data; sound source distance calculating means for calculating a distance to a sound source of audio included in the video based on the video encoded data; and sound source distance calculating means. And an audio reproduction control means for controlling a reproduction volume of the audio data decoded by the audio decoding means based on the distance information. 6. The sound source distance calculating means includes: shape information extracting means for extracting shape information of a video object from the encoded video data; and the video object includes the shape information extracted by the shape information extracting means. The image processing apparatus according to claim 5, further comprising: means for calculating a distance to a sound source to be transmitted. 7. The sound source distance calculation means, based on a size ratio of a video object of a previous frame extracted by the shape information extraction means and a video object included in a video object of a current frame. 7. The method according to claim 6, wherein the distance is obtained.
An image processing apparatus according to claim 1. 8. The video coded data is MPEG-coded data.
The image processing apparatus according to claim 1, wherein the image data is encoded data based on a four-encoding method. 9. An image processing method in an image processing device for decoding and reproducing encoded audio data and encoded video data, the audio decoding method comprising: decoding audio encoded data to generate audio data; A video decoding step of decoding video encoded data to generate video data; a sound source position obtaining step of determining a position of a video object based on the video encoded data; and a position information obtained in the sound source position obtaining step. An audio reproduction control step of controlling reproduction of the audio data decoded in the audio decoding step based on the audio data. 10. The sound source position obtaining step includes: a shape information extracting step of extracting shape information of a video object from the video encoded data; and a position of the video object from the shape information extracted in the shape information extracting step. And determining the position of the sound source by determining the position of the sound source. 11. The audio reproduction control step includes, based on a ratio between a position of the video object and a screen size obtained in the sound source position obtaining step, adjusting a reproduction volume balance of left and right channels of the decoded audio data. The image processing method according to claim 9, wherein adjustment is performed. 12. The image processing method according to claim 9, wherein in the audio reproduction control step, a reproduction volume balance of left and right channels of the audio data decoded in the audio decoding step is adjusted. 13. An image processing method in an image processing apparatus for decoding and reproducing encoded audio data and encoded video data, the audio decoding step comprising: decoding audio encoded data to generate audio data. A video decoding step of decoding video encoded data to generate video data; a sound source distance calculation step of calculating a distance to a sound source of audio included in the video based on the video encoded data; An audio playback control step of controlling a playback volume of the audio data decoded in the audio decoding step, based on the distance information calculated in the step,
An image processing method comprising: 14. The sound source distance calculating step includes: a shape information extracting step of extracting shape information of a video object from the video encoded data; and a video object included in the video object from the shape information extracted in the shape information extracting step. Calculating the distance to a sound source to be processed. 15. The sound source distance calculating step is based on a size ratio of a video object of a previous frame extracted in the shape information extracting step and a video object included in a video object of a current frame. The image processing method according to claim 14, wherein the distance is obtained. 16. The video coded data may be MPEG
The image processing method according to any one of claims 9 to 15, wherein the data is coded data according to a -4 coding method. 17. An image processing program for executing the image processing method according to claim 9. Description: 18. A computer-readable storage medium storing a program for executing the image processing method according to claim 9. Description: