JP2000209500A - A method of synthesizing and displaying a person image separately shot on a recorded background image and displaying and outputting the same, and a karaoke apparatus employing the method - Google Patents

Abstract

(57) [Summary] [Problem] To combine a person image with a background image, automatically adjust the position and size of the person image according to the content of the background image. SOLUTION: Image data serving as a background image is accumulated in a database, and the image data is associated with person synthesis prescription data for specifying an appropriate person synthesis layout according to the content of the background image. A video synthesizing unit that extracts specified video data from the database and reproduces the background video, and separates out a human video portion from a video signal captured separately and synthesizes and displays the human video portion on the background video. Based on the person-synthesizing prescription data associated with the image data to be output as a background image, synthesizing the cut-out part of the person-image portion and the size and size of the background image on the screen. The method is a person image synthesis method in which conditions are variably set.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for synthesizing and outputting a person's image separately shot on an existing recorded background image, and in particular, extracting only a person's portion from an image shot by a video camera. The present invention relates to a method of synthesizing a background image. Also, the present invention relates to a karaoke apparatus employing the method.

[0002]

2. Description of the Related Art Everyone has seen an image of a weather forecaster commenting on a weather map in the background of a weather forecast program on a television broadcast. The technique of synthesizing a person image that combines a person being photographed with a video camera with the background image and outputs the synthesized image is not limited to television broadcasting. If there is a video camera that captures a foreground of a person or the like, a video device that plays back a recorded (recorded) background image, and an editing device that has a chroma key combining function, similar combined images can be displayed on a display. Therefore, this video synthesis technology is being widely used for various purposes for entertainment and hobbies. For example, an application of this technology can be seen in a recent karaoke apparatus.

Usually, a karaoke apparatus outputs a mood image for a karaoke song on a display in synchronization with the sound output of the karaoke accompaniment music. A karaoke apparatus to which the above-described video synthesizing technique is applied reproduces an appropriate background video using a device for reproducing the mood video (such as a video CD player), and shoots the background video with a video camera. Chromakey composition of the singing person image. This makes it possible to create an image as if the singer is singing in a sightseeing spot or SF or fairy tale world around the world. As described above, by adding a person image synthesizing function to the karaoke apparatus, the entertainment of karaoke is improved, and the chance of using the karaoke apparatus is increased. .

[0004]

In the conventional person image synthesizing method, the person to be synthesized is displayed with the same size as the end even if the content (landscape, structure, etc.) included in the background image changes.
Certainly, the background video in television news and the like is a video of a news material, so it is more convenient that the display size of a person video such as an announcer does not change regardless of the video content.

On the other hand, in the case where a person is synthesized as if the person actually exists in a place where no person actually exists, the state of synthesis of the person (the display position and display size of the person on the screen) is changed according to the contents of the background. There is more realism. If you ’re composing a person against a still, distant landscape,
It may seem as if the person is present at the place without discomfort. However, the background image
In the case of going in / out, for example, the background, which should not move unless the display size of the person is changed according to the enlargement ratio of the background, causes an uncomfortable feeling that the background approaches or moves away from the person.

Further, in the case where a person stands on a street corner of a building in which a forest stands, if the person is not synthesized with an appropriate display size in accordance with the size of the building, the person will have a size like a "large monster". It looks like. When a person is synthesized with a background image such as a sandy beach with the sea in the background, if the person is "floating" on the sea, this is also very unnatural in terms of image expression. To avoid this, it is only necessary to variably control the composition while watching the composite video of the background video being reproduced and the human video being captured, but this requires advanced editing techniques and complicated editing operations. Become. Of course, since it is a manual operation, it takes a long time to reach an appropriate combination state.

[0007] In the case of using a person image synthesizing function supplementarily and additionally, such as a karaoke apparatus, there is no need for any operation other than the original operation (such as inputting music and adjusting the volume). I don't feel like using that function until I do a complicated operation. Therefore, an increase in utilization opportunities due to the addition of functions cannot be expected, and only equipment costs increase.

Accordingly, the present invention provides a person image synthesizing method for automatically deciding appropriate synthesizing conditions such as position and dimensions according to the contents of a background image and synthesizing a person image with the background without discomfort according to the determination. It is an object of the present invention to provide a karaoke apparatus employing the method.

[0009]

Means for Solving the Problems The first invention is specified by the following items (1) to (4). (1) This is a method in which a separately shot human image is combined with a recorded background image and displayed and output. (2) The video data serving as the background video is stored in a database. In this database, person synthesis prescription data that directly or indirectly specifies an appropriate person synthesis layout according to the content of a background image is associated with video data. (3) The specified video data is taken out from the database and a background video is reproduced, and a human video portion is cut out separately from the video signal and synthesized with the background video, and the synthesized video is displayed on a display. Video synthesizing means for outputting the image to the user. (4) The image synthesizing means, based on the person synthesizing prescription data associated with the image data to be output as a background image, places the cut-out person image portion at what position on the screen of the background image and at what size. Then, variably set the synthesis conditions such as whether to synthesize.

[0010] A second aspect of the present invention is a person image synthesizing method in which the image data is digital image data, and the person synthesized prescription data is multiplexed with the digital image data. The video data is recorded separately from the video data as data associated with the recording position of the video data within the video data, and the video synthesizing unit sequentially obtains the recording position of the video data being reproduced, and the person corresponding to the recording position. A third invention is a third aspect of the present invention, which is a person image synthesizing method in which a synthetic layout is variably controlled based on synthetic prescription data and synthesized and displayed.

The fourth invention provides the following items (41) to (4).
4). (41) A karaoke apparatus that employs the person image synthesis method according to any one of claims 1 to 3, wherein the background image is reproduced by extracting specified image data from the database, the image synthesis means, and the image database. And a video camera. (42) The music generation data that is the origin of the karaoke accompaniment music is stored for each karaoke song in the karaoke database. Karaoke performance processing means processes the music generation data of the designated music piece and outputs the accompaniment music as sound. (43) A background video designation sequence is defined for each karaoke song. The video reproducing means sequentially displays the specified video data from the video database in accordance with the specified background video sequence of the specified music, thereby displaying the background video on the display in synchronization with the audio output of the karaoke accompaniment music. . (44) According to the person composition prescription data, the image composition means appropriately variably controls the composition condition of the person image during the performance of the designated music piece and performs composition display. Further, in the fourth invention, for each karaoke song, the lyrics rendering data which is the origin of the lyrics image is stored in the karaoke database, and the lyrics rendering means processes the lyrics rendering data of the designated music and synchronizes with the accompaniment music. A fifth aspect of the present invention is a karaoke apparatus in which, when displaying the corresponding lyrics image on the display, the video synthesizing means combines and displays the person image and the lyrics image on the relevant background video.

[0012]

DESCRIPTION OF THE PREFERRED EMBODIMENTS === Basic Configuration and Operation of Karaoke Apparatus === FIG. 1 shows the configuration of a karaoke apparatus according to an embodiment of the present invention. The central control unit 11 has a CPU, RA
The karaoke apparatus 1 is a microcomputer including an M and a ROM, and performs data communication with peripheral components via a data bus 100 and a control bus 110 to control / control the karaoke apparatus 1.

<Data Processed by Karaoke Apparatus and Its Storage Unit> The hard disk drive 12 becomes the origin of the music generation data which is the origin of the karaoke accompaniment music and the lyrics image to be displayed on the display in synchronization with the music performance. A karaoke data set is stored using a song ID as a key, such as lyrics rendering data and a background video designation sequence described below.

The DVD-ROM changer 23 is a DVD-ROM changer.
A large number of ROMs (hereinafter, referred to as DVDs) are stored, and a built-in driving device drives an arbitrary DVD to randomly access an appropriate data recording position and read data recorded therein. The data recorded on the DVD is multiplexed data mainly composed of digital video data of the MPEG2-VIDEO standard for a long time.

The music generation data is sound source control data described in accordance with the MIDI standard, and describes at what time in the performance time sequence the synthesizer 13 should generate what accompaniment sound. The lyrics rendering data describes a formatted lyrics character, the timing of displaying / erasing the lyrics character string on the display, and a color changing pattern for the displayed lyrics character string. The background video designation sequence is data designating how to connect short-time videos displayed on the display 22 as background images of the lyrics image, and which data of which DVD stored in the DVD changer 23 is recorded. Data designating how long to play from the position is described in chronological order.

<Karaoke Performance Processing> The central control unit 11
The ID of the requested music input from the operation input unit such as the remote control transmitter 17 and the operation panel 18 is stored in the operation control unit 19
, The input order and the ID are stored in the RAM in association with each other, and the music performance reservation is registered. And
The karaoke data of the corresponding music is taken out according to the reservation order. The MIDI data is transferred to the synthesizer 13 to generate karaoke accompaniment music. The audio signal of the karaoke accompaniment music output from the synthesizer 13 is mixed with the singing audio signal from the microphone 15 and output as sound to the speaker 16.

On the other hand, the lyrics rendering data is processed in synchronization with the generation of the karaoke accompaniment music, the lyrics images that change color are sequentially generated as bitmap image data, and are developed in the video RAM 20. In addition, the DVD changer 23 is controlled according to the background video designation sequence, and the DVD changer 23 outputs multiplexed data including predetermined video data. The decoder 40 demultiplexes the read multiplexed data to extract digital video data,
The video data is decoded and NT corresponding to the background video is processed.
Play back to SC signal. The video controller 21 performs a process of superimposing the lyrics image in the video RAM 20 on the background video and outputs the video to the display 22.

The karaoke apparatus 1 of this embodiment has, in addition to the basic functions described above, a person image synthesizing function of photographing a singer who sings along with the karaoke music with the video camera 25 and synthesizing only the person part with the background image. Have.

=== Overview of Person Image Combining Function === The person image combining method of the present invention is basically based on image combining techniques such as chroma key combining. In this embodiment, a chroma key combining technique is applied. That is, the singer sings while standing at a predetermined position on the stage where the background wall and the stage are painted in a predetermined color (blue in this embodiment). The singer is photographed by the video camera 25. A blue portion (blue back portion) is cut out from the video captured by the video camera 25, and a video (person video) of only the singer is extracted. Then, the image control unit 30 performs a process of synthesizing the person image with the background image, and
2 is displayed.

However, the point that the present invention is decisively different from the conventional image synthesizing method is that the contents included in the background image, for example, landscapes and structures, dynamic changes thereof (such as zooming and panning), and background image The composite layout such as the display size and display position of a person image can be appropriately automatically controlled and combined according to the movement of each component (object: person or object) in the inside. In this embodiment, control data (personal synthesis prescription data) expressing information related to a composite layout is multiplex-recorded together with digital video data on a DVD in order to achieve this advanced person image synthesizing function.

In addition to the above-described superimposition processing of the lyric image, the video control unit 30 performs the input / output of the input video signal such as A / D and D / A conversion of the video signal, chroma key synthesis, enlargement / reduction or movement of the video, and the like. To perform various video effect processing on the
0 achieves the above-described person image synthesis function by performing image effect processing on the video signal of the background image and the captured image input based on the person synthesis prescription data.

=== People Synthetic Prescription Data ===
It is recorded as multiplexed data according to the stems standard on the same time series as the digital video data. Then, in accordance with the content of the background video reproduced at the same time as the processing of the composite prescription data, the user specifies the size and display coordinates of the person video to be synthesized on the display 22 screen as the display size and display coordinates. ing.

The display size is a relative size when the height of the screen of the display 22 is set to 1, and when this value is 1, it is necessary to combine the person images extracted from the photographed image without changing the scale. And If it is ２, a portrait image in which the height and width are reduced to に す る is set. If it is 2, the height and width are doubled. The display coordinates indicate the horizontal and vertical positions of the screen of the display 22 by x and y coordinates. The coordinates of the lower left corner are designated as (0, 0), and the coordinates of the upper right corner are designated as (640, 480). ing.

=== User Interface and Preliminary Settings =
== When a user specifies a karaoke song, an appropriate input operation to use the person image combining function (for example, a song number for using the person image combining function for the same song is separately prepared and the number is used as the input number) When the performance is designated, the central control unit 11 reserves the performance by activating the person image synthesizing function when performing the music.

Prior to the performance of the music, the singer stands on a special stage with a blue background. The position where the singer should stand is marked on this stage. The zoom magnification and the visual angle direction of the video camera 25 pointed at the stage are determined by a person having a normal physique standing on this mark (in this embodiment, a person of average adult height of Japanese) in a predetermined composition. Pre-fixed to fit. In this embodiment, it is fixed in the center of the field of view in the horizontal direction so as to be fully displayed vertically from the feet of the person to the top of the head. In addition,
The video camera 25 is controlled to start / stop shooting by an infrared signal from a video camera remote controller IF 26. Of course, the transmission path of the control signal may be wired.

The central control unit 11 instructs the image control unit 30 to start the person image synthesizing function when starting playing music. In addition, it instructs the decoder 40 to transmit the multiplex-separated person synthesized prescription data. Further, it causes the video camera remote controller IF 26 to transmit an infrared signal for causing the video camera 25 to start shooting.

=== Operation of Person Image Synthesizing Function === When music performance processing is started, the DVD
Multiplexed data including digital video data of a background video is output from the changer 23. The decoder 40 separates the multiplexed data into digital video data and the above-mentioned person video prescription data. The video data is decoded in the decoder 40 and input to the video control unit 30 as an analog video signal. The person image prescription data is input as digital data to the image control unit 30 via the data bus 100.

On the other hand, a video signal (analog signal) captured by the video camera is also input to the video control unit 30. The video control unit 30 converts the captured video into digital data and executes a process for the above-described human video synthesis function. First, a human video portion obtained by removing a blue background portion from a captured video is extracted. Then, the person image data is processed into image data in which display dimensions and display positions are appropriately converted based on the person prescription data.

With regard to the control of the display size, when the human image data is developed corresponding to each pixel on the display 22 screen, the pixels constituting the human image are reduced or enlarged by appropriately thinning out or interpolating. Or In this manner, the scale-converted person image data is generated.
Further, the display position of the scaled converted person image is moved according to the display coordinates. The person image is based on the mark position of the stage where the singer stands. This base point is moved to the specified display coordinates (x, y). In addition, since the top of the person may protrude from the screen at the time of enlargement,
In the y direction, the image is shifted downward by a new pixel generated by interpolation. For example, if the designation of the display size is 1.5, the base point is shifted downward by 480 × 0.5 = 240 pixels. In this way, the face of the person always enters the screen. The person image data whose scale and coordinates have been converted are converted into analog image signals, and inserted into the background image by conventional chromakey synthesis. Further, it also synthesizes the lyrics image developed in the video RAM 20 and outputs the image to the display 22.

FIG. 2 shows an outline of the above-described person image synthesizing function as a display state on a display screen. Here, an example is described in which a singer is combined with a background video that indicates that a car is parked in front of a building. In the conventional chroma key synthesis or when the display size is specified as 1 and the display position is specified as (320, 0), that is, in the case of performing the human image synthesis without performing the scale conversion or the moving process, the singer on the stage as a blue background is identified. A person video (A1) is extracted from the captured video. This is chroma-key composited with the background video (B) to display the composite video (C1).

On the other hand, according to the person image synthesizing method of the present invention, the image (A1) is digitally processed to generate a person image (A2) whose scale and position have been changed. This video (A
2) The background video (B) is chromakey-combined with the background video (B) to obtain a composite video (C2) with no unnatural feeling in the video expression.

=== Supplement, Other Embodiments === In the above embodiment, the scale of the person image is converted according to the display size of the person prescription data regardless of the singer's physique. However, if the singer is tall, it may run out of the field of view of the video camera. In that case,
The zoom magnification of the video camera may be changed in advance so that the singer just fits in the field of view of the captured video. Alternatively, the zoom magnification may be variably controlled while monitoring a person image extracted from a video image captured by the video camera, and the zoom magnification may be fixed when a blue-back color appears at the upper end of the screen. Conversely, when the singer is photographed small in the field of view of the video camera by a child or the like, the person image can be adjusted to the height of the display screen by changing the zoom magnification in the same manner. If the singer is within the field of view, the height of the person can be obtained from the extracted person video, so that the height of the person may be adjusted to the height of the screen by image processing without changing the zoom magnification. .

Further, instead of specifying the size of the person to be synthesized with the background image by a relative value based on the height of the screen, the size (height, width, etc.) of a predetermined portion such as the face of the person
May be specified. In this case, a predetermined part may be specified from the cut-out person image by image recognition, and the whole person may be similarly deformed and combined so that the part has a designated size.

When synthesizing a person image, in addition to the dimensions and positions described above, the image control unit 30 can perform operations such as rotating the person, making a mirror image, or copying and synthesizing a plurality of persons with the background image. In the case of video effect processing, it can be designated as person composition prescription data and composed with a layout according to it. In addition, a part of the person image such as the upper half or the lower half of the person image can be combined and displayed.

Further, it is easy to generate a background image by CG (computer graphics) technology.
In that case, the display position of the person image can be designated by a three-dimensional position in a virtual space by CG.
As a result, for example, a singer is placed on the stage of a concert hall, and the singer can be seen or hidden between large crowds in front of the stage. Can also be expressed.

In the above embodiment, the predetermined position on the stage is processed as the base point of the person image. However, the singer may be able to track the specified position even when moving on the stage. The horizontal width of the person can be obtained from the extracted person video. The intersection point between the center of the width and the stage surface (under the feet of the person) may be set as a base point. Alternatively, a video camera may be mounted on a camera platform that can freely rotate in the horizontal direction and control the rotation angle thereof, and track the singer so that the singer always appears in the center of the screen.

The person prescription data need not be directly defined as numerical values as in the above-described embodiment. Any information on the layout at the time of synthesis may be used. For example, the content of the background image is expressed by a keyword (sea, town, etc.)
The size and position corresponding to each keyword may be prepared in a table or the like.

It is apparent that the above-described method of synthesizing a person image can be applied not only to a karaoke apparatus but also to a method of synthesizing images from a plurality of sources using a personal computer or a video editing device. Of course, the background image and the person image need not be moving images, and may be still images. The same video compositing function can be realized if human compositing prescription data corresponding to the background video is prepared and the compositing process can be performed based on this data.

In the case where the background video is recorded on a medium such as an analog optical disk, the person prescription data cannot be prepared as multiplexed data of video data. When the person prescription data cannot be recorded in parallel with the video data as described above, the video recording position output at the time of reproduction, such as the frame number, the track number, or the elapsed playback time (linear time) from the video recording start point on the medium. (Frame data) and person-combined prescription data may be prepared in association with each other. Of course, when video data such as a disc changer is recorded over a plurality of media, the media ID is associated with the frame data. If this method is applied to a karaoke apparatus, a predetermined composite layout can be specified based on the frame data specified by the background video specifying sequence. Of course, it is also possible to prepare the pre-synthesized person data as data associated with the performance time series for each music piece.

[0040]

According to the method for synthesizing a person image according to the present invention, the prescription data is synthesized based on the prescription data based on the synthesized person prescription data expressing the information on the layout when the background image is synthesized with the person image. To perform the combined display process. Therefore, a person is automatically synthesized at an appropriate position, size, and the like in the background video without requiring complicated operations or advanced editing techniques. In addition, it is possible to follow a change in the background image. Therefore, a sense of reality is given to the synthesized video as if the person were present at the shooting site of the background video.

The recording format of the background image (digital, digital, etc.), such as multiplexing the person prescription data with the digital image data or using separate data in accordance with the reproduction time series of the image data.
The data format can be appropriately selected depending on the type of data. For this reason, an advanced person image combining function can be realized without depending on the data format of the background image.

When this person image synthesizing method is applied to a karaoke apparatus, a singer can be synthesized with a mood image displayed on a display at the time of playing a karaoke song as a background image without discomfort. In addition, if a lyric image is also inserted, the singer or companion can view the combined video simultaneously with the lyrics.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a karaoke apparatus according to an embodiment of the present invention.

FIG. 2 is a schematic explanatory diagram of a person image combining function in the embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Karaoke apparatus 11 Central control part 12 Hard disk drive 23 DVD changer 30 Video control part 40 Decoder

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 9/75 G06F 15/66 450 5D108 F term (Reference) 5B057 AA20 CA01 CB01 CE08 CE09 5C023 AA02 AA03 AA04 AA05 AA17 AA18 BA11 CA03 CA04 CA05 DA03 DA08 5C066 AA00 AA01 AA07 AA12 BA17 ED03 ED04 ED09 ED13 GA31 GB01 HA02 KE09 KM01 KM11 5C080 AA01 AA09 BB05 CC03 DD30 EE01 EE17 EE19 EE29 FF09 GG02 GG12 GG02 GG09 CA59 DA51 MM05 5D108 BA04 BB06 BD02 BE06 BF20

Claims (5)

[Claims] 1. The invention specified by the following items (1) to (4). (1) This is a method in which a separately shot human image is combined with a recorded background image and displayed and output. (2) The video data serving as the background video is stored in a database. In this database, person synthesis prescription data that directly or indirectly specifies an appropriate person synthesis layout according to the content of a background image is associated with video data. (3) The specified video data is taken out from the database and a background video is reproduced, and a human video portion is cut out separately from the video signal and synthesized with the background video, and the synthesized video is displayed on a display. Video synthesizing means for outputting the image to the user. (4) The image synthesizing unit, based on the person synthesis prescription data associated with the image data to be output as a background image, places the cut-out person image portion at what position on the screen of the background image and how large. Then, variably set the synthesis conditions such as whether to synthesize. 2. The method according to claim 1, wherein the image data is digital image data, and the person prescription data is multiplexed with the digital image data. 3. The personal prescription data according to claim 1, wherein the person prescription data is recorded separately from the video data as data associated with a recording position of the video data in the database. A person image synthesizing method, wherein a recording position of certain image data is sequentially obtained, and a composite layout is appropriately variably controlled on the basis of person prescription data corresponding to the recording position and synthesized and displayed. 4. The invention specified by the following items (41) to (44). (41) A karaoke apparatus that employs the person image synthesis method according to any one of claims 1 to 3, wherein the background image is reproduced by extracting specified image data from the database, the image synthesis means, and the image database. And a video camera. (42) The music generation data which is the origin of the karaoke accompaniment music is stored for each karaoke song in the karaoke database. Karaoke performance processing means processes the music generation data of the designated music piece and outputs the accompaniment music as sound. (43) A background video designation sequence is defined for each karaoke song. The video reproducing means sequentially displays the specified video data from the video database in accordance with the specified background video sequence of the specified music, thereby displaying the background video on the display in synchronization with the audio output of the karaoke accompaniment music. . (44) According to the person composition prescription data, the image composition means variably controls the composition condition of the person image during the performance of the designated music piece and performs composition display. 5. The karaoke apparatus according to claim 4, wherein
Lyric rendering data that is the origin of the lyrics image for each karaoke song is stored in the karaoke database, and the lyrics rendering means processes the lyrics rendering data of the designated song and synchronizes the corresponding lyrics image with the accompaniment music on the display. At the time of display output, the video synthesizing means synthesizes and displays the person image and the lyrics image on the corresponding background image.