JP2006217183A - Data processor and program for generating multimedia data - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a means of enabling a user to easily generate multimedia data including data representing the image and the voice of the user himself or herself. <P>SOLUTION: A video generating device 10 displays performance instructions and words to be uttered by the user with characters on a display 18 according to event sequence data that the user selects by operating a controller 11, and performs various processing for sound data representing the voice of the user captured by a microphone 12 and image data representing the image of the user picked up by a video camera 15 to generate multimedia data wherein the user joins as a character. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a data generation technique for generating multimedia data.

  There is a technology that makes it possible to easily create multimedia data that suits user preferences. For example, a system that automatically generates an animated greeting card with a message and sends it to a specified e-mail address in response to the user selecting an animation title, entering a message, and specifying the recipient's e-mail address is widely used. ing.

For example, Patent Document 1 discloses that a multimedia data template indicating a greeting card or the like can be changed by the user freely within an allowable range so that the user can actively display new multimedia data while demonstrating creativity. A technique that enables generation is disclosed.
Japanese Patent Laid-Open No. 2004-7407

  By the way, in the case of the above prior art, a user's moving image cannot be used for generating multimedia data. Therefore, there is a problem that the user cannot easily create multimedia data including his / her moving image.

  In view of the above situation, an object of the present invention is to provide a means for a user to easily create multimedia data including his / her own moving image data.

  In order to achieve the above object, the present invention provides operation instruction data that stores operation instruction data indicating instruction of an operation to be performed by a user in association with timing data indicating a timing at which the instruction should be notified to the user. Storage means, image processing instruction data storage means for storing image processing instruction data for instructing processing to be executed on the image data in association with timing data indicating timing at which the processing is to be executed, The operation instruction data transmitting means for transmitting the operation instruction data to a notification device that notifies the user of instruction at a timing according to the timing data associated with the operation instruction data, and the image processing instruction data, User image data at a timing according to the timing data associated with the image processing instruction data To provide a data processing apparatus characterized by comprising an image processing instruction data transmission means for transmitting to the image data processing apparatus for processing.

  According to the data processing device having such a configuration, as a result of appropriate processing performed at an appropriate timing by the image data processing device, the image data of the user who operates according to the instruction notified at an appropriate timing by the notification device, Image data indicating a moving image that makes the user more entertaining is generated.

  The present invention also provides operation instruction data storage means for storing operation instruction data indicating instruction of an operation to be performed by a user in association with timing data indicating a timing at which the instruction is to be notified to the user, and image data. The image processing instruction data for instructing the processing to be executed is stored in association with the timing data indicating the timing at which the processing is to be executed, and the operation instruction data indicates Operation instruction notification means for notifying the user of the instruction to be performed at a timing corresponding to the timing data associated with the operation instruction data, and the timing associated with the image processing instruction data and the image processing instruction data To an image data processing device that processes user image data at a timing according to the data To provide a data processing apparatus characterized by comprising an image processing instruction data transmission means for signals.

  According to the data processing apparatus having such a configuration, the operation instruction notification unit notifies the user of instruction of an appropriate operation at an appropriate timing, and the image data processing apparatus operates on the image data of the user operating according to the instruction. As a result of performing an appropriate processing at an appropriate timing, image data indicating a moving image that makes the user more entertaining is generated.

  The present invention also provides operation instruction data storage means for storing operation instruction data indicating instruction of an operation to be performed by a user in association with timing data indicating a timing at which the instruction is to be notified to the user, and image data. Image processing instruction data storage means for storing image processing instruction data instructing a processing process to be executed in association with timing data indicating a timing at which the processing process is to be executed, and the operation instruction data, Image data for receiving user image data from the imaging device and operation instruction data transmitting means for transmitting to the notification device for notifying the user at the timing according to the timing data associated with the operation instruction data A receiving unit and a processing process instructed by the image processing instruction data are applied to the image processing instruction data. At a timing corresponding to the timing data attached, to provide a data processing apparatus characterized by comprising an image data processing means for performing the image data of the user.

  According to the data processing device having such a configuration, as a result of appropriate processing being performed at an appropriate timing by the image data processing means on the image data of the user operating according to the instruction notified at an appropriate timing by the notification device, Image data indicating a moving image that makes the user more entertaining is generated.

  The present invention also provides operation instruction data storage means for storing operation instruction data indicating instruction of an operation to be performed by a user in association with timing data indicating a timing at which the instruction is to be notified to the user, and image data. The image processing instruction data for instructing the processing to be executed is stored in association with the timing data indicating the timing at which the processing is to be executed, and the operation instruction data indicates Operation instruction notifying means for notifying the user at a timing according to the timing data associated with the operation instruction data, image data receiving means for receiving user image data from the imaging device, and A processing process instructed by the image processing instruction data is performed with a tag associated with the image processing instruction data. At a timing corresponding to the timing data, to provide a data processing apparatus characterized by comprising an image data processing means for performing the image data of the user.

  According to the data processing apparatus having such a configuration, the operation instruction notification unit notifies the user of the appropriate operation instruction at an appropriate timing, and the image data processing unit applies the image data of the user operating according to the instruction. As a result of performing an appropriate processing at an appropriate timing, image data indicating a moving image that makes the user more entertaining is generated.

  In a preferred aspect, the data processing device may further include a display unit that displays an image indicated by image data generated as a result of the processing by the image data processing unit.

  According to the data processing apparatus having such a configuration, the user can enjoy a moving image displayed according to the generated image data.

  In another preferred aspect, the data processing device may further include an image data storage unit that stores image data generated as a result of the processing by the image data processing unit.

  According to the data processing apparatus having such a configuration, the user can use the generated image data when desired.

  In another preferred embodiment, the data processing device stores utterance instruction data indicating utterance instruction to be performed by the user in association with timing data indicating timing at which the instruction should be notified to the user. Storage means, and voice processing instruction data storage means for storing voice processing instruction data for instructing processing to be executed on the voice data in association with timing data indicating timing at which the processing is to be executed; Voice guidance notification means for notifying the user of guidance indicated by the voice guidance data at a timing according to timing data associated with the voice guidance data, and voice data of the user are received from the sound pickup device. The voice data receiving means and the processing process instructed by the voice processing instruction data are processed by the voice At a timing corresponding to the timing data associated with physical instruction data may be further provided constituting the audio data processing means for performing relative audio data of the user.

  According to the data processing apparatus having such a configuration, the utterance instruction notifying unit notifies the user of instruction of appropriate utterance at an appropriate timing, and the voice data indicating the voice uttered by the user according to the instruction is voiced. As a result of performing appropriate processing at an appropriate timing by the data processing means, video data including audio data indicating audio that makes the user more entertaining is generated.

  In a preferred aspect, the data processing device may further include sound generation means for generating a sound indicated by sound data generated as a result of the processing by the sound data processing means.

  According to the data processing apparatus having such a configuration, the user can enjoy the sound indicated by the generated sound data.

  In another preferred aspect, the data processing device may further include sound data storage means for storing sound data generated as a result of the processing by the sound data processing means.

  According to the data processing apparatus having such a configuration, the user can use the generated sound data when desired.

  In a preferred aspect, the data processing device receives operation instruction data for instructing an operation to be executed by at least one of the imaging device, the moving device for moving the imaging device, and the illumination device at a timing at which the operation is to be executed. An operation instruction data storage unit that stores the operation instruction data in association with the timing data shown, and an imaging device that generates the image data of the user at a timing corresponding to the timing data associated with the operation instruction data, It is good also as a structure further provided with the movement apparatus which moves an imaging device, and the operation instruction data transmission means which transmits to at least 1 of the illuminating device which irradiates light with respect to a user.

  According to the data processing device having such a configuration, an image showing a moving image that makes the user more enjoyable because an appropriate imaging process, a moving process, or an illumination irradiation process is performed at an appropriate timing by the imaging device, the moving device, or the lighting device. Data is generated.

  In another preferred embodiment, the data processing device is configured to execute operation instructions data indicating operation instructions to be executed by at least one of an imaging device, a moving device that moves the imaging device, and an illumination device. An operation instruction data storage unit that stores the operation instruction data in association with timing data, and an imaging apparatus that generates the image data of the user at a timing corresponding to the timing data associated with the operation instruction data. And an operation instruction data transmitting means for transmitting to at least one of a moving device for moving the imaging device and an illuminating device for irradiating light to a user, and also for transmitting to the image data processing means. The data processing means processes the image data of the user according to the operation instruction data. It may be configured to perform.

  According to the data processing device having such a configuration, the imaging device, the moving device, or the illumination device performs appropriate imaging processing, movement processing, or illumination irradiation processing at an appropriate timing, and the image data processing unit performs these processing. Since the processing of the image data in conjunction with is performed, image data indicating a moving image that makes the user more entertaining is generated.

  In another preferred aspect, the data processing apparatus further comprises image data generating means for sequentially generating image data that changes in accordance with temporal changes in user image data sequentially received by the image data receiving means, The image data processing means performs processing according to the image processing instruction data on the image data generated by the image data generation means instead of or in addition to the user image data. It is good also as composition which performs.

  According to the data processing device having such a configuration, image data indicating a moving image that changes according to the user's operation is generated, and therefore, image data indicating a moving image that makes the user more entertaining is generated.

  In another preferred embodiment, the data processing device calculates a billing amount for the user based on at least one of the time required for the notification by the motion instruction notification means or the time required for the processing by the image data processing means. It is good also as a structure further provided with the accounting data processing means which produces | generates the accounting data shown.

  In another preferable aspect, the data processing device is configured to charge a user with a charge amount based on at least one of the number of notifications by the motion instruction notification unit or the number of processings by the image data processing unit. It may be configured to further include billing data processing means for generating data.

  According to the data processing apparatus having such a configuration, an appropriate charge is made according to the usage status of the user.

  The present invention also provides a program that causes a computer to execute processing performed by the data processing apparatus.

  According to the program having such a configuration, for example, a general-purpose computer generates image data indicating a moving image that makes the user more entertaining.

[1. First Embodiment]
Hereinafter, as an embodiment of the present invention, a system for automatically generating video data of a short movie in which a user participates as a character in a booth that can be arranged at a game center or the like will be described. In the following description, an image is a generic term for moving images and still images, and an image data is a generic term for moving image data and still image data.

[1.1. Configuration of video generation system]
FIG. 1 shows a state in which a video generation system 1 according to an embodiment of the present invention is installed in a booth. FIG. 2 shows the configuration of the video generation system 1. The video generation system 1 includes a video generation device 10, a controller 11 for giving a user instruction to the video generation device 10, picks up a user's voice, outputs a sound signal indicating the voice to the video generation device 10, and is intermittent. In particular, the microphones 12a and 12b having a function of emitting ultrasonic waves of a predetermined frequency, the amplifier 13 that amplifies the sound signal output from the video generation device 10, and the sound signal output from the amplifier 13 is converted into sound and output. The speaker 14L and the speaker 14R, the video cameras 15a to 15d that sequentially generate and output image data indicating the captured images, and the camera moving devices 16a to 16d that change the positions and directions of the video cameras 15a to 15d by motors, respectively. Illumination devices 17a to 17f that irradiate a user with light of a specified color and brightness, and the video generation device 10. From the display 18 that displays an image according to the output image data, the ultrasonic receivers 19a to 19d that receive the ultrasonic wave and output an electrical signal indicating the waveform of the received ultrasonic wave to the video generating device 10, and the video generating device 10 A media drive 20 that stores output data in a storage medium such as a flash memory is provided. In addition, the video generation device 10 is connected to the Internet 21.

  The video generation system 1 includes a plurality of microphones, speakers, video cameras, camera moving devices, illumination devices, and ultrasonic receivers. However, in the following, when it is not necessary to distinguish a plurality of similar components, It is simply called as the microphone 12. The number of microphones 12 and the like included in the video generation system 1 is an example, and can be freely changed except that the number of ultrasonic receivers 19 is four or more. Furthermore, the number of other components can be freely changed, for example, by providing a plurality of displays 18.

  The microphone 12 is preferably a small wireless microphone, and can be easily attached to the user's chest, for example, with a clip or the like. The microphone 12a and the microphone 12b are each provided with an ultrasonic generator 121a and an ultrasonic generator 121b that generate ultrasonic waves, and intermittently generate ultrasonic waves as described above. The frequencies of the ultrasonic waves generated by 121a and the ultrasonic generator 121b are different from each other, and it is possible to distinguish which ultrasonic wave is emitted from which microphone 12. The ultrasonic generator 121 repeats the generation of ultrasonic waves at a time interval sufficiently longer than the time required for the ultrasonic waves emitted from the microphone 12 to reach the ultrasonic receiver 19 in the booth. More specifically, for example, the ultrasonic generator 121 emits ultrasonic waves for 0.5 seconds, does not emit ultrasonic waves for 1 second, and thereafter repeats the same processing. This is because, in the processing of the microphone position calculation unit 205, which will be described later, one of the ultrasonic waves received by the ultrasonic receivers 19a to 19d having different distances from the microphone 12 one after another is emitted from the microphone 12 at the same timing. This is because it is possible to easily distinguish whether it is a product.

  Next, the configuration of the video generation device 10 will be described. The video generation device 10 includes a storage unit 101, an input / output I / F (Interface) 102, and a control unit 103. The storage unit 101 stores a program for instructing various processes by the control unit 103 in advance, and temporarily stores data generated by the control unit 103. Furthermore, the storage unit 101 stores a plurality of story data 1010 in advance.

  Story data 1010 instructs control of camera work and playback of background sound (background sound) in the shooting of one short movie, as well as instructions for display of speech and acting instruction text, reproduction of acting instruction voice data, etc. It is data to be. In this embodiment, when shooting a short movie by the video generation system 1, the user first selects a favorite one from a plurality of stories (hereinafter referred to as “story”). Therefore, the storage unit 101 stores in advance story data 1010 corresponding to each of a plurality of stories that can be selected by the user. Hereinafter, the configuration of the story data 1010 will be described.

  FIG. 3 schematically shows the configuration of the story data 1010 stored in the storage unit 101. FIG. 3 illustrates three story data 1010, that is, story data 1010a to 1010c. Each story data 1010 includes management data, event sequence data group, source data group, and arrangement data.

  FIG. 4 illustrates the contents of the management data. The management data includes data indicating the name of the story data 1010, the number and attributes of characters appearing in the story (hereinafter, “character” means a role appearing in the story).

The event sequence data group (see FIG. 3) is a collection of files storing event sequence data. Event sequence data refers to event data with timing consisting of event data indicating various processes to be executed as the story progresses, and timing data indicating timing at which the process specified by the event data should be executed. It is a set. In the present embodiment, the event sequence data is classified into the following six types.
(A) Event sequence data BS (background sound): event sequence data for instructing performance of background music or background sound.
(B) Event sequence data BI (background image): Event sequence data instructing display of a background image.
(C) Event sequence data CI (character image): Event sequence data for instructing display of an image showing a character.
(D) Event sequence data DL (dialog): Event sequence data instructing dialogue and performance guidance character display and sound reproduction.
(E) Event sequence data ME (microphone effect): Event sequence data instructing the addition of effect processing to the user's voice and the reproduction of the effect sound.
(F) Event sequence data CW (camera work): control of the focus and zoom of the video camera, control of the position and direction of the video camera, operation control of the illumination device, and the background image and character image according to those controls Event sequence data for instructing adjustment processing such as size change and color tone adjustment to be performed on the image.

In the present embodiment, the file name of the file storing the various event sequence data is assumed to follow the following rules.
(A) File name of the file storing the event sequence data BS: “BS ####. Ivt” (where “#” is an arbitrary natural number, and so on).
(B) File name of the file storing the event sequence data BI: “BI ####. Ivt”.
(C) File name of the file storing the event sequence data CI: “CI ####. Ivt”.
(D) File name of the file that stores the event sequence data DL: “DL ####. Ivt”.
(E) File name of the file that stores the event sequence data ME: “ME ####. Ivt”.
(F) File name of the file storing the event sequence data CW: “CW ####. Ivt”.

  FIG. 5 illustrates the contents of the event sequence data BS. As described above, the event sequence data includes a plurality of event data with timing. The timing data included in the event data with timing indicates the execution timing of the process by the elapsed time from the start timing of each scene (hereinafter referred to as “scene”) included in the story. This is the same for any kind of event sequence data.

  On the other hand, the content of event data included in event data with timing differs depending on the type of event sequence data. Hereinafter, some of them will be exemplified, but it goes without saying that the types of event data are not limited to those described below.

  The event sequence data BS shown in FIG. 5 instructs the background music and the performance of the background sound by instructing the sound generation of the musical sound, the instruction to reproduce the sound data, and the instruction to add the effect to the sound generated by those instructions. . For example, the event data [1ch: note on: D4: 74] indicates that a musical tone having a tone pitch D4 assigned to the first channel is to be generated with a tone strength of 74. Here, the channel is an individual performance part included in the music. The event data [1ch: delay: 0.5 seconds] instructs to add a delay effect of 0.5 seconds to the sound of the first channel. In addition, event data [3ch: playback: SS0514. wav] instructs to reproduce the sound data stored in the sound data file “SS0514.wav” as the third channel.

  FIG. 6 illustrates the contents of the event sequence data BI. In the present embodiment, the background image does not mean a general single-layer background image, but means a composite of multiple-layer images indicating images other than the character image. That is, the event sequence data BI is data for instructing display of an image obtained by sequentially superimposing images arranged in a plurality of layers such as a background layer, a first layer, a second layer,. .

  For example, event data [background layer: display: BP0815. pic] instructs the background layer to perform display according to the still image data stored in the still image data file “BP0815.pic”. In addition, the event data [first layer: display window: (10, 20) to (120, 180)] includes the coordinates (10, 20) and coordinates (120, 180) on the screen as diagonal lines on the first layer. It is instructing to open a rectangular display window as an end point. When the display window is opened, the image assigned to the layer is displayed only inside the display window. Since the outside of the display window is transmissive display, an image of a lower layer is displayed in that portion. Further, the event data [first layer: fade-in: 10 seconds] instructs to perform a fade-in process for 10 seconds on the image specified to be displayed on the display window of the first layer. Also, the event data [first layer: display: BM0411. mov] instructs the display window of the first layer to sequentially display images obtained by reproducing the moving image data stored in the moving image data file “BM0411.mov”.

  Also, event data [second layer: display: monkey. pic] instructs the display window of the second layer to perform display according to the still image data stored in the still image data file “Saru.pic”. Here, “Saru.pic” is a file for storing still image data of the character “Saru”, and event data [second layer: display: mon. pic] is usually event data for instructing the image display of the character “monkey”. However, when the character “monkey” is the character selected by the user, the video generation system 1 uses the event data [second layer: display: monkey. pic], instead of using still image data stored in the still image data file “Saru.pic”, user still image data obtained by imaging processing may be used. Details thereof will be described later.

  FIG. 7 illustrates the contents of the event sequence data CI. For example, event data [Saru: CR0486. mod] instructs to use modeling data stored in the modeling data file “CR0486.mod” as data indicating the image of the character “monkey”. Modeling data is a set of part data indicating the shape of a solid (hereinafter referred to as “parts”) in a three-dimensional virtual space, the color of the parts, the connection state with adjacent parts, and the like. For example, the modeling data indicating the image of the character “Saru” is composed of part data indicating each of the body, head, neck, upper arm, lower arm, hand, waist, thigh, knee and foot of “Saru”. For example, the part data indicating the head includes data indicating that the lower center of the head indicated by the part data is point-connected to the upper center of the neck indicated by the other part data. According to such part data, the images of the individual parts are combined with each other in the three-dimensional virtual space, so that the three-dimensional image of the character is specified as a whole.

  Furthermore, for example, the head part data is composed of a set of part data indicating small parts such as eyes, eyebrows, nose, lips, and ears. As described above, as a result of the hierarchical construction of the part data, the modeling data can express a complex motion of the character image.

  In order to display a 3D image of a character on a 2D screen according to modeling data, it is necessary to determine the overall size of the character and the position and direction of each part in the 3D virtual space. The event data [Saru: Size 5] illustrated in FIG. 7 designates the size indicated by “5” among the sizes indicated by “1” to “9” as the overall size of the character. The event data [Saru: body position (15, 43, 23)] arranges a predetermined representative point of the character body at the position indicated by the coordinates (15, 43, 23) in the three-dimensional virtual space. Instructed to do. Furthermore, the event data [Saru: body direction (20 degrees, 5 degrees, 10 degrees)] rotates the body of the character, for example, 20 degrees clockwise as viewed from above with the vertical direction in the three-dimensional virtual space as an axis. The body is tilted so that the front direction of the body is directed upward by 5 degrees from the horizontal plane, and is further instructed to rotate clockwise by 10 degrees when viewed from the front with the front direction of the body as an axis.

  Event data [fourth layer: monkey: display] is an image of the character “monkey” in the three-dimensional virtual space whose size, position, and direction are determined as described above on the fourth layer of the two-dimensional screen. The display is instructed.

  Event data [monkey: upper arm direction (30 degrees, 45 degrees, 18 degrees)] indicates that the direction of the upper arm of the character 1 is changed to the direction indicated by (30 degrees, 45 degrees, 18 degrees). The meaning of these angles is the same as that described for the direction of the body above, but the axis of rotation of the upper arm according to each angle is determined by the direction of the body to which the upper arm is coupled and the position of the coupling point. Is done. That is, the event data [monkey: upper arm direction (30 degrees, 45 degrees, 18 degrees)] indicates a relative direction of the upper arm with respect to the body. As described above, the event sequence data CI indicates an image of a character having movement as a whole by instructing the position and direction of each part at each timing.

  FIG. 8 illustrates the contents of the event sequence data DL. For example, the event data [acting guidance: “Please advance slowly from the back center”] is a character string “Please advance slowly from the back center” in the display area of the performance instruction in the predetermined screen. The display is instructed. In the present embodiment, “acting guidance” refers to various pieces of information related to the user's motion, such as not only guidance regarding the content of the performance such as “Please perform with a surprised feeling” but also operational instructions accompanying the performance. In addition, the event data [line: “Saru: Ah, good morning”] displays the character string “Saru: Ah, good morning” in a predetermined display area of the screen. I am instructing.

  Further, for example, event data [reproduction: GS 0441. wav] instructs reproduction of sound data stored in the sound data file “GS0441.wav”. In the present embodiment, the sound data stored in the sound data file having a file name beginning with “GS” is sound data indicating the voice of the performance instruction. That is, event data [reproduction: GS 0441. wav] instructs the reproduction of the voice of the performance instruction “Please advance slowly from the rear center”. On the other hand, the sound data stored in the sound data file having a file name starting with “DS” is sound data indicating speech of speech. That is, for example, event data “reproduction: DS0796.wav” instructs reproduction of voice when a voice actor or the like utters the line “Oh, good morning”.

  FIG. 9 illustrates the contents of the event sequence data ME. For example, event data [reproduction: ES0971. wav] instructs reproduction of sound data stored in the sound data file “ES0971.wav”, for example, sound data indicating sound effects such as bird's chirping. The event data [Saru: Pan 2] arranges the sound image of the dialogue of the character “Saru” at the position indicated by “2” among the left and right positions indicated by “1” to “9”. I am instructing.

  FIG. 10 illustrates the contents of the event sequence data CW. For example, the event data [camera 1: source: right hand] instructs to allocate a camera arranged on the right hand toward the front, that is, the video camera 15b, as the first camera (camera 1). Event data [Camera 1: Position 2] moves (initially arranges) the video camera 15b assigned to the camera 1 to the position indicated by "2" among the positions indicated by "1" to "9". Is instructing. Further, the event data [Camera 1: Display: Third Layer] instructs to display image data obtained by the imaging process of the video camera 15b assigned to the camera 1 on the third layer on the screen.

  In the event data [illumination 3: yellow: 0 → 6: 7 seconds], the color of the third illumination (illumination 3) is set to yellow, and the brightness level of the illumination is changed from “0” to “6”. It is instructed to change over a second. In the present embodiment, the illuminations 1 to 6 in the event sequence data CW correspond to the illumination devices 17a to 17f, respectively. The event data [Camera 1: Movement: → 5: 3 seconds] indicates that the position of the video camera 15b assigned to the camera 1 is moved from the current position to the position indicated by “5” over 3 seconds. is doing. Furthermore, event data [Camera 1: Zoom: → 7: 5 seconds] changes the zoom magnification of the video camera 15b assigned to the camera 1 from the current magnification to the magnification indicated by “7” over 5 seconds. Is instructing.

  As described above, the event sequence data CW instructs various operations such as changing the position and direction of the video camera 15 and the zoom magnification, and also instructs various operations of the illumination device 17. These instructions by the event sequence data CW simultaneously mean instructions for correction processing for the background image and the character image. For example, the event data instructing the lighting device 17 to change the color simultaneously instructs color correction in the background image and the character image. Similarly, the event data for instructing the lighting device 17 to change the brightness simultaneously instructs the brightness correction in the background image and the character image.

  In addition, the event data for instructing the change of the position and direction of the video camera 15 simultaneously instructs a process of transforming the background image to show the same image viewed from different viewpoints. In addition, the event data instructing to change the zoom magnification and the focus level of the video camera 15 simultaneously instruct to apply an enlargement ratio and clearness changing process to the background image and the character image. At this time, for example, the event data that increases the focus of the video camera 15 weakens the focus of the background image located deeper in the depth direction than the imaging target of the video camera 15 and at the same time the imaging target of the video camera 15 and the depth direction. Means to increase the focus of the character image arranged at the same position. The above is various event sequence data included in the event sequence data group.

  The source data group included in the story data 1010 (see FIG. 3) is still image data, moving image data, sound data, and modeling data (hereinafter collectively referred to as reproduction data) that are instructed to be reproduced by the various event sequence data described above. It is a set of various files that store “source data”.

  Arrangement data included in the story data 1010 is data indicating a file name of event sequence data to be used in the story. In the present embodiment, after the user selects a favorite one from a plurality of stories, the background music, background image, dialogue, etc. used in the selected story are further provided with a plurality of dramas (hereinafter referred to as “arrange”). Each can be selected from the following. Therefore, the arrangement data includes a plurality of selectable event sequence data file names for each type of event sequence data.

  FIG. 11 is a diagram schematically showing the contents of the arrangement data with the time axis in the story in the horizontal direction. The vertical lines in FIG. 11 indicate the breaks of each scene included in the story, and the data such as [0h03m17s] shown at the top of each vertical line is the timing of each break when the start timing of the entire story is used as a reference. Is shown. “Scene 1: Encounter” or the like shown between the data indicating the timings is data indicating the scene name.

  In FIG. 11, each data row indicates data corresponding to each type of event sequence data. For example, the data row indicated by “BS” is data indicating event sequence data selectable by the user with respect to the event sequence data BS. Each data row includes a plurality of data in which the arrangement name is associated with the file name of the event sequence data for each scene. For example, the arrangement data illustrated in FIG. 11 is associated with “BS0814.ivt” in which the user is associated with the arrangement name “fresh” and the arrangement name “nostalgic” as the event sequence data BS used in the scene 1. This indicates that any one of “BS0571.ivt” can be selected.

  In FIG. 11, for example, for event sequence data BS, combinations of arrangement names that differ for each scene are shown. Therefore, the user can select a different arrangement for the event sequence data BS for each scene. On the other hand, for the event sequence data BI, for example, the same arrangement name combination is shown for each scene. Therefore, the user selects one arrangement for the event sequence data BI throughout the entire story.

  In FIG. 11, for example, as the event sequence data ME, a plurality of arrangement names are shown for each arrangement name that can be selected for the event sequence data DL. Therefore, the combination of arrangements that can be selected for the event sequence data ME changes depending on which arrangement the user has selected for the event sequence data DL. As will be described later, when the user selects a favorite arrangement name for each of the various event sequence data, event sequence data associated with the selected arrangement name is specified, and the event sequence thus specified is specified. Data will be used in the video generation system 1. The above is arrangement data.

  Returning to FIG. 2, the description of the configuration of the video generation apparatus 10 will be continued. The input / output I / F 102 includes a storage unit 101 and a control unit 103 included in the video generation device 10, a controller 11, a microphone 12, an amplifier 13, a video camera 15, a camera moving device 16, a lighting device 17, a display 18, and ultrasonic reception. This is an interface for inputting / outputting analog signals and digital signals between the device 19, the media drive 20 and the Internet 21. The input / output I / F 102 includes an A / D (Analog to Digital) converter and a D / A (Digital to Analog) converter, and performs conversion between an analog signal and a digital signal as necessary.

  In accordance with the story data 1010, the control unit 103 reproduces background sound, sound effects, speech model speech, and acting instruction speech, and performs various processes on the reproduced sound and sound data indicating the sound picked up by the microphone 12. The sound data obtained as a result is output to the amplifier 13. In addition, the control unit 103 performs operation control of the video camera 15, the camera moving device 16, and the illumination device 17 in accordance with the story data 1010. Further, the control unit 103 generates image data indicating a background image and a character image according to the story data 1010, and performs various types of processing on the generated image data and image data indicating an image captured by the video camera 15, The resulting image data is output to the display 18. Further, the control unit 103 causes the storage unit 101 to store sound data and image data obtained by performing the processing as video data.

  The configuration unit of the control unit 103 includes a sound data block 201 related to sound data processing, an image data block 202 related to image data processing, an operation control block 203 related to operation control of the video camera 15, the camera moving device 16, and the illumination device 17, an event Each position of the microphone 12 is calculated according to a signal input from the ultrasonic data receiver 19 and an event data reproduction unit 204 that transmits event data to the sound data block 201, the image data block 202, and the operation control block 203 according to the sequence data. The microphone position calculation unit 205 and the charging data processing unit 206 that performs processing related to charging for the user are classified. The control unit 103 includes an oscillator (not shown), and the processing of the components of the control unit 103 is synchronized based on a clock signal generated by the oscillator.

  The sound data block 201 includes a sound source unit 2011 that outputs sound data indicating various musical sounds according to event data transmitted from the event data reproducing unit 204, and a source of the story data 1010 according to event data transmitted from the event data reproducing unit 204. A sound data reproduction unit 2012 that reproduces sound data included in the data group (see FIG. 3) is provided. The sound data block 201 includes sound data indicating the sound picked up by the microphone 12 and sound data transmitted from the sound source unit 2011 and the sound data reproducing unit 2012 according to the event data transmitted from the event data reproducing unit 204. A sound data processing unit 2013 is provided for outputting to the amplifier 13 sound data obtained by performing processing such as various effect processing on each of them and then mixing them.

  The image data block 202 includes a moving image data reproducing unit 2021 that reproduces moving image data included in the source data group of the story data 1010 according to the event data transmitted from the event data reproducing unit 204, and an event transmitted from the event data reproducing unit 204. Character data generating unit 2022 that sequentially generates image data indicating a character display screen for dialogue and acting guidance according to the data, and modeling data included in the source data group of story data 1010 according to the event data transmitted from event data reproducing unit 204 A rendering processing unit 2023 that sequentially generates image data indicating a character image based on the image data and a trimming processing unit 2024 that selects only a specific portion from the image data input from the video camera 15 are provided.

  Image data generation processing based on modeling data performed by the rendering processing unit 2023 is generally called rendering processing. In rendering processing, image data representing an image seen from a viewpoint at a predetermined position when light is emitted from one or more light sources to a part having a shape and material indicated by modeling data in a three-dimensional virtual space is simulated. Generated by calculation. The rendering processing unit 2023 transmits the image data generated by such rendering processing to the image data processing processing unit 2025.

  The trimming processing unit 2024 first performs a process of selecting a user's image portion from image data input from the video camera 15. Specifically, the trimming processing unit 2024 determines that the data of the outer peripheral position among the image data input from the video camera 15 is data indicating the color of the wall of the booth (see FIG. 1), and the data and the data are included in the data. A range in which approximate data continues from the outer peripheral portion of the image toward the center is specified. The area so identified is the background area that represents the booth wall. The trimming processing unit 2024 deletes the data of the background area, for example. As a result, image data (hereinafter referred to as “user image data”) indicating an image obtained by extracting only the image portion of the user is generated.

  The trimming processing unit 2024 further performs a process of selecting the image portion of the marker from the user image data. The marker is a bracelet, anklet, or the like of a specific color that the user wears on a limb or the like in order to cause the video generation device 10 to specify the position of each part in the user's image. The marker is coated with, for example, fluorescent color phosphorescent paint so that the video generation device 10 is not confused with the user's clothes or the like and can be recognized even when the booth is dark. In addition, for example, the right wrist marker is yellow and the left wrist marker is pink so that the video generation apparatus 10 can recognize which marker is attached to the user's limb or the like. The color is defined. The trimming processing unit 2024 extracts data indicating the same or approximate color as the marker color registered in advance from the user image data. The data thus extracted is image data indicating a marker portion (hereinafter referred to as “marker image data”).

  The image data block 202 further includes an image data processing unit 2025. The image data processing unit 2025 follows the event data transmitted from the event data reproduction unit 204, the image data transmitted from each of the moving image data reproduction unit 2021, the character screen generation unit 2022, and the rendering processing unit 2023, and the story data 1010. After processing various effects such as effect processing on the still image data included in the source data group, image data obtained by combining them is transmitted to the display 18. The image composition process includes, for example, a process of superimposing another image on an image and a process of compositing both images so that another image can be seen through the image.

  The operation control block 203 includes a camera operation control unit 2031, a movement operation control unit 2032, and a movement operation control unit 2032 that perform operation control of the video camera 15, the camera moving device 16, and the lighting device 17 in accordance with event data transmitted from the event data reproducing unit 204. An illumination operation control unit 2033 is provided.

  The event data reproducing unit 204 uses the data corresponding to the arrangement name selected by the user among the event sequence data indicated by the arrangement data included in the story data 1010, and is indicated by the timing data included in the event sequence data. At the timing, each event data is transmitted to each component as follows according to the contents.

(A) Event data instructing sound generation included in event sequence data BS → sound source unit 2011.
(B) Event data → sound data reproduction unit 2012 for instructing reproduction of sound data included in the event sequence data BS, event sequence data DL, and event sequence data ME.
(C) Event data → sound data processing unit 2013 for instructing effect processing for sounds included in the event sequence data BS and the event sequence data ME.
(D) Event data → video data playback unit 2021 that instructs playback of video data included in the event sequence data BI.
(E) Event data → character screen generation unit 2022 for instructing display of a character string included in the event sequence data DL.
(F) Event sequence data instructing designation of modeling data included in the event sequence data CI and the position and direction of each part data → rendering processing unit 2023.
(G) Event sequence data BI, event data related to layer designation included in event sequence data CI, event data instructing still image display included in event sequence data BI, image included in event sequence data CI and event sequence data CW Event data for instructing effect processing, camera operation control, camera position, and illumination device operation control for image data processing unit 2025.
(H) Event data indicating the camera operation control and camera position included in the event sequence data CW → the rendering processing unit 2023.
(I) Event data instructing camera operation control included in the event sequence data CW → camera operation control unit 2031.
(J) Event data indicating the camera position included in the event sequence data CW → the movement operation control unit 2032.
(K) Event data instructing lighting device operation control included in the event sequence data CW → lighting operation control unit 2033.

  As described above, the event data instructing the camera operation control, the camera position, and the illumination device operation control included in the event sequence data CW is transmitted to the camera operation control unit 2031, the movement operation control unit 2032, and the illumination operation control unit 2033, respectively. At the same time, it is also transmitted to the image data processing unit 2025. This is because the image data processing unit 2025 applies a size to the background image or character image according to the zoom magnification of the camera, the focus strength, the camera position, the camera direction, the illumination color, the illumination intensity, etc. This is to perform change processing, blur processing, deformation processing, color adjustment processing, brightness adjustment processing, and the like. Further, the event data instructing the camera operation control and the camera position included in the event sequence data CW is also transmitted to the rendering processing unit 2023. In this case, the rendering processing unit 2023 identifies which part of the image data received from the trimming processing unit 2024 represents an image of which user in light of the zoom magnification, camera position, and camera direction of the camera. Because.

  The microphone position calculation unit 205 receives sound data indicating ultrasonic waves from each of the ultrasonic receivers 19a to 19d disposed above the booth (see FIG. 1). The ultrasonic waves picked up by the ultrasonic receivers 19a to 19d are ultrasonic waves emitted intermittently from the ultrasonic generator 121a and the ultrasonic generator 121b provided in each of the microphone 12a and the microphone 12b. As described above, the ultrasonic waves emitted from each of the ultrasonic generator 121a and the ultrasonic generator 121b have different frequencies from each other, and the microphone position calculation unit 205 receives 2 from each of the ultrasonic receivers 19a to 19d. Sound data indicating a sound in which ultrasonic waves of two different frequencies are mixed is received. Therefore, the microphone position calculation unit 205 filters the sound data received from the ultrasonic receiver 19 with a band-pass filter that passes only the components of the frequency band that includes that frequency and does not include other frequencies for each of the two frequencies. . The sound data indicating the sound obtained by such filtering processing is data indicating the ultrasonic wave emitted from the ultrasonic generator 121a and data indicating the ultrasonic wave emitted from the ultrasonic generator 121b, respectively.

The microphone position calculation unit 205 performs the above filtering process on the sound data received from each of the ultrasonic receivers 19a to 19d, thereby generating eight sets of data as follows.
(A) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121a and received by the ultrasonic receiver 19a (referred to as “ultrasonic data aa”).
(B) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121a and received by the ultrasonic receiver 19b (referred to as “ultrasonic data ab”).
(C) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121a and received by the ultrasonic receiver 19c (referred to as “ultrasonic data ac”).
(D) Data indicating an ultrasonic wave emitted from the ultrasonic generator 121a and received by the ultrasonic receiver 19d (referred to as “ultrasonic data ad”).
(E) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121b and received by the ultrasonic receiver 19a (referred to as “ultrasonic data ba”).
(F) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121b and received by the ultrasonic receiver 19b (referred to as “ultrasonic data bb”).
(G) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121b and received by the ultrasonic receiver 19c (referred to as “ultrasonic data bc”).
(H) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121b and received by the ultrasonic receiver 19d (referred to as “ultrasonic data bd”).

  The ultrasonic data aa, ab, ac, and ad are data indicating the same ultrasonic wave, but the distance between the ultrasonic generator 121a provided in the microphone 12a and each of the ultrasonic receivers 19a to 19d is different. The timing at which the microphone position calculation unit 205 receives them is different. The same applies to the ultrasonic data ba, bb, bc and bd. The microphone position calculation unit 205 calculates the positions of the microphones 12a and 12b in the booth based on the timing difference. Since the calculation method of the positions of the microphones 12a and b in the booth is the same, the calculation method will be exemplified below by taking the case of calculating the position of the microphone 12a as an example.

FIG. 12 is a view of the booth as viewed from above. In FIG. 12, the positions of the ultrasonic receivers 19a to 19d are point A (0, 0, r), point B (0, q, r), point C (p, q, r) and point D ( p, 0, r). Further, the position of the microphone 12a is shown as a point X (x, y, z). Here, p, q, and r are constants (meters) indicating the width, depth, and height of the booth. On the other hand, x, y and z are variables (meters) indicating the coordinates of the microphone 12a. Here, the following equation holds. Where t _A , t _B , t _C and t _D indicate the time (seconds) required for the sound emitted from the point X to reach each of the points A to D, and S propagates the sound in the air. Indicates the speed (meter / second).

By the way, as already described, the ultrasonic generator 121 does not emit ultrasonic waves for 1 second after continuing to emit ultrasonic waves for 0.5 seconds, for example. Therefore, the microphone position calculation unit 205 continues for about 0.5 second from each of the ultrasonic receivers 19a to 19d after the period in which the ultrasonic data is not received from any of the ultrasonic receivers 19a to 19d continues for about 1 second. The ultrasonic data aa to ad are received. The ultrasonic data received as such is data indicating the same ultrasonic wave emitted from the ultrasonic generator 121a. For example, the microphone position calculation unit 205 sets the rising timing of the value of the ultrasonic data aa, that is, the reception timing of the leading data that changes across the predetermined value in which the value of the ultrasonic data aa exceeds the level of the noise component from the bottom to the top. a reference to calculate the time until the rise timing of each of the ultrasound data ab~ad (seconds) as t _1, t ₂ and t _3.

For example, when the point X is at the position shown in FIG. 12, the following equation holds.

  By substituting Equations 1 to 4 into Equations 5 to 7, a ternary simultaneous equation consisting of three equations is obtained. Here, since x, y, and z have positive values, the microphone position calculation unit 205 can uniquely calculate x, y, and z by solving the simultaneous equations. The above is the method by which the microphone position calculation unit 205 calculates the position of the microphone 12.

  However, the method by which the microphone position calculation unit 205 calculates the position of the microphone 12 is not limited to the above. For example, when the position of the microphone 12 in the height direction is small enough to be ignored, only the position of the microphone 12 in the two-dimensional plane, that is, x and y may be obtained. In that case, since there are two variables, the number of ultrasonic receivers 19 may be three or more. Further, instead of providing the ultrasonic generator 121 in the microphone 12, the ultrasonic generator 121 is disposed at the position of the ultrasonic receivers 19 a to 19 d in FIG. 12, the ultrasonic receiver 19 is provided in the microphone 12, and the microphone position The calculation unit 205 may be configured to receive ultrasonic data from the microphone 12 so that the microphone position calculation unit 205 calculates the position of the microphone 12 by a calculation method similar to the above.

The microphone position calculation unit 205 has coordinate data X _a (x _a , y _a , z _a ) and X _b (x _b , y _b , z _b ) indicating the positions of the microphones 12 a and b calculated as described above. (Hereinafter referred to as “microphone coordinate data”) are sequentially transmitted to the rendering processing unit 2023. The microphone coordinate data is used in the rendering processing unit 2023 to specify which part of the image data transmitted from the trimming processing unit 2024 is data indicating which user.

  The billing data processing unit 206 calculates a billing amount for use of the video generation system 1 and performs processing for requesting settlement of the calculated billing amount based on event data transmission processing by the event data reproducing unit 204. The specific contents will be described together in the following operation description.

[1.2. Operation of the video generation system]
Using the video generation system 1 having the above configuration, the user can shoot a short movie in which he / she participates as a character and obtain the result as video data. Hereinafter, the operation of the video generation system 1 will be described. In the following description, it is assumed that two users use the video generation system 1 at the same time. The number of users who can use the video generation system 1 at the same time is shown in the number of microphones 12 included in the video generation system 1 and the management data of the story data 1010. Of the number of characters to be selected, the smaller one can be selected freely.

  Prior to using the video generation system 1, the user wears the microphone 12 on his / her chest. In addition, the user wears markers of a designated color on both wrists and both ankles. Thereafter, the user uses the controller 11 to instruct the video generation device 10 to start the operation.

[1.2.1. Selection operation by the user]
FIGS. 13, 14, and 15 illustrate screens displayed on the display 18 when the user uses the video generation system 1. When the user instructs the video generation device 10 to start an operation, the control unit 103 of the video generation device 10 is first included in each of the story data 1010 (see FIG. 3) stored in the storage unit 101. The story data name is read from the management data (see FIG. 4), and the story selection screen (FIG. 13 (a)) is displayed on the display 18.

  When the user gives a story selection instruction using the controller 11 on the story selection screen, the video generation device 10 reads the arrangement data included in the story data 1010 selected by the user, and each type of event sequence data is read out. The arrangement selection screens (FIGS. 13B to 13G) are sequentially displayed.

  When the user performs a selection operation using the controller 11 on the story and arrangement selection screen described above, the control unit 103 selects, according to the selection operation, selected arrangement data indicating the selected story data name and arrangement name, for example, “Saru Crab Battle: Nostalgic: Fantasy ...” is generated, and the generated selection arrangement data is stored in the storage unit 101.

  Subsequently, the video generation device 10 reads the character attribute included in the management data, and causes the display 18 to display a character selection screen (FIG. 14A). On the character selection screen, the user selects the character of the person wearing the microphone 12a and the character of the person wearing the microphone 12b as the characters corresponding to the microphone a and the microphone b, respectively. Further, the user selects a display method (hereinafter referred to as “character mode”) for displaying each character. The character mode includes “real” using the user's whole body image as the character image, “costume” using the character image in which the user's face image is embedded and a change corresponding to the user's movement, and the user One of “images” that uses a character image that has been changed in accordance with the user's movement is selected without using the above image.

  When the user performs a selection operation of a character corresponding to each microphone using the controller 11 on the character selection screen, the control unit 103 responds to the selection operation with character correspondence data indicating the correspondence between the microphone and the character, for example, “ “Mike a: Monkey” and “Mike b: Mother crab” are generated, and the generated character correspondence data is stored in the storage unit 101. Similarly, when the user performs a character mode selection operation using the controller 11 on the character selection screen, the control unit 103 generates character mode data indicating the selected character mode in accordance with the selection operation. The generated character mode data is stored in the storage unit 101.

  Subsequently, the video generation apparatus 10 displays a billing method selection screen (FIG. 14B) on the display 18. The billing method selection screen includes a list of billing amount calculation methods. As a method of calculating the charge amount, for example, “300 yen / scene” for charging according to the number of scenes where event data is played back, “150 yen / 5 minutes” for charging according to the playback time of event data, basic Various things such as “1500 yen + 300 yen / additional scenes” that charge according to the combination of the charge and the number of scenes for which event data is to be played back are conceivable.

  When the user performs a charging method selection operation using the controller 11 on the charging method selection screen, the control unit 103 generates and generates charging method data indicating the selected charging method in accordance with the selection operation. Charge method data is stored in the storage unit 101.

  Subsequently, the video generation apparatus 10 causes the display 18 to display an event data reproduction instruction screen (FIG. 14C). The playback instruction screen includes a scene name and a list indicating completion or incompleteness of take (shooting) of the scene. The user selects a scene for which event data is desired to be reproduced on the reproduction instruction screen. On the playback instruction screen, the user can select either the “practice mode” or the “production mode” (hereinafter referred to as “playback mode”). The “practice mode” is a playback mode in which a voice of a model for acting guidance and dialogue is reproduced, and a moving image of a character image according to the characters of the acting guidance and dialogue and the event sequence data CI is displayed. On the other hand, the “production mode” means that the voice of the script related to the character selected by the user is not reproduced, the user's voice is processed and pronounced, and the character of the This is a playback mode in which a user image or a character image whose movement changes according to the user's movement is displayed.

  When the user selects any scene name using the controller 11 on the playback instruction screen, the control unit 103 generates selected scene data indicating the selected scene name according to the selection operation, and generates the selected scene. Data is stored in the storage unit 101. Similarly, when the user selects any playback mode button using the controller 11, the control unit 103 generates playback mode data indicating the selected playback mode in accordance with the selection operation, and generates the generated playback mode. Data is stored in the storage unit 101.

  When the user performs an operation of selecting either “practice mode” or “production mode” using the controller 11, first, the billing data processing unit 206 of the video generation apparatus 10 is described later according to the operation. In order to use it for calculating the billing amount, reproduction start time data indicating the time at that time is generated, and the generated reproduction start time data is stored in the storage unit 101. Also, the billing data processing unit 206 generates a playback number counter for recording the number of reproduced scenes in the storage unit 101 and sets “1” in the playback number counter to be used for calculating the billing amount. .

  Subsequently, the event data reproduction unit 204 arranges data included in the story data 1010 (see FIG. 3) corresponding to the story name selected by the user according to the selected arrangement data and the selected scene data stored in the storage unit 101. Based on (see FIG. 11), various event sequence data corresponding to the arrangement name and scene name selected by the user are specified.

  For example, when the arrangement data illustrated in FIG. 11 is followed, “scene 1: encounter” is selected by the user, the background sound arrangement name “fresh”, the background image arrangement name “dramatic”, and the character image arrangement. When the name “fancy”, the dialog arrangement name “stage play style”, the microphone effect arrangement name “unreal”, and the camera work arrangement name “technical” are selected, the event data playback unit 204 sets “BS0814. “ivt”, “BI0795.ivt”, “CI0801.ivt”, “DL0685.ivt”, “ME0150.ivt”, and “CW0401.ivt” are specified as the event sequence data files to be used.

  When the event data reproducing unit 204 identifies the event sequence data file as described above, the event data reproducing unit 204 reproduces the event data according to the event sequence data stored in the identified file. That is, the event data reproducing unit 204 transmits the event data included in each event sequence data to the component unit corresponding to the type of the event data at the timing indicated by the associated timing data. Each component performs various processes according to the event data received from the event data reproducing unit 204. However, the event data playback process by the event data playback unit 204 and the operation of each component when receiving the event data are partially different depending on whether the playback mode is “practice mode” or “production mode”. Therefore, the operation of the video generation system 1 when event data is reproduced for each of the reproduction modes will be described below.

[1.2.2. Event data playback (practice mode)]
First, when playback mode data “practice mode” is stored in the storage unit 101, the event data playback unit 204 transmits event data to the sound data block 201 as follows.
(A) Event data instructing sound generation included in event sequence data BS → sound source unit 2011.
(B) Event data → sound data reproduction unit 2012 for instructing reproduction of sound data included in the event sequence data BS, event sequence data DL, and event sequence data ME.
(C) Event data → sound data processing unit 2013 for instructing effect processing for sounds included in the event sequence data BS and the event sequence data ME.

  The sound source unit 2011 reproduces the musical sound constituting the background sound according to the event data of (a). In addition, the sound data playback unit 2012 performs playback of sounds other than the musical sounds constituting the background sound, playback of sound effects, playback of voices for acting instruction, and playback of model voices in accordance with the event data of (b) above. Do. When the event data instructed to be reproduced includes a channel number, the sound source unit 2011 and the sound data reproducing unit 2012 add the channel number to the reproduced sound data. In addition, when the character data is included in the event data instructed to be reproduced, the sound data reproducing unit 2012 adds the character name to the reproduced sound data. The sound data thus generated is transmitted from the sound source unit 2011 and the sound data reproduction unit 2012 to the sound data processing unit 2013.

  The sound data processing unit 2013 performs various types of processing such as adding a delay and changing pan on the sound data received from the sound source unit 2011 and the sound data reproducing unit 2012 according to the event data of (c). When the sound data processing unit 2013 receives event data including a channel number, the sound data processing unit 2013 performs processing according to the event data on the sound data to which the channel number is added. When the sound data processing unit 2013 receives event data including a character name, the sound data processing unit 2013 performs processing according to the event data on the sound data to which the character name is added. In accordance with the event data, sound data that has been processed as necessary is output from the sound data processing unit 2013 to the amplifier 13, amplified by the amplifier 13, converted into sound by the speaker 14, and pronounced.

In parallel with the above processing, the event data reproducing unit 204 transmits event data to the image data block 202 as follows.
(D) Event data → video data playback unit 2021 that instructs playback of video data included in the event sequence data BI.
(E) Event data → character screen generation unit 2022 for instructing display of a character string included in the event sequence data DL.
(F) Event sequence data instructing designation of modeling data included in the event sequence data CI and the position and direction of each part data → rendering processing unit 2023.
(G) Event sequence data BI, event data related to layer designation included in event sequence data CI, event data instructing still image display included in event sequence data BI, image included in event sequence data CI and event sequence data CW Event data for instructing effect processing, camera operation control, camera position, and illumination device operation control for image data processing unit 2025.

  In accordance with the event data of (d) above, the moving image data reproduction unit 2021 reproduces the image constituting the background image, adds the layer number included in the event data instructed to reproduce to the reproduced image data, It transmits to the image data processing unit 2025. In addition, the character screen generation unit 2022 generates image data indicating a display screen for acting instruction and dialogue characters according to the event data of (e), and transmits the generated image data to the image data processing unit 2025. In addition, the rendering processing unit 2023 generates image data indicating a character image according to the event data of (f) above, adds a character name included in the event data instructed to generate to the generated image data, Transmit to the data processing unit 2025.

  When the image data processing unit 2025 receives event data instructing still image display according to still image data included in the source data group of the story data 1010, the image data processing unit 2025 stores still image data instructed by the event data. Read from 101. In addition, when receiving event data instructing effect processing including a layer number, the image data processing unit 2025 performs processing according to the event data on the image data to which the layer number is added. In addition, when receiving event data instructing effect processing including a character name, the image data processing unit 2025 performs processing according to the event data on the image data to which the character name is added.

  Furthermore, when the image data processing unit 2025 receives event data instructing camera operation control, camera position, and lighting device operation control from the event data reproduction unit 204, the image data processing unit 2025 receives the image received from the moving image data reproduction unit 2021 and the rendering processing unit 2023. Each of the data and the still image data read from the storage unit 101 is subjected to a size change process, a blur process, a deformation process, a color adjustment process, a brightness adjustment process, and the like according to the contents of the event data. For example, the image data processing unit 2025 enlarges or reduces the size of each image data according to event data instructing to change the zoom magnification. At that time, the image data processing unit 2025 actually increases the enlargement / reduction ratio of the image data arranged in the front layer, for example, and decreases the enlargement / reduction ratio of the image data arranged in the back layer. A pseudo image is generated when the zoom magnification of the video camera is changed.

  Further, the image data processing unit 2025 performs a blurring process on each image data according to the event data instructing to change the focus strength. At that time, the image data processing unit 2025 actually changes the degree of blurring of the image data arranged in the front layer and the degree of blurring of the image data arranged in the back layer, for example, to actually make the video. A pseudo image is generated when the focus of the camera is changed. Further, the image data processing unit 2025 performs transformation processing on each image data in accordance with the event data instructing to change the camera position. At that time, the image data processing unit 2025 actually increases the deformation rate of the image data arranged in the front layer, for example, and reduces the deformation rate of the image data arranged in the back layer. A pseudo image is generated when the is moved.

  In addition, the image data processing unit 2025 performs color tone and brightness change processing on each image data in accordance with event data instructing to change the illumination color and brightness. At that time, the image data processing unit 2025 increases the degree of change in the color tone and brightness of the image data arranged in the previous layer, for example, and changes the color tone and brightness of the image data arranged in the back layer. By reducing the degree of, an image in the case where the color or brightness of the illumination is actually changed is generated in a pseudo manner.

  The image data processing unit 2025 superimposes the image data obtained by performing various types of processing as described above according to the order of the arranged layer numbers, and finally receives the image data received from the character screen generation unit 2022. Superimpose to compose one image data. The image data processing unit 2025 transmits the image data obtained by the synthesis to the display 18. The display 18 displays a screen including a background image, character image, performance instruction, and dialogue characters according to the received image data. FIG. 14D illustrates a screen displayed on the display 18 in the practice mode.

  While the event data is being reproduced by the event data reproducing unit 204 by the operation of the video generation system 1 as described above, the user practices the speech while listening to the speech of the speech, and the character displayed on the display 18 Practice the movement while watching the movement of the image. At that time, the user can devise his / her actions, facial expressions, dialogue rotation, etc. with reference to the voice of the performance instruction.

  When the event data playback unit 204 finishes playing the event data for one scene, the billing data processing unit 206 generates playback end time data indicating the current time, and stores the generated playback end time data in the storage unit 101. Let Thereafter, the control unit 103 displays the reproduction instruction screen as shown in FIG. 14C on the display 18 again.

[1.2.3. Event data playback (production mode)]
On the other hand, when the playback mode data “production mode” is stored in the storage unit 101, the operation performed by the video generation system 1 when reproducing the event data is common in many respects to that in the practice mode. It differs in the following points.

First, the event data reproducing unit 204 transmits the following event data to the operation control block 203 in addition to the event data transmitted in the practice mode.
(H) Event data indicating the camera operation control and camera position included in the event sequence data CW → the rendering processing unit 2023.
(I) Event data instructing camera operation control included in the event sequence data CW → camera operation control unit 2031.
(J) Event data indicating the camera position included in the event sequence data CW → the movement operation control unit 2032.
(K) Event data instructing lighting device operation control included in the event sequence data CW → lighting operation control unit 2033.

  The camera operation control unit 2031, the movement operation control unit 2032, and the illumination operation control unit 2033 are based on the event data (i), (j), and (k), respectively, and the video camera 15, the camera movement device 16, and the illumination device 17. Perform motion control. In accordance with the operation control, the video camera 15 changes the zoom magnification and the focus strength, the camera moving device 16 changes the position and direction of the video camera 15, and the illumination device 17 changes the color and brightness of the illumination. As a result, the video camera 15 can sequentially generate image data rich in change. The video camera 15 transmits the image data generated by the imaging process to the trimming processing unit 2024.

  The trimming processing unit 2024 performs a trimming process for extracting a user's image portion for each of the image data transmitted from the video cameras 15a to 15d to generate user image data. The trimming processing unit 2024 transmits user image data to the rendering processing unit 2023 together with data indicating which video camera 15 has generated the user image data. Further, the trimming processing unit 2024 performs a trimming process for extracting the image portion of the marker for each color of the marker worn by the user on each of both hands and feet for each of the generated user image data, and generates marker image data. . The trimming processing unit 2024 transmits the marker image data to the rendering processing unit 2023 together with data indicating which user image data is generated from the marker image data.

  On the other hand, the microphone position calculation unit 205 receives data indicating ultrasonic waves from each of the ultrasonic receivers 19a to 19d, and generates microphone coordinate data indicating the positions of the microphones 12a and 12b based on the received data. The microphone position calculation unit 205 transmits the generated microphone coordinate data to the rendering processing unit 2023 together with data indicating which of the microphones 12a and b corresponds.

  When the rendering processing unit 2023 receives user image data and marker image data from the trimming processing unit 2024 and microphone coordinate data from the microphone position calculation unit 205, the rendering processing unit 2023 receives the camera position and the camera position currently received from the event data reproduction unit 204. Based on the event data indicating the camera direction and the microphone coordinate data received from the microphone position calculation unit 205, the position of the user image data in the image data generated by the video camera 15, the position of the video camera 15, and the position of the microphone 12 Collation with the position of the user determined from the relationship is performed. The collation processing by the rendering processing unit 2023 specifies which user image data belongs to which user even when a plurality of users are captured by one video camera 15.

  Subsequently, the rendering processing unit 2023 calculates marker coordinate data indicating the coordinates of the representative point of the marker portion indicated by the marker image data, for example, the center of gravity. Each marker is identified by which color it is attached to which part of the user, and each marker image data has a user indicating the user image data used for its generation added, so that the rendering processing unit 2023 can specify whether the calculated marker coordinate data indicates the marker position attached to which part of which user.

  The rendering processing unit 2023 is selected by the user using the user image data associated with each user as described above and the marker coordinate data associated with each part of each user. For the character, the character image data generation process is performed as follows.

  When the character mode data “real” is stored in the storage unit 101, the rendering processing unit 2023 performs a trimming process on the character selected by the user instead of the image data of the character image generated by the rendering process according to the event data. The user image data transmitted from the unit 2024 is transmitted to the image data processing unit 2025 as it is. At that time, the rendering processing unit 2023 attaches the character name “Saru” to the user image data of the user wearing the microphone 12a and the microphone 12b according to the character correspondence data stored in the storage unit 101, for example. The character name “mother crab” is added to the user image data of the user who is being transmitted. As a result, the image data processing unit 2025 can specify in which layer the user image data received from the rendering processing unit 2023 should be arranged.

  When the character mode data “costume” is stored in the storage unit 101, the rendering processing unit 2023 relates to the character selected by the user, the microphone coordinate data received from the microphone position calculation unit 205, and the calculated marker coordinate data. The position and direction of each part of the character are calculated based on the above, and a rendering process is performed according to the position and direction calculated from the marker coordinate data instead of the position and direction of each part specified by the event data. In addition, the rendering processing unit 2023 extracts an elliptical portion positioned at the top from the user image data as image data of the user's face portion. The rendering processing unit 2023 replaces the image data of the face part of the character image generated by the rendering process with the image data of the user's face part. As a result, the rendering processing unit 2023 generates image data obtained by combining the image of the user's face with the character image in which the position and direction of each part changes according to the user's operation. The rendering processing unit 2023 transmits the image data generated as described above to the image data processing processing unit 2025 together with the character name.

  When the character mode data “image” is stored in the storage unit 101, the rendering processing unit 2023 performs substantially the same processing as that of the character mode data “costume” on the character selected by the user. However, the process of replacing the image data of the face portion of the character image generated by rendering with the image data of the user's face portion is not performed. On the other hand, the rendering processing unit 2023 is located in the upper part of the image data of the user's face part, and the elliptical part located side by side as the image data of the eye part and vertically below the face part. The oval part is extracted as image data of the lip part. The rendering processing unit 2023 calculates the position information of the face part of the eye part and the lip part thus extracted, and determines the position of the eye part and the lip part of the character image in the rendering process based on the position information. . As a result, the rendering processing unit 2023 generates image data of a character image in which the position and direction of each part changes according to the movement of the user's limbs, eyes, and lips. The rendering processing unit 2023 transmits the image data generated as described above to the image data processing processing unit 2025 together with the character name.

  Through the processing of the image data block 202 as described above, in the production mode, a screen using a user image or a character image that changes according to the user's operation is displayed on the display 18. FIG. 14E illustrates a screen displayed on the display 18 when the character mode “costume” is selected in the production mode.

  In the production mode, when the image data processing unit 2025 receives event data instructing display of a still image of the character selected by the user, the image data processing unit 2025 stores still image data instructed by the event data. Instead of reading from 101, one frame of image data transmitted from the rendering processing unit 2023 is used. Therefore, the image synthesized by the image data processing unit 2025 and displayed on the display 18 includes an image in which the user image and the user face image are incorporated in the character image according to the character mode selected by the user, Or a character image is included. When the user still image data or the character image corresponding to the user's movement is used, the video camera 15 captures the user image data corresponding to the image data for one frame used by the image data processing unit 2025. Depending on the timing, according to the event sequence data DL, for example, the voice of the performance instruction and the display of characters such as “Shoot a still image. 3, 2, 1, Yes!” Are performed to the user. You may do it.

  On the other hand, also in the sound data block 201, processing different from the practice mode is performed in the actual mode. First, the event data reproducing unit 204 does not transmit event data instructing reproduction of the model voice of the dialogue related to the character selected by the user to the sound data reproducing unit 2012 according to the character correspondence data stored in the storage unit 101. . As a result, the sound data playback unit 2012 plays back only the speech of characters not selected by the user.

  Further, the event data reproducing unit 204 does not transmit the event data instructing the reproduction of the performance instruction voice to the sound data reproducing unit 2012. As a result, the voice of the performance instruction is not played back by the sound data playback unit 2012.

  The sound data processing unit 2013 is not the sound data transmitted from the sound data reproducing unit 2012 as the sound data indicating the sound of the character selected by the user, but the sound indicating the sound picked up by the microphone 12. Use data. At that time, the sound data processing unit 2013 specifies which sound data received from which microphone 12 is used as sound data indicating the line of which character in accordance with the character correspondence data stored in the storage unit 101. As a result, the sound data processing unit 2013 can specify the sound data that is the target of the effect processing instructed by the event data.

  Through the processing of the sound data block 201 as described above, a sound using the user's voice is generated from the speaker 14 in the production mode.

  By the way, in the production mode, the sound data generated by the sound data processing unit 2013 is output to the amplifier 13 and is sequentially stored in the storage unit 101. Similarly, image data generated by the image data processing unit 2025 is transmitted to the display 18 and is sequentially stored in the storage unit 101. However, the image data stored in the storage unit 101 is data in which the performance instruction and dialogue character display screen data received by the image data processing unit 2025 from the character screen generation unit 2022 are not superimposed.

  When the event data playback unit 204 finishes playing the event data for one scene, the billing data processing unit 206 stores the time at that time in the storage unit 101 as playback end time data. In addition, the control unit 103 integrates the sound data and the image data sequentially stored in the storage unit 101, adds a file name such as “scene 1.mov” that can specify a scene, and stores it as video data. Thereafter, the control unit 103 displays the reproduction instruction screen as shown in FIG. 14C on the display 18 again. However, in this case, in the list of scene names included in the playback instruction screen, the take column corresponding to the scene for which video data has been generated is displayed as “completed”.

  On the playback instruction screen (FIG. 14C), the user freely selects a scene to be practiced or photographed, and repeats the playback of event data in the practice mode or the production mode as described above. As each scene is reproduced, the billing data processing unit 206 stores the reproduction start time data and the reproduction end time data in the storage unit 101 and adds 1 to the reproduction number counter of the storage unit 101. When the user completes the take for all scenes included in the selected story, the display 18 displays a notification screen for the completion of all takes as shown in FIG. When the user selects “play” on the notification screen indicating that all takes have been completed, the moving image data playback unit 2021 of the video generation device 10 sequentially plays back the video data of each scene according to the order of the scenes. As a result, the user can enjoy displaying the short movie on which the user's own image or the character image moving by his / her movement appears on the display 18.

  If the user does not like the short movie generated by the video generation system 1, the user selects “Return” to display the playback instruction screen (FIG. 14C) on the display 18, and the take has already been completed. It is also possible to re-shoot the scene. If the user likes the short movie produced by the video generation system 1, the storage medium such as a flash memory is loaded into the media drive 20, and then “Save” is selected to select the short movie. Video data can be stored on a storage medium. Further, after selecting “Send”, the user can transmit video data of a short movie to a desired communication terminal via the Internet 21 by designating an e-mail address or the like, for example. As a result, the user can repeatedly enjoy the short movie generated by the video generation system 1 at home or send it to a friend.

[1.2.4. Billing process]
When the user selects “END” on the notification screen for the completion of all takes (FIG. 14F), the video generation apparatus 10 performs a charging process for the use of the video generation system 1 by the user. First, in accordance with the user's “end” selection operation, the billing data processing unit 206 performs billing amount calculation processing according to the billing method data stored in the storage unit 101. For example, when the accounting method data is “150 yen / 5 minutes”, the accounting data processing unit 206 reproduces the event data by the video generation device 10 based on the reproduction start time data and the reproduction end time data stored in the storage unit 101. Is calculated by multiplying the value obtained by dividing the calculated time (minutes) by 5 by 1500.

  For example, when the accounting method data is “300 yen / scene”, the accounting data processing unit 206 multiplies the value of the reproduction counter stored in the storage unit 101 by 300 to calculate the accounting amount (yen). . These billing amount calculation methods are examples, and various other billing amount calculation methods are conceivable. In this embodiment, the accounting amount is calculated based on the playback time of the event data and the number of scenes where the playback of the event data is performed. However, for example, the operation from the start of the video generation system 1 to the end is performed. Various other information may be used for calculating the charge amount, such as using the entire time for calculating the charge amount or using the number of characters selected by the user for calculating the charge amount.

  Subsequently, the control unit 103 causes the display 18 to display a payment amount selection method selection screen shown in FIG. 15A using the charging amount calculated by the charging data processing unit 206. On the payment method selection screen, the user can select, for example, either payment by credit card or withdrawal from a bank account. When “credit card” or “bank account” is selected by the user, the control unit 103 displays the screen shown in FIG. 15B or FIG. 15C, and is necessary for settlement by credit card or withdrawal from the bank account. Prompt the user for information. When necessary information is input by the user and “OK” is selected, the control unit 103 connects to a credit card company or bank server via the Internet 21, for example, and uses the information input by the user. Thus, the server is instructed to perform a billing settlement process or a remittance process.

  In the present embodiment, the payment method of the charge amount is exemplified by a credit card and a bank account, but other payment methods such as cash settlement may be adopted. For example, in the video generation system 1, when it is possible to select payment by cash, the video generation system 1 includes a cash register connected to the video generation apparatus 10, and discharges change in response to a cash input to the cash register by the user. Do. Furthermore, the video generation system 1 may be provided with a printing device connected to the video generation device 10 so as to issue documents such as a receipt and a credit card usage certificate. When the control unit 103 finishes the settlement processing or remittance processing instruction to the server, the control unit 103 displays the screen shown in FIG. 15D on the display 18 and completes a series of operations.

  As described above, according to the video generation system 1, according to event sequence data prepared in advance, image data generated by appropriate processing using user image data captured in real time, and sound pickup in real time Video data composed of sound data generated by appropriate processing using the user's voice data is automatically generated. At that time, dialogue characters and character images that serve as examples of actions, voice production of acting guidance and display of characters, etc. are displayed, so that the user can easily participate in the generation of video data and the audience. Can generate video data that can be more entertaining.

  Further, according to the video generation system 1, the user selects video data corresponding to his / her preference and the atmosphere of the place by selecting one corresponding to a favorite arrangement from a plurality of event sequence data prepared in advance. Can be generated.

  Further, according to the video generation system 1, for example, processing appropriate to image data such as a background image or a character image according to a change in illumination color or brightness, such as a change in the position of a video camera or a zoom magnification. Therefore, video data that can more entertain the audience can be generated.

  By the way, the video generation apparatus 10 may be realized by dedicated hardware, or may be realized by causing a general-purpose computer to execute processing according to an application program. When the video generation device 10 is realized by a general-purpose computer, each component of the control unit 103 includes a CPU (Central Processing Unit) included in the general-purpose computer and a DSP (Digital Signal Processor) that operates under the control of the CPU. Are performed as functions of a general-purpose computer by performing processing in accordance with each module included in the above in parallel.

  In the above description, the event data reproducing unit 204 also transmits event data to be transmitted to the operation control block 203 to the image data block 202 to link the operations of the operation control block 203 and the image data block 202. Although the case where it implement | achieves was demonstrated, the event data reproduction | regeneration part 204 transmits the event data transmitted to the operation control block 203 also to the sound data block 201, for example, and the operation of the operation control block 203 and the sound data block 201 is You may make it implement | achieve interlocking. The same applies to the interlocking of the operations of the sound data block 201 and the image data block 202.

[2. Second Embodiment]
In the first embodiment described above, the video generation system 1 is configured by a combination of a video generation device 10 and various devices such as a microphone 12 connected to the video generation device 10. However, it is possible to freely change which of the constituent parts of the video generation system according to the present invention is combined and arranged in the same housing. FIG. 16 is a block diagram showing a configuration of a video generation system 2 in which the video generation system according to the present invention is realized by an arrangement different from that of the video generation system 1 shown in FIG. Hereinafter, the video generation system 2 will be described as a second embodiment of the present invention.

  However, since the configuration and operation of the video generation system 2 are similar to those of the video generation system 1 in many respects, only the difference between the video generation system 2 and the video generation system 1 will be described below. In FIG. 16, among the components of the video generation system 2, the same components as those of the video generation system 1 are denoted by the same reference numerals. In FIG. 16, for simplification of the drawing, an arrow indicating a detailed data flow between the components is omitted, and only one component of the same kind including a plurality of components is shown.

  The video generation system 2 includes a sound data processing device 31, an image data processing device 32, and a central management device 33 instead of the video generation device 10 of the video generation system 1. The sound data processing device 31 includes a control unit 1031 that includes components included in the sound data block 201 in the video generation system 1, a storage unit 1011, and an input / output I / F 1021. The image data processing device 32 includes a configuration unit included in the image data block 202 in the video generation system 1, a control unit 1032 including the ultrasonic receiver 19 and the microphone position calculation unit 205, a storage unit 1012, and an input / output I / F 1022. I have. The central management device 33 includes a control unit 1033 including an event data reproduction unit 204 and a billing data processing unit 206 in the video generation system 1, a storage unit 1013, and an input / output I / F 1023.

  The input / output I / Fs 1021 to 1023 are interfaces for inputting / outputting data to / from other devices in the same manner as the input / output I / F 102 of the video generation apparatus 10. Similar to the storage unit 101 of the video generation device 10, the storage units 1011 to 1013 store programs for instructing various processes by the control units 1031 to 1033 in advance, and temporarily store data generated by the control units 1031 to 1033. To remember. Further, for example, the storage unit 1013 of the central management device 33 stores a plurality of story data 1010 in advance.

  The video camera 15, the camera moving device 16, and the lighting device 17 in the video generation system 2 include a camera operation control unit 2031, a moving operation control unit 2032, and an illumination operation control unit 2033, respectively.

  In the video generation system 2, when the user instructs the reproduction of the event data using the controller 11, the central management device 33 follows the event sequence data specified by the user's selection operation, and according to the contents of the event data, the sound data processing device 31. Event data is transmitted to each of the image data processing device 32, the video camera 15, the camera moving device 16, and the lighting device 17. The central management device 33 transmits source data required for processing instructed by the event data to the sound data processing device 31 or the image data processing device 32 prior to or in parallel with the transmission of the event data. However, instead of the central management device 33 transmitting the source data, the sound data processing device 31 and the image data processing device 32 store the source data necessary for each processing in the storage unit 1011 and the storage unit 1012 in advance. Also good.

  The sound data processing device 31 performs sound data reproduction processing and processing processing according to the event data received from the central management device 33, and outputs the generated sound data to the amplifier 13. When the production mode is selected by the user, the sound data processing device 31 transmits the generated sound data to the central management device 33, and the central management device 33 stores the sound data transmitted from the sound data processing device 31. Store in the unit 1013.

  The image data processing device 32 performs image data reproduction processing and processing processing according to the event data received from the central management device 33, and transmits the generated image data to the display 18. When the production mode is selected by the user, the image data processing device 32 transmits the generated image data to the central management device 33, and the central management device 33 stores the image data transmitted from the image data processing device 32. Store in the unit 1013.

  As described above, the sound data and the image data sequentially stored in the storage unit 1013 in the production mode are integrated as video data in which the user participates as a character, and is played back, transmitted, stored in a storage medium, etc. according to a user instruction Is made.

  Also with the video generation system 2 described above, as with the video generation system 1 in the first embodiment, the user can easily participate in the generation of video data and can also enjoy video data that can make the audience more entertaining. Can be generated.

  By the way, the sound data processing device 31, the image data processing device 32, and the central management device 33 may be realized by dedicated hardware, or may be realized by causing a general-purpose computer to execute processing according to an application program. Good. When the sound data processing device 31, the image data processing device 32, and the central management device 33 are realized by a general-purpose computer, each component of the control unit 1031, the control unit 1032, and the control unit 1033 includes a CPU included in the general-purpose computer and the CPU. A DSP operating under control performs a process according to each module included in the application program in parallel and is realized as a function of a general-purpose computer.

[3. Third Embodiment]
The video generation system 1 and the video generation system 2 described above include dedicated devices that a general user does not seem to possess, such as the microphone 12 including the ultrasonic generator 121, the camera moving device 16, and the illumination device 17. Therefore, it is difficult for a general user to generate video data according to the present invention at home. However, the video generation system according to the present invention need not include all the components included in the video generation system 1 or the video generation system 2. Hereinafter, a case in which the present invention is implemented by connecting a home display, a video camera, a microphone, an amplifier, a speaker, etc. to a personal computer and causing the CPU and DSP of the personal computer to execute processing according to an application program will be described. This will be described as a third embodiment.

  FIG. 17 is a block diagram showing the configuration of the video generation system 3 according to the third embodiment. However, since the configuration and operation of the video generation system 3 are common in many respects to those of the video generation system 1, only the differences between the video generation system 3 and the video generation system 1 will be described below. In FIG. 17, among the components of the video generation system 3, the same components as those of the video generation system 1 are given the same reference numerals. In FIG. 17, for simplification of the drawing, only one component of the same kind consisting of a plurality is shown.

  The video generation system 3 further connects a home microphone 12, an amplifier 13, a speaker 14, a video camera 15, and a media drive 20 to the personal computer 40 and the personal computer 40 to which the keyboard 41 and the display 18 are connected. It has a configuration that can be obtained. The keyboard 41 includes a plurality of keys, and has a function of inputting a user instruction to the personal computer 40 by transmitting a signal corresponding to the pressed key to the personal computer 40 in response to a key pressing operation by the user. .

  The personal computer 40 includes a storage unit 101 composed of RAM (Random Access Memory), ROM (Read Only Memory), HD (Hard Disk), etc., and a control unit 103 composed of one or more CPUs and DSPs. An output I / F 102 is provided.

  The control unit 103 executes processing according to the application program stored in the storage unit 101, thereby performing a sound data block 201 including a sound source unit 2011, a sound data reproduction unit 2012, and a sound data processing unit 2013, and a moving image Functions of an image data block 202 including a data reproduction unit 2021, a character screen generation unit 2022, a rendering processing unit 2023, a trimming processing unit 2024, and an image data processing unit 2025, a camera operation control unit 2031, and an event data reproduction unit 204 Is realized.

  The microphone 12 of the video generation system 3 does not include the ultrasonic generator 121, and the ultrasonic receiver 19 is not connected to the personal computer 40. Accordingly, the personal computer 40 does not perform processing for calculating the microphone position using ultrasonic waves, and does not perform user identification processing based on the microphone position. Instead, the trimming processing unit 2024 of the personal computer 40 performs each user's specifying process based on the user's image data.

  More specifically, first, each of the plurality of users wears different color markers on the chest or the like. The image data captured by the video camera 15 includes a marker image as well as the user image. When receiving the image data from the video camera 15, the trimming processing unit 2024 generates user image data by the trimming process, and extracts data indicating each color of the marker worn on the chest or the like by the user from the user image data. The trimming processing unit 2024 identifies which color data is extracted from which user image data position, and indicates the corresponding color for each of the user image data indicating images of different users. Data is added and transmitted to the rendering processing unit 2023. The rendering processing unit 2023 can specify which user image data indicates which user based on the data indicating the color added to the user image data.

  The method in which the personal computer 40 associates the user with the user's image data is not limited to the above. For example, when each image data of the user is stored in the personal computer 40 in advance, and the image data block 202 of the personal computer 40 generates a plurality of user image data from the image data received from the video camera 15, it is stored in advance. It is also possible to compare the image data of each user and the generated user image data so as to associate them with each other having a high degree of similarity.

  As described above, when the video generation system does not have a function of calculating the microphone position, it is difficult for the video generation system to specify an accurate position of each user in the three-dimensional space. Based on the size, it is possible to specify an approximate position of each user in the three-dimensional space.

  Further, the camera moving device 16 and the lighting device 17 are not connected to the personal computer 40, and therefore the personal computer 40 does not perform the control processing of the moving operation of the video camera 15 and the control processing of the lighting device 17. Instead, it is obtained when the video camera 15 is moved by the image data processing unit 2025 of the personal computer 40 performing various types of image data processing according to the event data received from the event data playback unit 204. Image data or image data similar to image data obtained when the color or brightness of illumination is changed is generated.

  According to the video generation system 3 described above, the user can easily generate video data in which he / she participates as a character at home or the like.

The mode that the video generation system concerning 1st Embodiment of this invention was installed in the booth is shown. It is the block diagram which showed the structure of the video production | generation system concerning 1st Embodiment of this invention. It is the figure which showed typically the structure of the story data concerning 1st Embodiment of this invention. It is the figure which illustrated the contents of the management data concerning a 1st embodiment of the present invention. It is the figure which illustrated the content of the event sequence data regarding the background sound concerning 1st Embodiment of this invention. It is the figure which illustrated the content of the event sequence data regarding the background image concerning 1st Embodiment of this invention. It is the figure which illustrated the content of the event sequence data regarding the character image concerning 1st Embodiment of this invention. It is the figure which illustrated the content of the event sequence data regarding the dialog concerning 1st Embodiment of this invention. It is the figure which illustrated the content of the event sequence data regarding the microphone effect concerning 1st Embodiment of this invention. It is the figure which illustrated the content of the event sequence data regarding the camera work concerning 1st Embodiment of this invention. It is the figure which showed typically the content of the arrangement data concerning 1st Embodiment of this invention. It is the figure which looked at the booth which has arrange | positioned the ultrasonic receiver concerning 1st Embodiment of this invention from upper direction. It is the figure which illustrated the screen displayed on the display of the video generation system concerning a 1st embodiment of the present invention. It is the figure which illustrated the screen displayed on the display of the video generation system concerning a 1st embodiment of the present invention. It is the figure which illustrated the screen displayed on the display of the video generation system concerning a 1st embodiment of the present invention. It is the block diagram which showed the structure of the video production | generation system concerning 2nd Embodiment of this invention. It is the block diagram which showed the structure of the video production | generation system concerning 3rd Embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 * 2 * 3 ... Video generation system, 10 ... Video generation apparatus, 11 ... Controller, 12 ... Microphone, 13 ... Amplifier, 14 ... Speaker, 15 ... Video camera, 16 ... Camera movement apparatus, 17 ... Illumination apparatus, 18 ... Display, 19 ... Ultrasonic receiver, 20 ... Media drive, 21 ... Internet, 31 ... Sound data processing device, 32 ... Image data processing device, 33 ... Central management device, 40 ... Personal computer, 41 ... Keyboard, 101/1011 · 1012 · 1013 ··· storage unit, 102 · 1021 · 1022 · 1023 · · I / F, 103 · 1031 · 1032 · 1033 · · · control unit, 121 · ultrasonic generator, 201 · sound data block · 202 · image data Block, 203 ... Operation control block, 204 ... Event data reproduction unit, 205 ... Ma Position calculation unit 206 ... billing data processing unit 1010 ... story data 2011 ... sound source unit 2012 ... sound data reproduction unit 2013 ... sound data processing unit 2021 ... video data reproduction unit 2022 ... character screen generation unit , 2023 ... rendering processing unit, 2024 ... trimming processing unit, 2025 ... image data processing processing unit, 2031 ... camera operation control unit, 2032 ... moving operation control unit, 2033 ... lighting operation control unit.

Claims (13)

Operation instruction data storage means for storing operation instruction data indicating instruction of an operation to be performed by the user in association with timing data indicating a timing at which the instruction should be notified to the user;
Image processing instruction data storage means for storing image processing instruction data for instructing processing to be executed on image data in association with timing data indicating timing at which the processing is to be executed;
The operation instruction data transmitting means for transmitting the operation instruction data to a notification device that notifies the user of the instruction at a timing according to the timing data associated with the operation instruction data;
Image processing instruction data transmitting means for transmitting the image processing instruction data to an image data processing device that processes user image data at a timing according to timing data associated with the image processing instruction data. A data processing apparatus. Operation instruction data storage means for storing operation instruction data indicating instruction of an operation to be performed by the user in association with timing data indicating a timing at which the instruction should be notified to the user;
Image processing instruction data storage means for storing image processing instruction data for instructing processing to be executed on image data in association with timing data indicating timing at which the processing is to be executed;
Operation instruction notification means for notifying the user of instruction indicated by the operation instruction data at a timing according to timing data associated with the operation instruction data;
Image processing instruction data transmitting means for transmitting the image processing instruction data to an image data processing device that processes user image data at a timing according to timing data associated with the image processing instruction data. A data processing apparatus. Operation instruction data storage means for storing operation instruction data indicating instruction of an operation to be performed by the user in association with timing data indicating a timing at which the instruction should be notified to the user;
Image processing instruction data storage means for storing image processing instruction data for instructing processing to be executed on image data in association with timing data indicating timing at which the processing is to be executed;
The operation instruction data transmitting means for transmitting the operation instruction data to a notification device that notifies the user of the instruction at a timing according to the timing data associated with the operation instruction data;
Image data receiving means for receiving user image data from the imaging device;
Image data processing means for performing processing on the image data of the user at a timing corresponding to timing data associated with the image processing instruction data. Data processing device. Operation instruction data storage means for storing operation instruction data indicating instruction of an operation to be performed by the user in association with timing data indicating a timing at which the instruction should be notified to the user;
Image processing instruction data storage means for storing image processing instruction data for instructing processing to be executed on image data in association with timing data indicating timing at which the processing is to be executed;
Operation instruction notification means for notifying the user of instruction indicated by the operation instruction data at a timing according to timing data associated with the operation instruction data;
Image data receiving means for receiving user image data from the imaging device;
Image data processing means for performing processing on the image data of the user at a timing corresponding to timing data associated with the image processing instruction data. Data processing device. 5. The data processing apparatus according to claim 3, further comprising a display unit configured to display an image indicated by image data generated as a result of the processing performed by the image data processing unit. The data processing apparatus according to claim 3, further comprising an image data storage unit that stores image data generated as a result of the processing by the image data processing unit. Utterance instruction data storage means for storing utterance instruction data indicating instruction of utterance to be performed by the user in association with timing data indicating timing at which the instruction should be notified to the user;
Voice processing instruction data storage means for storing voice processing instruction data for instructing processing to be executed on the voice data in association with timing data indicating timing at which the processing is to be executed;
Utterance instruction notification means for notifying the user of instruction indicated by the utterance instruction data at a timing according to timing data associated with the utterance instruction data;
Voice data receiving means for receiving user voice data from the sound pickup device;
Voice data processing means for performing a processing process instructed by the voice processing instruction data on the voice data of the user at a timing according to timing data associated with the voice processing instruction data. A data processing apparatus according to any one of claims 1 to 4. The data processing apparatus according to claim 7, further comprising: sound generation means for generating a sound indicated by sound data generated as a result of the processing by the sound data processing means. The data processing apparatus according to claim 7, further comprising: sound data storage means for storing sound data generated as a result of processing by the sound data processing means. An operation for storing operation instruction data instructing an operation to be executed by at least one of the imaging device, a moving device for moving the imaging device, and an illumination device in association with timing data indicating a timing at which the operation is to be executed Instruction data storage means;
The operation instruction data is irradiated with light to an imaging device that generates user image data, a moving device that moves the imaging device, and a user at a timing according to timing data associated with the operation instruction data. The data processing device according to claim 1, further comprising: an operation instruction data transmitting unit that transmits to at least one of the lighting devices to be performed. An operation for storing operation instruction data instructing an operation to be executed by at least one of the imaging device, a moving device for moving the imaging device, and an illumination device in association with timing data indicating a timing at which the operation is to be executed Instruction data storage means;
The operation instruction data is irradiated with light to an imaging device that generates user image data, a moving device that moves the imaging device, and a user at a timing according to timing data associated with the operation instruction data. An operation instruction data transmitting means for transmitting to at least one of the lighting devices to be performed and also for transmitting to the image data processing means,
5. The data processing apparatus according to claim 3, wherein the image data processing unit performs processing on the image data of the user according to the operation instruction data. Image data generating means for sequentially generating image data that changes in response to temporal changes in user image data sequentially received by the image data receiving means;
The image data processing means performs processing according to the image processing instruction data on the image data generated by the image data generation means instead of or in addition to the user image data. The data processing apparatus according to claim 4, wherein: A process of storing operation instruction data indicating instruction of an operation to be performed by the user in association with timing data indicating a timing at which the information is to be notified to the user;
Processing for storing image processing instruction data for instructing processing to be performed on image data in association with timing data indicating timing at which the processing should be performed;
A process of notifying the user of the guidance indicated by the motion guidance data at a timing according to the timing data associated with the motion guidance data;
A process of receiving user image data from the imaging device;
Processing the user's image data at a timing according to the timing data associated with the image processing instruction data according to the image processing instruction data;
Processing for displaying an image indicated by image data generated as a result of the processing;
A program for causing a computer to execute processing for storing image data generated as a result of the processing.

Images