JP5687961B2 - Synchronous playback apparatus and synchronous playback method - Google Patents

Description

  The present invention relates to a synchronized playback apparatus, a synchronized playback method, a synchronized playback program, and a recording medium on which the synchronized playback program is recorded.

  2. Description of the Related Art Conventionally, video that is synchronized with music sound is widely played back in synchronization with the music sound. Such synchronized playback is performed, for example, by recording a video of a dance danced to the music sound simultaneously with the recording of the background sound including the music sound by a video camera device having a recording function. Are played back simultaneously.

  By the way, the sound quality of the music sound when the recording result by the video camera device having the recording function is reproduced is greatly inferior to the case where the music content corresponding to the music sound is reproduced by the music content reproduction output device. The situation where the playback sound quality of the music sound that is synchronized with the video is not high is also the case where a technique called “CD-EXTRA” for viewing a moving image or a still image on a personal computer is adopted. (See paragraphs [0002] to [0004] of Patent Document 1).

  For this reason, a technique for reproducing video in synchronization with the reproduction sound of digital music content has been proposed (see paragraphs [0009] to [0022] of Patent Document 1; hereinafter referred to as “conventional example”). In this conventional technology, as an example, on a compact disc (CD), a fixed time interval or specified during music playback using an audio session in which information for playback of high-quality music is recorded. An image reproduction request signal is issued at a predetermined time interval, and an image for which the association with the music sound is specified in advance is reproduced one screen at a time in synchronization with the image reproduction request signal. In addition, in the technology of the conventional example, as another example, an image playback request signal is issued at the start of music playback using an audio session, and then an image that has been previously designated to be associated with music sound is displayed for a certain period of time. Each screen is reproduced at intervals or at specified time intervals.

JP-A-10-143978

  The above-described conventional technique is a technique that allows an arbitrary video to be reproduced in synchronization with the reproduced music sound. For this reason, in the technology of the conventional example, in order to reproduce a moving image that matches the reproduction timing of the reproduced music sound, it is necessary to specify a fine time interval in accordance with the progress of the reproduction of the music sound. Such necessity is similarly necessary even if the video is a video synchronized with the music sound.

  By the way, the image of the dance that is danced to the music sound as described above is originally an image synchronized with the music sound. When reproducing a video synchronized with such music sound, it is generally preferable to maintain the synchronization relationship at the time of shooting when the reproduced video and the reproduced music sound are synchronized while improving the reproduction sound quality of the music sound. However, to achieve such synchronization by using only the time interval designation using the conventional technique, the editor needs a high level of skill.

  For this reason, in order to synchronize the reproduced video obtained by reproducing the photographing result of the video synchronized with the music sound and the reproduced musical sound based on the music content corresponding to the music sound, the music sound and the video at the time of photographing are synchronized. A technology that can easily maintain the relationship is awaited. Meeting this requirement is one of the problems to be solved by the present invention.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a new synchronized playback apparatus and synchronized playback method for displaying a video in synchronization with the reproduced music sound.

The invention according to claim 1 is a synchronous playback device that displays video in synchronization with the playback music sound, and that obtains playback data including video data synchronized with the music sound; Reproduction synchronization information in which the synchronization relationship between the reproduction timing of the reproduced music sound based on the music content corresponding to and the reproduction timing based on the acquired reproduction data is the same as the synchronization relationship between the music sound and the video Generating means for generating sound; sound output control means for outputting a sound including the reproduced music sound from a sound output unit; and synchronizing with the reproduction timing of the reproduced music sound based on the generated reproduction synchronization information while a display control means for displaying a reproduced image based on data of the obtained image on the display unit; wherein the generating means is a musical co corresponding to the music sound by analyzing the music sound Further subjected to specific Ceiling, extracting the reproduction synchronization information by using the specific results, it is a synchronous playback device according to claim.

The invention according to claim 9 is a synchronized playback method used in a synchronized playback apparatus that displays video in synchronization with the playback music sound, and acquires playback data including video data synchronized with the music sound. A synchronization relationship between the playback timing of the playback music sound based on the music content corresponding to the music sound and the playback timing based on the acquired playback data, and the synchronization relationship between the music sound and the video A generation step of generating reproduction synchronization information that is the same as: a sound output control step of outputting a sound including the reproduction music sound from a sound output unit; and a reproduction of the reproduction music sound based on the generated reproduction synchronization information while achieving synchronization between the reproduction timing, the display control step of displaying on the display unit the obtained reproduced video based on the data of the video; equipped with, in the generating step, the solution to the music sound Was further subjected to specific music content corresponding to the music sound, the extracts of the reproduction synchronization information by using a particular result is a synchronized playback method characterized by.

A tenth aspect of the present invention is a synchronous reproduction program characterized by causing a calculation means to execute the synchronous reproduction method according to the ninth aspect.

According to an eleventh aspect of the present invention, there is provided a recording medium in which the synchronous reproduction program according to the tenth aspect is recorded so as to be readable by the calculation means.

It is a block diagram for demonstrating the structure of the synchronous reproduction apparatus which concerns on 1st Embodiment of this invention. It is a block diagram for demonstrating the structure of the synchronous reproduction apparatus which concerns on 2nd Embodiment of this invention. It is a block diagram for demonstrating the structure of the synchronous reproduction apparatus which concerns on 1st Example of this invention. It is a figure for demonstrating the content of the information for reproduction | regeneration of FIG. 4 is a flowchart for explaining reproduction synchronization information generation processing by the synchronous reproduction apparatus of FIG. 3. 4 is a flowchart for explaining synchronous video display processing by the synchronous reproduction device of FIG. 3. 4 is a flowchart for explaining synchronous sound output processing by the synchronous reproduction device of FIG. 3. FIG. 4 is a flowchart for explaining edit synchronization information generation processing by the synchronous playback device of FIG. 3. FIG. FIG. 4 is a flowchart for explaining a synchronized video switching process and a synchronized sound output process performed in parallel with the editing synchronization information generation process by the synchronized playback apparatus of FIG. 3; It is a block diagram for demonstrating the structure of the synchronous reproduction apparatus which concerns on 2nd Example of this invention. It is a figure for demonstrating the content of the acquisition information of FIG. It is a figure for demonstrating the content of the information for reproduction | regeneration of FIG. 11 is a flowchart for explaining an acquisition operation by the synchronous playback device of FIG. 10. 11 is a flowchart for explaining reproduction synchronization information generation processing by the synchronous reproduction apparatus of FIG. 10.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description and drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant description is omitted.

[First Embodiment]
First, a first embodiment of the present invention will be described with reference to FIG.

<Configuration>
FIG. 1 shows a schematic configuration of a synchronized playback device 700A according to the first embodiment.

  As shown in FIG. 1, the synchronized playback device 700A includes an imaging unit 710, an acquisition unit 720A, a generation unit 730A, and a playback information storage unit 735. In addition, the synchronized playback device 700A includes a display control unit 740 and a display unit 745. Here, the display unit 745 includes a display unit 746 and a display unit 747 as a specific display unit.

  The synchronized playback device 700A includes a sound source data unit 750, sound output control means 760A, and a sound output unit 765. Furthermore, the synchronized playback device 700A includes an input unit 770.

  The synchronized playback device 700A captures a video synchronized with the playback music sound based on the music content stored in the sound source data unit 750, and thereafter, the playback music sound at the time of shooting and the captured video This is a device that synchronizes and reproduces the reproduced music sound and the captured video in the same synchronous relationship as the synchronous relationship.

  The photographing means 710 includes a video camera or the like for photographing a peripheral video and outputting the photographing result as video data. The result of shooting by the shooting unit 710 performed in synchronization with the output of the reproduced music sound from the sound output unit 765 is sent to the acquisition unit 720A and the display control unit 740 as video data.

  The acquisition unit 720A receives the video data sent from the photographing unit 710. In addition, the acquisition unit 720 </ b> A receives an acquisition operation start command and an end command sent from the input unit 770. Then, the acquisition unit 720A acquires video data received during a period from when the acquisition operation start command is received until the acquisition operation end command is received. The video data acquired by the acquisition unit 720A is sent to the generation unit 730A as acquired video data.

  The generation unit 730A receives the acquired video data sent from the acquisition unit 720A. Further, the generation unit 730A receives the reproduction synchronization information generation command sent from the input unit 770. Further, the generation unit 730A receives the start command and the end command of the acquisition operation sent from the input unit 770, similarly to the acquisition unit 720A described above.

  In addition, the generation unit 730A receives the start instruction and the end instruction of music sound reproduction sent from the input unit 770. Here, in the music sound playback start command, the music content to be played back is specified.

  Further, the generation unit 730A receives the synchronous reproduction start command and end command sent from the input unit 770. Here, in the synchronous reproduction start command, one of reproduction synchronization information or edit synchronization information stored in the reproduction information storage unit 735 is designated. In the synchronous reproduction start command, the presence / absence of simultaneous display referred to by the display control means 740 is further specified.

  Upon receiving these instructions, generation means 730A detects the reception time. Then, the generation unit 730A generates reproduction synchronization information based on the detection result of the reception time. Details of the reproduction synchronization information generation process will be described later.

  Note that the reproduction synchronization information generated by the generation unit 730A synchronously reproduces the reproduced music sound and the captured video in the same synchronous relationship as the synchronous relationship between the reproduced musical sound and the captured video at the time of shooting. This information is used at the time. In the first embodiment, the playback synchronization information includes the reception start time of the acquisition operation, the acquisition time of the acquisition operation using the playback start time of the playback music sound output from the sound output unit 765 as the reference time during the generation process of the playback synchronization information. The corresponding playback video data and music content designation information are added to the relative time information of the reception time of the operation end command and the reception time of the music sound playback or synchronous playback end command with respect to the reference time. It has become. Note that the relative time information regarding the music sound playback end command may or may not be included in the playback synchronization information.

  Further, the generation unit 730A receives the reproduction synchronization information editing start command and end command and the synchronous video switching command sent from the input unit 770. Here, in the reproduction synchronization information editing start command, one of reproduction synchronization information or editing synchronization information stored in the reproduction information storage unit 735 is designated. Further, in the synchronous video switching command (reproduction content designation), one of a plurality of types of reproduction video data stored in the reproduction information storage unit 735 is designated.

  The generation unit 730A executes editing synchronization information generation processing in accordance with these instructions. A process for generating such editing synchronization information will be described later.

  Note that the editing synchronization information generated by the generation unit 730A synchronizes and reproduces the reproduced music sound and the captured video in the same synchronous relationship as the synchronous relationship between the reproduced musical sound and the captured video at the time of shooting. On the other hand, the information is used for reproducing the video in which the captured video is sequentially switched in accordance with the synchronous video switching designation made by the user.

  The reproduction information storage unit 735 receives the reproduction video data, reproduction synchronization information, and edit synchronization information sent from the generation unit 730A. Then, the reproduction information storage unit 735 stores the reproduction video data, the reproduction synchronization information, and the editing synchronization information therein. The reproduction information storage unit 735 can be accessed by the generation unit 730A, the display control unit 740, and the sound output control unit 760A.

  The display control means 740 receives video data sent from the photographing means 710. Further, the display control means 740 receives the synchronous reproduction start command and end command sent from the input means 770 in the same manner as the generation means 730A described above.

  When the display control means 740 receives the synchronous reproduction start command, the display control means 740 starts a synchronous display process with reference to the reproduction synchronous information or the edit synchronous information designated by the synchronous reproduction start command. Details of the synchronous display processing will be described later.

  Similarly to the generation unit 730A described above, the display control unit 740 receives a reproduction synchronization information editing start command and an end command and a synchronous video switching command sent from the input unit 770. Then, the display control means 740 executes synchronous video switching display processing in accordance with these instructions. Details of the synchronous video switching display process will be described later.

  The display means 746 and 747 include a display device such as a video projector or a liquid crystal display panel. Here, the display means 746 receives the synchronous video data for synchronous video display sent from the display control means 740. Then, the display means 746 reproduces and displays the synchronized video corresponding to the synchronized video data.

  The display means 747 receives real-time video data for real-time video display sent from the display control means 740. Then, the display means 746 reproduces and displays the real time video corresponding to the real time video data. The display means 746 and 747 can display images independently of each other.

  The sound source data unit 750 stores various music contents as digital contents. The sound source control unit 760A can access the sound source data unit 750.

  The sound output control means 760A receives the music sound reproduction start command and end command sent from the input means 770 in the same manner as the generation means 730A described above. When the sound output control means 760A receives a music sound reproduction start command, the sound output control means 760A starts the music reproduction process. In such music playback processing, the music content data specified in the music sound playback start command is sequentially read. Then, the sound output control means 760A generates reproduced music sound data based on the read music content data, and sends it to the sound output unit 765.

  The sound output control means 760A receives the synchronous reproduction start command and end command sent from the input means 770 in the same manner as the generation means 730A and the display control means 740 described above. When the sound output control means 760A receives a synchronous reproduction start command, the sound output control means 760A starts a synchronous sound output process. Details of the sync sound output processing will be described later.

  Also, the sound output control means 760A receives the reproduction synchronization information editing start command and end command sent from the input means 770 in the same manner as the generation means 730A and display control means 740 described above. When the sound output control means 760A receives the reproduction synchronization information editing start command, the sound output control means 760A starts the synchronization sound output process in parallel with the edit synchronization information generation process by the generation means 730A described above. Details of the sync sound output processing will be described later.

  The sound output unit 765 is configured to include a speaker. The sound output unit 765 receives output sound (reproduced music sound in the first embodiment) data sent from the sound output control means 760A. The sound output unit 765 reproduces and outputs an output sound corresponding to the output sound data.

  The input means 770 includes a keyboard or the like. In response to a user's input operation, the input means 770 sends information corresponding to the input operation to an element that needs to be transmitted. The information corresponding to such an input operation includes the above-mentioned reproduction synchronization information generation command, acquisition operation start command and termination command, music sound playback start command and termination command, synchronized playback start command and termination command, playback synchronization information An editing start command and an end command, and a synchronous video switching command are included.

<Operation>
Next, the operation of the synchronized playback apparatus 700A configured as described above will be described mainly focusing on the playback synchronization information generation process, the synchronized playback process, and the edit synchronization information generation process.

<< Playback synchronization information generation process >>
When the user performs an input operation of a reproduction synchronization information generation command to the input unit 770, a reproduction synchronization information generation command is sent from the input unit 770 to the generation unit 730A. The reproduction synchronization information generation process is executed by the generation unit 730A that has received the reproduction synchronization information generation instruction.

  Receiving the reproduction synchronization information generation command, the generation unit 730A starts the reproduction synchronization information generation process. Note that, when a music sound playback or synchronized playback start command is received during a period in which the generation process is not being performed, the generation unit 730A plays the music sound playback as preparation for subsequent generation processing of the playback synchronization information. Alternatively, the progress of reproduction of the reproduced music sound performed in response to the synchronous reproduction start command is monitored.

  When the generation processing of the reproduction synchronization information is started, the generation unit 730A receives the acquisition operation start command and the end command in the period until the acquisition operation end command is received, and the music sound playback or synchronous playback start command and end command. Reception is monitored and each reception time is detected. Further, the generation unit 730A performs buffering processing of the video data sent from the acquisition unit 720A during the acquisition period from the reception of the acquisition operation start command to the reception of the end command.

  Upon receiving the acquisition operation end command, the generation unit 730A sends the buffered acquired video data to the playback information storage unit 735 as playback video data.

  Next, the generation unit 730A, based on the detection result of the reception time of the start instruction of the music sound reproduction or the synchronous reproduction during the reproduction synchronization information generation process, or the monitoring result before the reproduction synchronization information generation process described above, Using the playback start time of the playback music sound output from the sound output unit 765 during the playback synchronization information generation process as a reference time, the reception time of the acquisition operation start command, the reception time of the acquisition operation end command, the music sound playback, or Relative time information with respect to each reference time of the reception time of the synchronous reproduction end command is calculated.

  Note that the reception time of the music sound playback or synchronous playback end command may or may not be detected between the reception of the playback synchronization information generation command and the reception of the acquisition operation end command. is there. Therefore, when the reception time of the music sound playback end command is detected, the relative time information of the reception time of the music sound playback or synchronous playback end command is calculated. If the reception time of the end command is not detected, the relative time information of the reception time of the music sound reproduction end command is not calculated.

  Next, the generating unit 730A generates reproduction synchronization information by adding the current reproduction video data and the music content designation information designated by the music sound reproduction start command to the calculated relative time information. Then, the generation unit 730A sends the generated reproduction synchronization information to the reproduction information storage unit 735. Thereafter, the generation unit 730A ends the generation process of the reproduction synchronization information.

《Synchronized playback processing》
When the user performs an input operation of a synchronized playback start command to the input unit 770, a synchronized playback start command is sent from the input unit 770 to the generating unit 730A, the display control unit 740, and the sound output control unit 760A. . The synchronized playback processing is executed as synchronized video display processing by the display control means 740 that has received this synchronized playback start command and synchronized sound output processing by the sound output control means 760A.

  Upon receiving the synchronous reproduction start command, the display control means 740 starts the synchronous video display process. In such a synchronized video display process, the display control means 740 displays the video to be played at each time point according to the synchronized relationship with the playback music sound indicated by the playback synchronization information or the edit synchronization information specified in the synchronous playback start command. Are read from the reproduction information storage unit 735 sequentially.

  Then, the display control unit 740 generates synchronized video data based on the read data. The synchronized video data thus generated is sent to the display means 746. As a result, the display unit 746 displays the synchronized video.

  Further, the display control means 740 provides real-time video for displaying the video being shot in real time based on the video data sent from the shooting means 710 when simultaneous display is specified in the synchronous playback start command. Data is generated and sent to the display means 747 in the display unit 745. As a result, simultaneously with the display of the synchronized video on the display means 746, the display means 747 displays the real-time video.

  When the user receives a synchronous playback end command input to the input unit 770 in order to end the current synchronized playback, the display control unit 740 ends the synchronized video display process and generates synchronized video data. finish. In the synchronous video display processing, when simultaneous display control of synchronous video display and real-time video display is performed, the display control means 740 that has received the synchronous playback end command receives the synchronous video data and real-time video. End both data generation.

  When receiving the synchronized playback start command, the sound output control means 760A starts the synchronized sound output process. In this synchronized sound output process, the sound output control means 760A refers to the reproduction synchronization information or the edit synchronization information designated by the synchronous reproduction start command, and the part of the music content corresponding to the music sound to be reproduced at each time point Are sequentially read from the sound source data section 750. Then, the sound output control means 760A generates reproduction music sound data based on the read data, and sends the generated reproduction music sound data to the sound output unit 765. As a result, the reproduced music sound is reproduced and output from the sound output unit 765.

  Then, upon receiving a synchronous playback end command for ending the current synchronized playback, the sound output control means 760A ends the synchronized sound output processing and ends the generation of the reproduced music sound data.

《Edit synchronization information generation processing》
When the user performs an input operation of the reproduction synchronization information editing start command to the input unit 770, the reproduction synchronization information editing start command is transmitted from the input unit 770 to the generation unit 730A, the display control unit 740, and the sound output control unit. 760A. The editing synchronization information generation process is executed by the generation means 730A that has received the reproduction synchronization information editing start command.

  Upon receiving the reproduction synchronization information editing start command, the generation unit 730A starts the editing synchronization information generation process. In the generation process of the edit synchronization information, the generation unit 730A first reads the reproduction synchronization information or the edit synchronization information specified in the reproduction synchronization information editing start command from the reproduction information storage unit 735 as provisional edit synchronization information. . As will be described later, when the playback synchronization information editing start command is received, the display control means 740 and the sound output control means 760A start the synchronized playback processing in the same manner as when the above-described synchronized playback start command is received. It is supposed to be.

  Thereafter, every time a synchronized video switching command is received during an editing period until a playback synchronization information editing end command is received, the generation means 730A first generates a reference time (synchronized playback sound output of the playback music sound output) in the editing period. The relative time of the current time with respect to the (start time) is specified. Subsequently, the generation unit 730A updates the provisional editing synchronization information at that time by adding the information on the specified relative time and the information on the reproduction video data specified in the synchronous video switching command. .

  If the provisional editing synchronization information read by the generation unit 730A at the start of the editing period is editing synchronization information, the generation unit 730A issues an initial synchronous video switching command after the start of the synchronous reproduction process. When received, the provisional editing synchronization information portion after the relative time corresponding to the reception time of the first synchronized video switching command is deleted.

  Thereafter, upon receiving an instruction to finish reproduction synchronization information editing, the generation means 730A, if the temporary editing synchronization information has been updated, the temporary editing synchronization information at that point in time as new editing information generated this time Confirm the synchronization information. Then, the generation unit 730A sends the generated new editing synchronization information to the reproduction information storage unit 735.

  When the display control means 740 receives the reproduction synchronization information editing start command, the display control means 740 displays the reproduction synchronization information or the editing synchronization information designated by the reproduction synchronization information editing start command, as in the case of the synchronous reproduction processing described above. Start the referenced synchronous display process. When receiving the synchronized video switching command before receiving the playback synchronization information editing end command, the display control means 740 refers to the playback synchronization information corresponding to the video data designated by the synchronized video switching command, The specified video data portion to be reproduced after the video switching is specified while maintaining the synchronous relationship with the reproduction timing of the reproduced music sound.

  Subsequently, the display control means 740 sequentially reads the data of the video data portion specified this time from the reproduction information storage unit 735 sequentially until receiving the next synchronous video switching command or the reproduction synchronization information editing end command. Synchronized video data is generated and sent to the display means 746 in the display unit 745. As a result, the display unit 746 displays a synchronized video based on the video data designated by the synchronized video switching command.

  Upon receiving the reproduction synchronization information editing end command, the display control unit 740 ends the synchronized video data generation process. As a result, the supply of the synchronized video data to the display unit 745 is stopped.

  When receiving the reproduction synchronization information editing start command, the sound output control means 760A receives the reproduction synchronization information or editing synchronization designated by the reproduction synchronization information editing start command, as in the case of the above-described synchronization sound output processing. Synchronous sound output processing with reference to the information is started. Subsequently, the sound output control unit 760A continues the reproduction output process of the reproduced music sound until receiving an instruction to end reproduction synchronization information editing.

  When receiving the reproduction synchronization information editing end command, the sound output control means 760A ends the reproduction output processing of the reproduced music sound. As a result, the supply of the reproduced music sound data to the sound output unit 765 is stopped.

  As described above, in the first embodiment, the obtaining unit 720A obtains video data synchronized with the reproduced music sound photographed by the photographing unit 710. Subsequently, the generation unit 730A determines the synchronization relationship between the playback timing of the playback music sound at the time of shooting and the playback timing of the playback video based on the acquired video data, and the synchronization relationship between the playback music sound and the video at the time of shooting. The reproduction synchronization information for making the same is generated. With reference to the reproduction synchronization information generated in this way, the display control unit 740 causes the display unit 746 in the display unit 745 to display the synchronized video using the acquired video data, and the sound output control unit 760A performs the reproduction. The music sound is output from the sound output unit 765.

  Therefore, according to the first embodiment, it is possible to reproduce the reproduced music sound and the captured video after shooting in the same synchronization relationship as that of the reproduced music sound and the video at the time of shooting. .

  Further, in the first embodiment, when simultaneous display is specified in the synchronous reproduction start command, the display control unit 740 is captured by the imaging unit 710 while displaying the synchronized video on the display unit 746. The real-time video being displayed is displayed by the display means 747. For this reason, it is possible to simultaneously display the past video and the current video synchronized with the same reproduced music sound.

  In the first embodiment, every time a synchronized video switching command is received during synchronized playback, the generation means 730A synchronizes with the progress of the playback music sound being played back and outputs the synchronization specified in the synchronized video switching command. Edit synchronization information for switching to video is generated. When receiving a synchronous video switching command during synchronous playback, the display control means 740 switches to the synchronous video specified in the synchronous video switching command and causes the display means 746 to display it. For this reason, so-called timeline editing of the past video synchronized with the same reproduced music sound can be easily performed.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIG.

<Configuration>
FIG. 2 shows a schematic configuration of a synchronized playback device 700B according to the second embodiment.

  As shown in FIG. 2, the synchronized playback device 700B is different from the synchronized playback device 700A described above in that the acquisition unit 720B, the generation unit 730B, and the sound are replaced with the acquisition unit 720A, the generation unit 730A, and the sound output control unit 760A. Output control means 760B is provided. The synchronized playback device 700B further includes sound collection means 715 and an acquired information storage unit 725. Hereinafter, description will be made mainly focusing on these differences.

  In the second embodiment, the sound source data unit 750 can be accessed by the generation unit 730B and the sound output control unit 760B.

  The synchronized playback device 700B captures a video synchronized with the music sound output from a device other than the synchronized playback device 700B, and then has the same synchronization relationship between the captured music sound and the captured video at the time of shooting. Thus, the playback music sound based on the music content stored in the sound source data section 750 and the captured video are synchronously played back.

  The sound collecting means 715 includes a microphone that picks up surrounding sounds. The result of sound collection by the sound collection means 715 (peripheral sound that is the background sound of the captured video) is sent to the acquisition means 720B as sound collection data.

  The acquisition unit 720B receives the image data sent from the photographing unit 710 and the acquisition operation start command and end command sent from the input unit 770 in the same manner as the acquisition unit 720A described above. Furthermore, the acquisition unit 720B receives the sound collection data sent from the sound collection unit 715. And the acquisition means 720B performs the acquisition process of acquisition video data, acquisition music sound data, and acquisition residual sound data based on the said video data and the said sound collection data. Details of the acquisition process will be described later.

  The acquired information storage unit 725 receives the acquired video data, acquired musical sound data, and acquired residual sound data sent from the acquiring unit 720B. Then, the reproduction information storage unit 735 stores the acquired video data, the acquired music sound data, and the acquired residual sound data therein. The acquisition unit 720B and the generation unit 730B can access the acquisition information storage unit 725.

  The generation unit 730B receives a reproduction synchronization information generation command and a reproduction synchronization information editing start command and an end command among the commands received from the input unit 770 by the generation unit 730A. Except for the reproduction synchronization information generation command among these commands, the generation unit 730B executes the same processing as the generation unit 730A.

  Upon receiving the reproduction synchronization information generation instruction, the generation unit 730B starts the reproduction synchronization information generation process. Details of the reproduction synchronization information generation processing will be described later.

  The sound output control means 760B receives a synchronous reproduction start command and an end command and a synchronous video switching command among the commands received by the sound output control means 760A from the input means 770. When a synchronous playback end command and a synchronous video switching command are received among these commands, the sound output control unit 760B performs the same processing as the sound output control unit 760A.

  In the second embodiment, the synchronous reproduction start command further includes designation of whether or not the residual sound is reproduced simultaneously.

  When receiving the synchronous reproduction start command, the sound output control means 760B starts the synchronous sound output process. Details of the sync sound output processing will be described later.

<Operation>
Next, the operation of the synchronized playback device 700B configured as described above will be described mainly focusing on the acquisition operation, the playback synchronization information generation process, the synchronized playback process, and the edit synchronization information generation process.

<< Acquisition process >>
When the user inputs an acquisition operation start command to the input unit 770, an acquisition operation start command is sent from the input unit 770 to the acquisition unit 720B. The acquisition process is executed by the acquisition unit 720B that has received the acquisition operation start command.

  Upon receiving the acquisition operation start command, the acquisition unit 720B starts the acquisition process. During the acquisition process, the acquisition unit 720B receives the video data transmitted from the imaging unit 710 during the acquisition period from when the acquisition operation start command is received until the acquisition operation end command is received. Upon receiving the acquisition operation end command, the acquisition unit 720B sends the video data acquired during the period to the acquisition information storage unit 725 as acquired video data.

  The acquisition unit 720B receives the sound collection data sent from the sound collection unit 715 during the acquisition period. Upon receiving the acquisition operation end command, the acquisition unit 720B analyzes the sound collection data acquired during the period, and removes the music sound and the remaining sound other than the music (the music sound is removed from the sound collection result). Sound) and separated into music sound data and residual sound data. Then, the acquisition unit 720B sends the music sound data and residual sound data obtained during the acquisition period to the acquisition information storage unit 725 as acquisition music sound data and acquisition residual sound data.

  When the acquired video data, acquired music sound data, and acquired residual sound data are sent to the acquisition information storage unit 725 in this way, the acquisition unit 720B ends the acquisition process. Then, the acquisition unit 720B enters a waiting state for a new acquisition operation start command.

<< Playback synchronization information generation process >>
When the user performs an input operation of a reproduction synchronization information generation command on the input unit 770, a reproduction synchronization information generation command is sent from the input unit 770 to the generation unit 730B. The reproduction synchronization information generation process is executed by the generation unit 730B that has received the reproduction synchronization information generation instruction.

  Upon receiving the reproduction synchronization information generation instruction, the generation unit 730B starts the reproduction synchronization information generation process. In the reproduction synchronization information generation process, the generation unit 730 </ b> B first reads the acquired music sound data from the acquisition information storage unit 725. Subsequently, the generation unit 730B analyzes the acquired music sound data and extracts metadata indicating the characteristics of the music sound corresponding to the acquired music sound data, and then the music content in the sound source data unit 750 corresponding to the metadata. Is identified. Then, the generation unit 730B generates reproduction synchronization information based on the specified music content.

  When the reproduction synchronization information is generated, the generation unit 730B reads the acquired video data and the acquired residual sound data from the acquisition information storage unit 725, and generates the reproduction information storage unit 735 as the video data for reproduction and the residual sound data for reproduction. And the generated reproduction synchronization information is sent to the reproduction information storage unit 735 as reproduction synchronization information associated with the reproduction video data and the reproduction residual sound data on a one-to-one basis. As a result, new reproduction video data, reproduction residual sound data, and reproduction synchronization information are added to the reproduction information storage unit 735.

  When the new reproduction video data, reproduction residual sound data, and reproduction synchronization information are sent to the reproduction information storage unit 735, the generation unit 730B generates the new reproduction video data, reproduction residual sound data, and reproduction synchronization information. The acquired video data, acquired music sound data, and acquired residual sound data in the acquired information storage unit 725 used for generating are deleted, and the reproduction synchronization information generation process is terminated. Then, the generation unit 730B waits for a new reproduction synchronization information generation command.

<< Synchronous playback process and edit synchronization information generation process >>
The synchronized video display process in the synchronized playback process of the second embodiment is performed by the display control means 740 in the same manner as in the first embodiment described above.

  The synchronized sound output process in the synchronized playback process of the second embodiment is executed by the sound output control means 760B. This synchronized sound output process is executed by the sound output control means 760B that has received a synchronous reproduction start command.

  When receiving the synchronous reproduction start command, the sound output control means 760B starts the synchronous sound output process. In this synchronized sound output process, the sound output control means 760B refers to the reproduction synchronization information or the edit synchronization information designated by the synchronous reproduction start command, and the music content data corresponding to the music sound to be reproduced at each time point Are sequentially read from the sound source data section 750. Also, when receiving the synchronous reproduction start command, the sound output control means 760B refers to the designated reproduction synchronization information or editing synchronization information when the remaining sound simultaneous reproduction is designated, and each time point The data of the residual sound data for reproduction to be reproduced in step S1 is sequentially read from the reproduction information storage unit 735.

  Subsequently, the sound output control means 760B generates synchronized output sound data corresponding to the synchronized output sound to be reproduced and output from the sound output unit 765. Here, when it is designated that the remaining sound is not reproduced simultaneously, the sound output control means 760B generates synchronized output sound data corresponding to the synchronized output sound consisting only of the reproduced music sound. On the other hand, when it is designated that there is residual sound simultaneous reproduction, the sound output control means 760B generates synchronized output sound data corresponding to the synthesized sound of the reproduced music sound and the residual sound. Then, the sound output control unit 760B sends the generated synchronized output sound data to the sound output unit 765. As a result, the sound output unit 765 reproduces and outputs the synchronized output sound.

  When receiving a synchronous reproduction end command for ending the current synchronous reproduction process, the sound output control unit 760B ends the synchronous sound output process and ends the generation of the synchronous output sound data.

  The generation of editing synchronization information in the second embodiment is performed by the generation unit 730B executing the same processing as the generation processing of editing synchronization information by the generation unit 730A in the first embodiment.

  As described above, in the second embodiment, the acquisition unit 720 </ b> B receives video data synchronized with the music sound photographed by the photographing unit 710 and audio data collected by the sound collecting unit 715. Subsequently, the acquisition unit 720B analyzes the collected voice data, identifies the music sound and the remaining sound other than the music, and separates the music sound data and the remaining sound data. Then, the acquisition unit 720B acquires video data, music sound data, and residual sound data in a period from when the acquisition operation start command is received until the acquisition operation end command is received.

  Next, the generation unit 730B includes the reproduction timing of the music sound at the time of shooting, the reproduction timing of the reproduction video based on the acquired video data, and the reproduction timing of the reproduction video based on the acquired residual sound data. Reproduction synchronization information is generated to make the synchronization relationship the same as the synchronization relationship between the music sound at the time of shooting, the video, and the residual sound. With reference to the reproduction synchronization information generated in this way, the display control means 740 causes the display means 746 in the display unit 745 to display the synchronized video based on the acquired video data, and the sound output control means 760B A sound output unit 765 outputs a synthesized sound of the sound and the residual sound.

  Therefore, according to the second embodiment, the reproduced music sound, the captured video image, and the residual sound are recorded after shooting in the same synchronization relationship as that of the reproduced music sound at the time of shooting and the video and the residual sound. And can play.

  In the second embodiment, as in the case of the first embodiment, when the simultaneous playback display is designated in the synchronous playback command, the display control unit 740 displays the synchronized video on the display unit 746. The display unit 747 displays the real-time video imaged by the imaging unit 710. For this reason, it is possible to simultaneously display the past video and the current video synchronized with the same reproduced music sound.

  Further, in the second embodiment, as in the case of the first embodiment, the generation unit 730B synchronizes with the progress of the reproduced music sound being reproduced and output every time a synchronous video switching command is received during the synchronous reproduction. Then, edit synchronization information for switching to the synchronized video specified in the synchronized video switching command is generated. When receiving a synchronous video switching command during synchronous playback, the display control means 740 switches the synchronous video specified in the synchronous video switching command and causes the display means 746 to display it. For this reason, so-called timeline editing of the past video synchronized with the same reproduced music sound can be easily performed.

[Modification of Embodiment]
The present invention is not limited to the first and second embodiments described above, and various modifications are possible.

  For example, in the first and second embodiments, one photographing unit is prepared and a real-time image can be displayed by one display unit. On the other hand, a plurality of photographing means are prepared, and a plurality of display means for displaying real-time video are prepared, and a video photographed by one or more photographing means selected from the plurality of photographing means is displayed as one. You may enable it to display simultaneously using the above display means.

  In the first and second embodiments, one video to be synchronously displayed is synchronously displayed. However, one or more selected video data is prepared after preparing a plurality of display means for synchronous display. Synchronous video based on the above may be displayed simultaneously.

  In the first and second embodiments described above, the reproduction data corresponding to the edit synchronization information (the reproduction video data corresponding to the edit synchronization information in the first embodiment, the edit synchronization information in the second embodiment). Corresponding reproduction video data and reproduction residual sound data) may be generated. In this case, the edit synchronization information and the reproduction data corresponding to the edit synchronization information can be supplied to another reproduction apparatus using the recording medium or the communication line.

  In the first and second embodiments described above, when the synchronized video is displayed, it may be possible to feed and beat back.

  Further, in the first embodiment, sound collecting means is further provided, and in the same manner as in the second embodiment, the reproduction residual sound data is included in the reproduction information, and at the time of synchronous reproduction. A synthesized sound obtained by synthesizing the reproduced music sound and the remaining sound may be reproduced and output.

  In the first embodiment, the generation unit generates the reproduction synchronization information based on the detection results of the reception times of various commands sent from the input unit. On the other hand, the generation means receives the progress information of the reproduction music sound from the sound output control means and receives the progress information of the video acquisition operation from the acquisition means, and based on the received information, the reproduction synchronization information May be generated.

  In the second embodiment, it is assumed that the music content corresponding to the music sound is stored in the sound source data section. On the other hand, when it is found that the music content corresponding to the music sound is not stored in the sound source data section, the generation means downloads the music content corresponding to the music sound from the music server on the Internet. The sound source data portion may be stored.

  In the first and second embodiments described above, for example, a synchronized video or a video obtained by inverting a real-time video can be projected and displayed in order to make it look the same as a landscape reflected in a mirror seen during dance practice. You may do it.

  In the first and second embodiments described above, the synchronous playback device is provided with a photographing unit, a display unit, a sound source data unit, and a sound output unit (in the second embodiment, a sound collection unit). On the other hand, when another device has some or all of these functions, the functions of the other device may be used.

  Further, the synchronized playback device of the first and second embodiments is configured as a computer device including a computer as a calculation unit, and the functions of the acquisition unit, the generation unit, the display control unit, and the sound output control unit described above. Part or all of the functions can be realized by executing a program. These programs may be acquired in a form recorded on a portable recording medium such as a CD-ROM or DVD, or may be acquired in a form distributed via a network such as the Internet. Can do.

  Embodiments of the synchronized playback apparatus according to the present invention will be described below. In the following description and drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant description is omitted.

[First embodiment]
First, a first embodiment of the synchronous playback apparatus of the present invention will be described with reference mainly to FIGS.
<Configuration>
FIG. 3 shows a schematic configuration of the synchronized playback device 100A according to the first embodiment. Note that the synchronized playback apparatus 100A is an aspect of the synchronized playback apparatus 700A (see FIG. 1) of the first embodiment described above.

As shown in FIG. 3, the synchronized playback device 100A includes a control unit 110A as an acquisition unit 720A, a generation unit 730A, a display control unit 740, and a sound output control unit 760A, and a storage unit 120 as a playback information storage unit 735. And. The synchronized playback apparatus 100 also includes a sound source unit 130 as a sound source data unit 750, a sound output unit 140 as a sound output unit 765, a projection unit 150 ₁ as a display unit 746, and a projection unit 150 ₂ as a display unit 747. And. Furthermore, the synchronized playback device 100A includes an information display unit 160, an operation input unit 170 as an input unit 770, and a shooting unit 180 as a shooting unit 710.

  The control unit 110A performs overall control of the entire synchronized playback device 100A and executes various processes. The control unit 110A will be described later.

  The storage unit 120 includes a hard disk device that is a nonvolatile storage device. The storage unit 120 stores reproduction information RDA. The contents of the reproduction information RDA will be described later.

  The sound source unit 130 includes a nonvolatile storage element. This sound source unit stores music content information MCD related to various music.

  The sound output unit 140 includes a speaker. The sound output unit 140 receives output sound (reproduced music sound in the first embodiment) data sent from the control unit 110A. The sound output unit 140 reproduces and outputs an output sound corresponding to the output sound data under the control of the control unit 110A.

Projection unit 150 ₁ of the above is configured to include a video projector. The projection unit 150 ₁ receives the synchronized video data sent from the control unit 110A. The projection unit 150 _1, under the control of the control unit 110A, a synchronous video corresponding to the synchronized video data, projected and displayed on a screen or wall.

Similar to the projection unit 150 ₁ , the projection unit 150 ₂ includes a video projector. The projection unit 150 ₂ is subjected to a real-time image data sent from the control unit 110A. The projection unit 150 _2, under the control of the control unit 110A, a real time image corresponding to the real-time image data, projected and displayed on a screen or wall.

  The information display unit 160 includes a display device such as a liquid crystal panel. The information display unit 160 receives information display data sent from the control unit 110A. Then, the information display unit 160 displays an image corresponding to the information display data. The information display unit 160 displays operation state information, operation guidance information, and the like of the synchronized playback device 100A under the control of the control unit 110A.

  The operation input unit 170 includes a key unit provided in the main body of the synchronous playback device 100A and / or a remote input device including the key unit. Here, as a key part provided in the main body part, a touch panel provided in a display device of the information display unit 160 can be used. In addition, it can replace with the structure which has a key part, or can also employ | adopt the structure input with a sound using a voice recognition technique in combination. In addition, image recognition for a photographed image by a camera and motion recognition by motion capture can be used for operation input.

  When the user operates the operation input unit 170, the operation content of the synchronized playback device 100A is set and an operation command is input. Such operation designation includes an acquisition operation start command and an end command, a playback synchronization information generation command, a music sound playback start command and an end command, a synchronized playback start command and an end command, a playback synchronization information editing start command and an end command. A command and a synchronous video switching command are included. The content input to the operation input unit 170 is sent to the control unit 110A as operation input data.

  The photographing unit 180 includes a photographing camera. The photographing unit 180 photographs a peripheral video under the control of the control unit 110A. The result of photographing by the photographing unit 180 is sent to the control unit 110A as video data.

  Next, the control unit 110A will be described. The control unit 110A includes a central processing unit (CPU) and its peripheral circuits. When the control unit 110A executes various programs, the functions as the acquisition unit 720A, the generation unit 730A, the display control unit 740, and the sound output control unit 760A described above are realized. Details of the processing executed by the control unit 110A will be described later.

  When the control unit 110A receives the music sound playback start command sent from the operation input unit 170, the control unit 110A starts the music playback process. In such music playback processing, the music content data specified in the music sound playback start command is sequentially read from the sound source unit 130. Then, the control unit 110 </ b> A generates reproduced music sound data based on the read music content data, and sends it to the sound output unit 140. As a result, the reproduced music sound corresponding to the reproduced music sound data is reproduced and output from the sound output unit 140.

  Next, the content of the reproduction information RDA stored in the storage unit 120 will be described. As shown in FIG. 4, the reproduction information RDA is composed of synchronous reproduction information and edit reproduction information. Here, the synchronized playback information can include individual synchronized playback information # 1, individual synchronized playback information # 2,. Each of the individual synchronized playback information #j (j = 1, 2,...) Is composed of playback video data #j and playback synchronization information #j. Further, the editing / playback information can include editing synchronization information # 1, editing synchronization information # 2,.

[Operation]
Next, the operation of the synchronized playback apparatus 100A configured as described above will be described mainly focusing on the processing of the control unit 110A during the playback synchronization information generation processing, synchronized playback processing, and edit synchronization information generation processing. .

<< Playback synchronization information generation process >>
In the reproduction synchronization information generation process, as shown in FIG. 5, first, in step S11, the control unit 110A determines whether or not a reproduction synchronization information generation instruction has been received. If the result of this determination is negative (step S11: N), the process of step S11 is repeated.

  If the user performs an input operation of the reproduction synchronization information generation command to the operation input unit 170 during the repetition of the process of step S11, the reproduction synchronization information generation command is transmitted from the operation input unit 170 to the control unit 110A. Sent. When the reproduction synchronization information generation instruction is received and the result of determination in step S11 is affirmative (step S11: Y), the process proceeds to step S12.

  When receiving a music sound playback or synchronized playback start command during the repetition of the process of step S11, the control unit 110A performs music sound playback or synchronization as preparation for the subsequent generation of playback synchronization information. The progress of reproduction of the reproduced music sound performed in response to the reproduction start command is monitored.

  In step S12, the control unit 110A monitors reception of an acquisition operation start command and an end command, and a music sound playback or synchronized playback start command and an end command in a period until receiving the acquisition operation end command, Each reception time is detected. In addition, the control unit 110A uses the storage unit 120 to perform buffering processing of the video data sent from the imaging unit 180 during the acquisition period from reception of the acquisition operation start command to reception of the end command.

  Next, in step S13, the control unit 110A determines whether or not an acquisition operation end command has been received. If the result of this determination is negative (step S13: N), the process returns to step S12. Then, the detection of the command reception time and the video data buffering process are continued.

  If the user inputs an acquisition operation end command to the operation input unit 170 during the repetition of the processes of steps S12 and S13, the acquisition operation end command is sent from the operation input unit 170 to the control unit 110A. Sent. When the acquisition operation end command is received and the result of determination in step S13 is affirmative (step S13: Y), the process proceeds to step S14.

  In step S14, the control unit 110A stores the buffered video data in the storage unit 120 as new playback video data. Subsequently, the control unit 110A determines that the sound output unit during the current acquisition period based on the detection result of the reception time of the music sound playback or synchronized playback start command or the monitoring result before the playback synchronization information generation process described above. 140, using the playback start time of the playback music sound output from 140 as the reference time, the reference time of the acquisition operation start command, the acquisition operation end command, the music sound playback or synchronized playback end command Relative time information with respect to time is calculated.

  Note that the reception time of the music sound playback or synchronous playback end command may or may not be detected between the reception of the playback synchronization information generation command and the reception of the acquisition operation end command. is there. For this reason, when the reception time of the music sound playback or synchronous playback end command is detected, the relative time information of the reception time of the music sound playback or synchronous playback end command is calculated. When the reception time of the end command for synchronous playback is not detected, the relative time information of the reception time of the end command for music sound playback or synchronous playback is not calculated.

  Next, the control unit 110A adds the reproduction video data acquired this time and the music content designation information designated by the music sound reproduction or synchronous reproduction start command to the calculated relative time information, Generate synchronization information. Then, the control unit 110A stores the generated reproduction synchronization information in the storage unit 120 as new reproduction synchronization information. Thereafter, the control unit 110A ends the reproduction synchronization information generation process.

  New individual synchronized playback information is composed of the new playback video data and the new playback synchronization information.

《Synchronized playback processing》
As the synchronized playback process, the synchronized video display process and the synchronized sound output process are executed in parallel.

  In the synchronous video display process, as shown in FIG. 6, first, in step S21, the control unit 110A determines whether or not a synchronous reproduction start command has been received. If the result of this determination is negative (step S21: N), the process of step S21 is repeated.

  If the user performs an operation of inputting a synchronous reproduction start command to the operation input unit 170 while the process of step S21 is repeated, the synchronous reproduction start command is sent from the operation input unit 170 to the control unit 110A. . When the synchronous reproduction start command is received and the result of determination in step S21 is affirmative (step S21: Y), the process proceeds to step S22.

In step S22, the control unit 110A determines the playback video data to be played back at each time point according to the synchronization relationship with the playback music information indicated by the playback synchronization information or the edit synchronization information specified in the synchronous playback start command. The partial data is read from the storage unit 120 sequentially. Subsequently, the control unit 110A generates synchronized video data based on the read data. Then, the control unit 110A sends the generated synchronized video data to the projection unit 150 _1. As a result, the projection unit 150 _1, a screen or wall, synchronized video is projected and displayed.

Further, when simultaneous display is specified in the synchronous playback start command, the control unit 110A displays real-time video that is being shot based on video data sent from the shooting unit 180. Generate real-time video data. Then, the control unit 110A sends the generated real-time image data to the projection unit 150 _2. As a result, simultaneously with the projection display of the synchronized video by the projection unit 150 _1, the projection display of the real-time video by the projection unit 150 ₂ is performed.

  Next, in step S23, the control unit 110A determines whether or not an instruction to end synchronous reproduction has been received. If the result of this determination is negative (step S23: N), the process returns to step S22. Then, the synchronized video display process is continued.

  If the user performs an operation of inputting a synchronous reproduction end command to the operation input unit 170 during the repetition of steps S22 and S23, the synchronous reproduction end command is transmitted from the operation input unit 170 to the control unit 110A. Sent. When the synchronous reproduction end command is received and the result of determination in step S23 is affirmative (step S23: Y), the control unit 110A ends the synchronous video display process.

  In the synchronous sound output process, as shown in FIG. 7, first, in step S31, the control unit 110A determines whether or not a synchronous reproduction start command has been received. If the result of this determination is negative (step S31: N), the process of step S31 is repeated.

  If the user performs an operation of inputting a synchronous reproduction start command to the operation input unit 170 during the repetition of the process of step S31, a synchronous reproduction start command is sent from the operation input unit to the control unit 110A. When the synchronous reproduction start command is received and the result of the determination in step S31 is affirmative (step S31: Y), the process proceeds to step S32.

  In step S32, the control unit 110A refers to the reproduction synchronization information or the edit synchronization information designated by the synchronous reproduction start command, and sequentially transmits the data of the music content portion to be reproduced at each time point from the sound source unit 130. read. Subsequently, the control unit 110A generates reproduction music sound data based on the read data. Then, the control unit 110A sends the generated reproduction music sound data to the sound output unit 140. As a result, the reproduced music sound is reproduced and output from the sound output unit 140.

  Next, in step S33, the control unit 110A determines whether or not a synchronous reproduction end command has been received. If the result of this determination is negative (step S33: N), the process returns to step S32. Then, the synchronized sound output process is continued.

  If the user performs an operation of inputting a synchronous playback end command to the operation input unit 170 during the repetition of steps S32 and S33, the synchronous playback end command is sent from the operation input unit 170 to the control unit 110A. Sent. When the synchronous reproduction end command is received and the result of determination in step S33 is affirmative (step S33: Y), the control unit 110A ends the synchronous sound output processing.

《Edit synchronization information generation processing》
In the editing synchronization information generation process, as shown in FIG. 8, first, in step S41, the control unit 110A determines whether or not a reproduction synchronization information editing start command has been received. If the result of this determination is negative (step S41: N), the process of step S41 is repeated.

  If the user performs an input operation of the reproduction synchronization information editing start command to the operation input unit 170 during the repetition of the process of step S41, the reproduction synchronization information editing start command is transmitted from the operation input unit 170 to the control unit. 110A. When the reproduction synchronization information editing start command is received and the result of determination in step S41 is affirmative (step S41: Y), the process proceeds to step S42.

  In step S42, the control unit 110A specifies the reproduction synchronization information or the edit synchronization information specified in the reproduction synchronization information editing start command. Then, the control unit 110A reads the identified synchronization information from the storage unit 120 as temporary editing synchronization information.

  Next, in step S43, the control unit 110A determines whether or not a reproduction synchronization information editing end command has been received. If the result of this determination is negative (step S43: N), the process proceeds to step S44.

  In step S44, control unit 110A determines whether or not a synchronous video switching command has been received. If the result of this determination is negative (step S44: N), the process returns to step S43.

  If the user performs an input operation of the synchronous video switching command to the operation input unit 170 during the repetition of the processes of steps S43 and S44, the synchronous video switching command is sent from the operation input unit 170 to the control unit 110A. . When the synchronous video switching command is received and the determination result in step S44 is affirmative (step S44: Y), the process proceeds to step S45.

  In step S45, the control unit 110A specifies the relative time of the current time with respect to the time corresponding to the reference time related to the synchronous reproduction process (the start time of the reproduced music sound output) during the editing synchronization information generation process this time. Subsequently, the control unit 110A updates the provisional editing synchronization information at that time by adding the specified relative time information and the specifying information of the reproduction video data specified in the synchronous video switching command. Then, the process returns to step S43.

  When the provisional editing synchronization information read at the start of editing the reproduction synchronization information is editing synchronization information, the control unit 110A receives the first synchronized video switching command after that, and then receives the first synchronized video switching command. The temporary editing synchronization information after the relative time corresponding to the received time is deleted.

  When the user performs an input operation of a reproduction synchronization information editing end command to the operation input unit 170 during the repetition of steps S43 to S45, a reproduction synchronization information editing end command is transmitted from the operation input unit 170 to the control unit. 110A. When the reproduction synchronization information editing end command is received and the result of the determination in step S43 is affirmative (step S43: Y), the process proceeds to step S46.

  In step S46, control unit 110A determines whether or not provisional editing synchronization information has been updated in response to reception of the synchronous video switching command. If the result of this determination is negative (step S46: N), the control unit 110A immediately ends the editing synchronization information generation process.

  On the other hand, when the result of the determination in step S46 is affirmative (step S46: Y), the process proceeds to step S47. In step S47, the control unit 110A determines the temporary editing synchronization information at that time as new editing synchronization information. Then, the control unit 110A stores the new editing synchronization information in the storage unit 120. Thereafter, the control unit 110A ends the editing synchronization information generation process.

  Note that the control unit 110A performs a synchronized video switching display process and a synchronized sound output process in parallel with the edit synchronization information generation process described above.

  In the synchronized video switching display process and the synchronized sound output process, as shown in FIG. 9, first, in step S51, the control unit 110A determines whether or not a reproduction synchronization information editing start command has been received. If the result of this determination is negative (step S51: N), the process of step S51 is repeated.

  If the user performs an input operation of the reproduction synchronization information editing start command to the operation input unit 170 during the repetition of the process of step S51, the reproduction synchronization information editing start command is transmitted from the operation input unit 170 to the control unit. 110A. When the reproduction synchronization information editing start command is received and the result of determination in step S51 is affirmative (step S51: Y), the process proceeds to step S52.

  In step S52, the control unit 110A refers to the reproduction synchronization information or the edit synchronization information designated by the reproduction synchronization information editing start command, and performs the synchronized video display process and the process in the same manner as the above-described synchronized reproduction process. Performs synchronized sound output processing.

  Next, in step S53, the control unit 110A determines whether or not a reproduction synchronization information editing end command has been received. If the result of this determination is negative (step S53: N), the process proceeds to step S54.

  In step S54, control unit 110A determines whether or not a synchronous video switching command has been received. If the result of this determination is negative (step S54: N), the process returns to step S53.

  If the user performs an input operation of the synchronous video switching command to the operation input unit 170 during the repetition of the processes of steps S53 and S54, the synchronous video switching command is sent from the operation input unit 170 to the control unit 110A. . When the synchronous video switching command is received and the result of determination in step S54 is affirmative (step S54: Y), the process proceeds to step S55.

In step S55, the control unit 110A refers to the playback synchronization information corresponding to the video data designated by the synchronized video switching command, and plays back after switching the video while maintaining the synchronization with the playback timing of the playback music sound so far. The part of the designated video data to be specified is specified. Subsequently, the control unit 110A sequentially reads the data of the specified video data portion, and generates synchronized video data. Then, the control unit 110A sends the generated synchronized video data to the projection unit 150 _1. As a result, the synchronized video projected and displayed by the projection unit 150 ₁ is switched. Thereafter, the process returns to step S53.

  If the user performs an input operation of a reproduction synchronization information editing end command to the operation input unit 170 during the repetition of steps S53 to S55, a reproduction synchronization information editing end command is sent from the operation input unit 170 to the control unit. 110A. When the reproduction synchronization information editing end command is received and the determination result in step S53 is affirmative (step S53: Y), the control unit 110A ends the synchronized video switching display process and the synchronized sound output process.

As described above, in the first embodiment, the control unit 110A acquires the video data synchronized with the reproduced music sound photographed by the photographing unit 180. Subsequently, the control unit 110A determines the synchronization relationship between the playback timing of the playback music sound at the time of shooting and the playback timing of the playback video based on the acquired video data, and the synchronization relationship between the playback music sound and the video at the time of shooting. The reproduction synchronization information for making the same is generated. Referring to the reproduction synchronization information thus generated, the control unit 110A, along with to project display synchronized video based on the acquired image data to the projection unit 150 _1, to output the reproduced music sound from the sound output unit 140.

  Therefore, according to the first embodiment, the reproduced music sound and the photographed video can be reproduced after photographing in the same synchronization relationship as that between the reproduced music sound and the video at the time of photographing. .

Further, in this first embodiment, when the simultaneous reproduction display is specified in the synchronized playback command, the control unit 110A is, while the projection display by the projection unit 150 ₁ synchronized video, is photographed by the photographing unit 180 the real-time image is projected and displayed by the projection unit 150 _2. For this reason, it is possible to simultaneously display the past video and the current video synchronized with the same reproduced music sound.

Further, in the first embodiment, every time a synchronized video switching command is received during synchronized playback upon generation of editing synchronization information, the control unit 110A synchronizes with the progress of the reproduced music sound being reproduced and output. Edit synchronization information for switching to the synchronized video specified in the switching command is generated. Further, when receiving the synchronization synchronized video switching instruction during playback of in generating the editing synchronization information, control unit 110A is switched to the image specified in the synchronized video switching command, it is projected and displayed by the projection unit 150 _1. For this reason, so-called timeline editing of the past video synchronized with the same reproduced music sound can be easily performed.

[Second Embodiment]
Next, a second embodiment according to the present invention will be described with reference mainly to FIGS.

  FIG. 10 shows a schematic configuration of a synchronized playback device 100B according to the second embodiment. The synchronized playback apparatus 100B is an aspect of the synchronized playback apparatus 700B (see FIG. 2) of the second embodiment described above.

  As shown in FIG. 10, the synchronized playback apparatus 100B includes a control unit 110B instead of the control unit 110A and a sound collection unit 190, as compared with the synchronized playback apparatus 100A of the first embodiment described above. The storage unit 120 also functions as the acquisition information storage unit 725, and the storage unit 120 is different in that the acquisition information AQD and the reproduction information RDB are stored. Hereinafter, description will be given mainly focusing on these differences.

  Similar to the control unit 110A described above, the control unit 110B includes a central processing unit and its peripheral circuits. When the control unit 110B executes various programs, the functions as the acquisition unit 720B, the generation unit 730B, the display control unit 740, and the sound output control unit 760B described above are realized. Details of the processing executed by the control unit 110B will be described later.

  The sound collection unit 190 is configured to include a microphone. The sound collection unit 190 collects ambient sounds under the control of the control unit 110B. The sound collection result by the sound collection unit 190 is sent to the control unit 110B as sound collection data.

  As shown in FIG. 11, the acquired information AQD is composed of acquired video data, acquired musical sound data, and acquired residual sound data. Further, as shown in FIG. 12, the reproduction information RDB is different from the reproduction information RDA in the first embodiment described above (see FIG. 4) as individual synchronized reproduction information #j (j = 1, 2,... Only differs in that it further includes reproduction residual sound data #j.

<Operation>
Next, with regard to the operation of the synchronized playback apparatus 100B configured as described above, the control unit 110B mainly focuses on the acquisition operation, the playback synchronization information generation process, the synchronized playback process, and the edit synchronization information generation process. I will explain.

<< Acquisition process >>
In the acquisition process, as shown in FIG. 13, first, in step S61, the control unit 110B determines whether or not an acquisition operation start command has been received. If the result of this determination is negative (step S61: N), the process of step S61 is repeated.

  When the user inputs an acquisition operation start command to the operation input unit 170 during the repetition of the process of step S61, the acquisition operation start command is sent from the operation input unit 170 to the control unit 110B. . When this acquisition operation start command is received and the result of determination in step S61 is affirmative (step S61: Y), the process proceeds to step S62.

  In step S <b> 62, the control unit 110 </ b> B receives the video data sent from the imaging unit 180 and the sound collection data sent from the sound collection unit 190 during the acquisition period until receiving the acquisition operation end command, and the storage unit 120. Perform buffering using. Subsequently, in step S63, the control unit 110B determines whether or not an acquisition operation end command has been received. If the result of this determination is negative (step S63: N), the process returns to step S62. Then, the buffering process of the video data and the collected sound data is continued.

  When the user performs an input operation of an acquisition operation end command to the operation input unit 170 during the repetition of the processes of steps S62 and S63, the acquisition operation end command is transmitted from the operation input unit 170 to the control unit 110B. Sent. When the acquisition operation end command is received and the result of determination in step S63 is affirmative (step S63: Y), the process proceeds to step S64.

  In step S64, the control unit 110B analyzes the collected sound data buffered in the storage unit 120, identifies the music sound and the remaining sound other than the music, and makes the music sound data and the remaining sound data. To separate. Then, the control unit 110B uses the video data buffered in the storage unit 120 and the separated music sound data and residual sound data as the acquired video data, acquired musical sound data, and acquired residual sound data. Store in the storage unit 120. Thereafter, the control unit 110B ends the acquisition process.

<< Playback synchronization information generation process >>
In the reproduction synchronization information generation process, as shown in FIG. 14, first, in step S71, the control unit 110B determines whether or not a reproduction synchronization information generation instruction has been received. If the result of this determination is negative (step S71: N), the process of step S71 is repeated.

  If the user performs an input operation of the reproduction synchronization information generation command to the operation input unit 170 during the repetition of the process of step S71, the reproduction synchronization information generation command is transmitted from the operation input unit 170 to the control unit 110B. Sent. When the reproduction synchronization information generation command is received and the determination result in step S71 is affirmative (step S71: Y), the process proceeds to step S72.

  In step S72, the control unit 110B specifies the music sound. In specifying the music sound, first, the control unit 110B reads the acquired music sound data from the storage unit 120. Subsequently, the control unit 110B analyzes the acquired music sound data and extracts metadata indicating the characteristics of the music sound corresponding to the acquired music sound data. And control unit 110B specifies the music content in the sound source unit 130 corresponding to the said metadata.

  Next, in step S73, the control unit 110B specifies the portion of the acquired music sound data in the specified music content. And based on the part of the acquired music sound data in the specified music content and the identification information of the specified music content, playback synchronization information equivalent to the playback synchronization information generated in the first embodiment is generated. Subsequently, the control unit 110B reads the acquired video data and the acquired residual sound data from the storage unit 120, stores them in the storage unit 120 as new video data for reproduction and residual sound data for reproduction, and the generated reproduction. The synchronization information is stored in the storage unit 120 as new reproduction synchronization information. As a result, new playback video data, playback residual sound data, and playback synchronization information are added to the storage unit 120.

  Next, the control unit 110B stores the acquired video data, the acquired music sound data, and the acquired residual sound data used for generating the new playback video data, the residual sound data for playback, and the playback synchronization information. Erase from within. Thereafter, the control unit 110B ends the reproduction synchronization information generation process.

<< Synchronous playback process and edit synchronization information generation process >>
The synchronized video display process in the synchronized playback process of the second embodiment is performed by the control unit 110B performing the same process as the synchronized video display process by the control unit 110A in the first embodiment described above.

  Further, in the synchronized sound output process in the synchronized playback process of the second embodiment, the control unit 110B performs the above-described first embodiment when the simultaneous playback of the residual sound is not specified in the synchronized playback start command. This is performed by executing the same processing as the synchronous sound output processing by the control unit 110A in the example. On the other hand, when it is specified in the synchronous reproduction start command that the residual sound is simultaneously reproduced, the control unit 110B executes the synchronous sound output process as follows.

  In the synchronous sound output processing in the case where such a residual sound is designated to be simultaneously reproduced, the control unit 110B refers to the reproduction synchronization information or the edit synchronization information designated by the synchronous reproduction start command, and reproduces at each time point. The data of the music content part corresponding to the music sound to be played is sequentially read from the sound source unit 130, and the data of the residual sound data for reproduction to be reproduced at each time point is read from the storage unit 120 sequentially.

  Subsequently, the control unit 110B generates synchronized output sound data corresponding to the synthesized sound of the reproduced music sound and the residual sound based on the two types of read data. Then, the control unit 110B sends the generated synchronized output sound data to the sound output unit 140. As a result, the sound output unit 140 reproduces and outputs a synthesized sound of the reproduced music sound and the remaining sound.

  Regardless of the designation of whether or not the residual sound is to be simultaneously reproduced in the synchronous reproduction start command, when receiving the synchronous reproduction end command, the control unit 110B ends the synchronous sound output process.

  The generation of the edit synchronization information in the second embodiment is performed by the control unit 110B executing the same process as the edit synchronization information generation process by the control unit 110A in the first embodiment described above.

  As described above, in the second embodiment, the control unit 110B receives the video data synchronized with the music sound photographed by the photographing unit 180 and the sound data collected by the sound collecting unit 190. Subsequently, the control unit 110B analyzes the collected voice data, identifies the music sound and the remaining sound other than the music, and separates the music sound data and the remaining sound data. Then, the control unit 110B acquires video data, music sound data, and residual sound data in a period from when the acquisition operation start command is received until the acquisition operation end command is received.

Next, the control unit 110B sets the playback timing of the music sound at the time of shooting, the playback timing of the playback video based on the acquired video data, and the playback timing of the playback video based on the acquired residual sound data. Reproduction synchronization information is generated to make the synchronization relationship the same as the synchronization relationship between the music sound at the time of shooting, the video, and the residual sound. Referring to the reproduction synchronization information thus generated, the sound control unit 110B is, causes projected and displayed synchronized video based on the acquired image data by the projection unit 150 _1, a synthesized sound of the reproduced music sound and a residual noise Output from the output unit 140.

  Therefore, according to the second embodiment, the reproduced music sound, the captured video image, and the residual sound are recorded after shooting in the same synchronization relationship as that of the reproduced music sound at the time of shooting and the video and residual sound. And can play.

Further, in the second embodiment, as in the case of the first embodiment, when the simultaneous reproduction display is specified in the synchronized playback command, the control unit 110B is then projected and displayed by the projection unit 150 ₁ synchronized video while, is projected and displayed by the projection unit 150 ₂ real-time image being photographed by the photographing unit 180. For this reason, it is possible to simultaneously display the past video and the current video synchronized with the same reproduced music sound.

In the second embodiment, as in the case of the first embodiment, the control unit 110B plays the reproduced music being reproduced and output every time a synchronous video switching command is received during the synchronous reproduction when generating the edit synchronous information. In synchronization with the progress of the sound, edit synchronization information for switching to the synchronized video specified in the synchronized video switching command is generated. Further, when receiving the synchronization synchronized video switching instruction during playback of in generating the editing synchronization information, control unit 110B switches the specified image in the synchronized video switching command, it is projected and displayed by the projection unit 150 _1. For this reason, so-called timeline editing of the past video synchronized with the same reproduced music sound can be easily performed.

[Modification of Example]
The present invention is not limited to the first and second embodiments described above, and various modifications are possible.

  For example, in the first and second embodiments, the projection unit including the projection projector is used for the synchronized video and the real-time video. However, a display unit including a display device such as a liquid crystal display panel may be used. .

  Further, the modifications described for the first and second embodiments described above may be applied to the first and second examples.

  In the above embodiment, the process for the synchronized playback is performed by executing the program by the control units 110A and 110B. However, all or a part of the process for the synchronized playback is performed by a dedicated LSI. You may make it carry out with the hardware using (Large Scale Integrated circuit) etc.

100A, 100B ... Synchronous playback device 110A, 110B ... Control unit (acquisition means, generation means, display control means,
Sound output control means)
140 ... Sound output unit (sound output unit)
150 ₁ ... Projection unit (display means)
150 ₂ ... Projection unit (display means (specific display means))
170 ... Operation input unit (input means)
180 ... photographing unit (photographing means)
190 ... Sound collecting unit (sound collecting means)
700A, 700B ... Synchronous playback device 710 ... Imaging means 715 ... Sound collection means 720A, 720B ... Acquisition means 730A, 730B ... Generation means 740 ... Display control means 745 ... Display means 746 ... Display means 747 ... Display means (specific display means)
760A, 760B ... Sound output control means 765 ... Sound output section 770 ... Input means

Claims (11)

A synchronized playback device that displays video in synchronization with the playback music sound,
Obtaining means for obtaining reproduction data including video data synchronized with the music sound;
The synchronization relationship between the playback timing of the playback music sound based on the music content corresponding to the music sound and the playback timing based on the acquired playback data is the same as the synchronization relationship between the music sound and the video Generating means for generating reproduction synchronization information;
Sound output control means for outputting a sound including the reproduced music sound from a sound output unit;
Comprising a,; based on the generated reproduction synchronization information, wherein while achieving synchronization between the reproduction timing of the reproduced music sound, and a display control means for displaying a reproduced image based on data of the obtained image on the display unit
The generation means further analyzes the music sound and further specifies music content corresponding to the music sound, and extracts the reproduction synchronization information using the specific result.
A synchronized playback apparatus characterized by the above. It further comprises a photographing means for photographing the surrounding video,
The acquisition means acquires data of a photographing result by the photographing means during the output from the sound output unit of the reproduced music sound as the reproduction data.
The synchronized playback apparatus according to claim 1, wherein: The reproduction data further includes residual sound data excluding the music sound included in the background sound of the video,
The sound output control means synthesizes the reproduced music sound and the remaining sound while synchronizing with the reproduction timing of the reproduced music sound output from the sound output unit based on the generated reproduction synchronization information. To output the synthesized sound from the sound output unit,
The synchronous reproduction apparatus according to claim 1 or 2, wherein Photographing means for photographing peripheral images;
A sound collecting means for collecting the surrounding sound;
The acquisition means includes data of a photographing result by the photographing means during output of the reproduced music sound from the sound output unit, and a residual obtained by removing the music sound from the sound collecting result by the sound collecting means Sound data is acquired as the playback data,
The sound output control means synthesizes the reproduced music sound and the remaining sound while synchronizing with the reproduction timing of the reproduced music sound output from the sound output unit based on the generated reproduction synchronization information. To output the synthesized sound from the sound output unit,
The synchronized playback apparatus according to claim 1, wherein: The display unit includes a plurality of display means capable of displaying images independently of each other.
The display control means displays the real-time video being shot by the shooting means on a specific display means that is at least one of the plurality of display means, and displays a playback video based on the acquired video data. Display on at least one display means other than the display means;
The synchronized playback apparatus according to claim 2 or 4, It further comprises a sound collecting means for collecting the surrounding sound,
The synchronized playback apparatus according to claim 1 , wherein the generation unit analyzes the music sound included in the sound collection result of the sound collection unit. It further comprises a photographing means for photographing the surrounding video,
The display unit includes a plurality of display means capable of displaying images independently of each other.
The display control means displays the real-time video being shot by the shooting means on a specific display means that is at least one of the plurality of display means, and displays a playback video based on the acquired video data. Display on at least one display means other than the display means;
The synchronous playback apparatus according to claim 1 or 6 , It further comprises an input means for inputting a playback content designation,
When there are a plurality of types of reproduction data acquired by the acquisition unit, the generation unit reproduces the reproduction according to a reproduction content designation input to the input unit during output of the reproduced music sound from the sound output unit. Further extracting edit synchronization information for specifying playback data to be played back in synchronization with the music sound;
The synchronized playback apparatus according to claim 1 , wherein A synchronized playback method used in a synchronized playback device that displays video in synchronization with the playback music sound,
An acquisition step of acquiring reproduction data including video data synchronized with the music sound;
The synchronization relationship between the playback timing of the playback music sound based on the music content corresponding to the music sound and the playback timing based on the acquired playback data is the same as the synchronization relationship between the music sound and the video A generation step of generating reproduction synchronization information;
A sound output control step of outputting a sound including the reproduced music sound from a sound output unit;
Comprising a,; based on the generated reproduction synchronization information, wherein while achieving synchronization between the reproduction timing of the reproduced music sound, the display control step of displaying the reproduced image based on data of the obtained image on the display unit
In the generating step, the music sound is analyzed to further specify music content corresponding to the music sound, and the reproduction synchronization information is extracted using the specific result.
And a synchronized playback method. A synchronous reproduction program for causing a calculation means to execute the synchronous reproduction method according to claim 9 . 11. A recording medium in which the synchronous reproduction program according to claim 10 is recorded so as to be readable by an arithmetic means.

Images