JP2010252008A - Imaging device, displaying device, reproducing device, imaging method and displaying method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging device, a displaying device, a reproducing device, an imaging method and a displaying method, which allow a user to enjoy captured moving images easily and immediately. <P>SOLUTION: The device includes: an imaging section 2 for outputting image data of moving images; a face detecting section 3 for determining whether or not a face is included in an image based on the image data; a face database 4b in which data for identification for identifying a face is stored; a representative image-determining section 1b for determining whether the face included in the image data matches the data for identification based on the determination result by the face detecting section 3 and the data for identification stored in the face database 4b and for deciding that it should be a representative image when both are matched; and a displaying section 8 for displaying the representative image decided by the representative image-determining section 1b. From image data of a plurality of moving images, a commonly appearing person 26 is displayed at the end of the moving images. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an imaging device, a display device, a playback device, an imaging method, and a display method, and more specifically, a so-called video camera, digital camera, camera-equipped mobile phone, etc. The present invention relates to a display device, a playback device, and a display method for displaying or playing back these moving images.

  In recent years, the recording capacity of digital devices has increased, and it has become easier to record a large amount of moving images. However, these recorded moving images are often left unedited, and there is a need for a technique that enables easy handling and enjoyment of moving images during playback.

  Therefore, for example, in Patent Document 1, by extracting the number, position, and size of the faces of the characters from the video as face posture data, and using this extraction result, the characters are classified. A video classification device that enables image retrieval is disclosed. Japanese Patent Application Laid-Open No. 2004-228561 discloses an imaging apparatus that records an ending image at the end of a moving image that has been shot.

International Publication No. 2006/025272 JP 2008-301357 A

  As described above, various proposals have been made heretofore, but all of them are based on a complicated operation process, and it has been impossible to immediately enjoy the captured moving image.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a photographing device, a display device, a reproducing device, a photographing method, and a display method capable of easily and immediately enjoying a captured moving image. And

  In order to achieve the above object, a photographing apparatus according to a first aspect of the present invention is a photographing apparatus having a photographing mode and a reproduction mode, a recording unit for recording a photographed moving image during photographing in the photographing mode, and during photographing in the photographing mode. A first determination unit for determining a person appearing in the captured moving image and recording the determination result in the recording unit; and a plurality of shooting operations when collecting and reproducing a plurality of the captured moving images during reproduction in the reproduction mode. A second determination unit that determines a person who appears in common, and a reproduction unit that uses the determination result of the second determination unit to extract a part of the moving image and sequentially reproduce it.

An imaging apparatus according to a second invention includes an imaging unit that captures an image, a moving image recording unit that records an image continuously acquired from the imaging unit as a moving image, and a facial image obtained by the imaging unit. During the shooting operation, the person specifying unit for specifying a person appearing in the moving image recorded in the moving image recording unit and the specified person according to the person specifying result when the recorded moving image is displayed as a thumbnail are included. A display control unit that displays an image as a representative image.
A display device according to a third invention is common to a character determination unit for determining a person appearing in a moving image, an image display unit for displaying a plurality of images side by side, and a plurality of images displayed on the image display unit. And a representative image determination unit that determines a representative image of the moving image according to a determination result of the character determination unit.

  According to a fourth aspect of the present invention, there is provided an imaging device for continuously capturing an image of a subject to obtain a moving image, and among the moving images obtained by the imaging unit, a still image representing a main character of the moving image. A still image selection unit to be selected; and a display control unit that displays the still image and additional information representing the main character in association with each other when the moving image is reproduced.

According to a fifth aspect of the present invention, in the fourth aspect, the additional information representing the character is a result of collation between the image obtained by the imaging unit and a database recorded in the recording unit, It is the obtained person name information.
According to a sixth aspect of the present invention, in the fourth aspect, the still image associated with the additional information is displayed as a thumbnail representing reproduction or a moving image at the beginning or end of the moving image.

  According to a seventh aspect of the present invention, there is provided a photographing apparatus according to the fourth aspect, wherein the sound collecting unit picks up the surrounding sound and the sound determination for determining whether or not there is a laughter based on the picked up sound. The main character is a character who can pick up a person who has spoken at a timing before the time when the laughing voice is detected by the voice determination unit.

  A playback device according to an eighth aspect of the present invention relates to a plurality of moving images, a first main subject determination unit that determines a main subject included in each moving image, and a first main subject group included in the plurality of moving images. A second main subject determination unit that determines a second main subject; and a digest reproduction unit that preferentially reproduces a portion where the second main subject appears.

  A display device according to a ninth aspect of the present invention includes a moving image storage unit that stores image data of a moving image, a determination unit that determines whether or not a main subject exists for the still image that constitutes the moving image, and the determination unit that determines A display unit that determines a representative image based on the main subject and displays the representative image.

In a display device according to a tenth aspect based on the ninth aspect, the display section displays the main subject as a thumbnail for each of a plurality of moving images.
In a display device according to an eleventh aspect based on the ninth aspect, the display section selects a representative image from a plurality of main subjects and displays the representative image.
In a display device according to a twelfth aspect according to the ninth aspect, the main subject is a person that matches a person image registered in advance.

A display device according to a thirteenth invention includes a moving image storage unit that stores image data of a moving image, a determination unit that determines whether or not there is a person registered in advance for the still image constituting the moving image, A display unit that determines a representative image based on the person determined by the determination unit and displays the representative image.
In the display device according to a fourteenth aspect based on the thirteenth aspect, the determination unit determines an image in which a person appearing in common among a plurality of moving images is shown as a representative image.

  An imaging device according to a fifteenth aspect of the invention is an imaging unit that outputs image data of a moving image, a face detection unit that determines whether a face is included in an image based on the image data, a face Based on the face database in which identification data for identification is stored, the determination result by the face detection unit, and the identification data stored in the face database, the face in the image data is the identification data. And a representative image determining unit that determines a representative image if the two match, and a display unit that displays the representative image determined by the representative image determining unit.

  According to a sixteenth aspect of the present invention, there is provided a photographing method for photographing an image, recording continuously obtained images as a moving image, identifying a person appearing in the recorded moving image during a photographing operation, and recording the recorded image. When displaying a moving image as a thumbnail, an image including the specified person is displayed as a representative image according to the person specifying result.

  According to a seventeenth aspect of the present invention, there is provided a display method for determining a person appearing in a moving image, displaying a plurality of images side by side, determining a person who appears in common in the plurality of displayed images, and determining the character Then, the representative image of the moving image is determined.

  According to the present invention, it is possible to provide a photographing device, a display device, a playback device, a photographing method, and a display method capable of easily and immediately enjoying a moving image that has been photographed easily.

It is a block diagram which shows the structure of the camera concerning one Embodiment of this invention. It is a figure which shows the additional information used when selecting the main subject in the camera concerning one Embodiment of this invention. It is a figure which shows the example which extracts a representative image from several moving images in the camera concerning one Embodiment of this invention. It is a figure which shows the other example which extracts a representative image from several moving images in the camera concerning one Embodiment of this invention. It is a figure which shows the example which extracts a representative image from one moving image in the camera concerning one Embodiment of this invention. It is a figure which shows a mode that several moving images are selected and reproduced | regenerated in the camera concerning one Embodiment of this invention. It is a figure which shows a mode that one moving image is selected and partially reproduced in the camera concerning one Embodiment of this invention. It is a flowchart which shows the operation | movement of the camera control of the camera concerning one Embodiment of this invention. It is a flowchart which shows the operation | movement of the image reproduction of the camera concerning one Embodiment of this invention. FIG. 10 is a flowchart showing an image reproduction operation of a camera according to an embodiment of the present invention, and is a continuation of the flowchart shown in FIG. 9. It is a figure which shows the thumbnail display performed by selecting a representative image from several moving images in the camera concerning the modification of one Embodiment of this invention. It is a flowchart which shows the operation | movement of the thumbnail display of the camera concerning the modification of one Embodiment of this invention.

  Hereinafter, a preferred embodiment using a camera to which the present invention is applied will be described with reference to the drawings. FIG. 1 is a block diagram showing an electric circuit of a camera 10 according to an embodiment of the present invention. The camera 10 is a digital camera, and includes an image processing and control unit 1, an imaging unit 2, a face detection unit 3, a recording unit 4, an operation determination unit 6, an audio recording unit 7, a display unit 8, a clock unit 9, and a communication unit 12. Etc.

  The imaging unit 2 includes an imaging lens, an exposure control unit such as a shutter, an imaging element, a driving and reading circuit for the imaging element, and converts the subject image formed by the imaging lens into moving image data by the imaging element. This image data is output to the signal processing and control unit 1 and the face detection unit 3.

  The image processing and control unit 1 controls the entire sequence of the camera 10 according to a stored program. In addition, the image signal output from the imaging unit 2 is captured, and various image processes such as thinning processing, edge enhancement, color correction, and image compression are performed, and live view display, recording in the recording unit 4, and recording on the display unit 8 are performed. Image processing such as playback display is performed. The image processing and control unit 1 includes a representative image determination unit 1b, a composite display unit 1c, and a voice analysis unit 1d.

  As will be described later, the representative image determination unit 1b determines a representative image to be used for thumbnail display, ending, and the like based on the image data of the moving image. The main subject (candidate for representative image) is determined at the time of shooting (the function of “when shooting” in the figure), and the representative image is determined based on the main subject after the shooting is finished and before the start of reproduction (in the figure) "When playing" function). The composition display unit 1c synthesizes the representative image selected for moving image ending at the end of the moving image. The voice analysis unit 1d performs voice analysis using surrounding voice data input from the voice recording unit 7 to be described later, and analyzes, for example, whether the voice data includes a laughing voice and whether the voice is loud or small. .

  The face detection unit 3 inputs image data from the imaging unit 2, and determines whether or not a human face portion is included in the image based on the image data. If the face is included as a result of the determination, the position and size of the face are determined, and a plurality of persons recorded in a face database (face DB) 4b to be described later are determined. It is also determined whether or not the same specific person is included. The face detection unit 3 also determines the smile level when a face is included. The smile level is a numerical value of the degree of smile. The higher the value, the higher the smile level.

  The recording unit 4 records image data of a moving image acquired by the imaging unit 2 and subjected to image processing by the image processing and control unit 1 when a shooting instruction is given by the release button. Further, when recording image data, this additional information is stored in association with information such as the position, size, and characteristics of the face. Further, the additional information includes the analysis result by the voice analysis unit 1d, shooting date / time information acquired by the clock unit 9 described later, and the like. The face DB 4b is a database for registering facial images of the photographer's family and friends and their characteristics. The face DB 4b also registers the person's name in association with the face image.

  The operation determination unit 6 determines an operation state of an operation member such as a release button and outputs the determined operation state to the signal processing and control unit 1. The sound recording unit 7 includes a microphone for collecting surrounding sounds, a sound processing circuit for performing amplification of sound signals, an AD conversion circuit for changing analog sound signals to digital sound data, and the like. Signal processing and output to the voice analysis unit 1d in the control unit 1.

  The display unit 8 performs various displays such as live view display, playback display, and menu display. The display unit 8 also serves as a touch panel and can detect a position touched by a photographer's finger or the like. The clock unit 9 has a calendar function and a clock function, and outputs shooting date / time information during shooting. The communication unit 12 transmits image data or the like to the outside of the camera 10, and the communication means includes communication such as infrared communication, wireless communication, and wired communication, and may be external output using a USB memory or the like.

  Next, image data recorded in the recording unit 4 and additional information recorded accompanying the image data will be described with reference to FIG. FIG. 2A is an example of related information of the image 1 recorded as a moving image, and FIG. 2B shows an example of screen division. In the present embodiment, there are nine divisions of the areas A1 to A9. . In each of the areas A1 to A9, as shown in FIG. 2C, the face size is divided into three stages D1 to D3.

  In FIG. 2A, time (seconds) indicates an elapsed time from the start of shooting for the image 1 of the moving image. As shown in FIG. 2A, at the start of shooting (at 0 seconds), two faces appear in the image. One of them is the face size D2 (see FIG. 2B) at the position of the specific person PA, A4 (center on the left side of the screen), the smile degree is 6, and the other person is the unknown person P The face size D3 and the smile degree are 7 at the positions of X and A6 (center on the right side of the screen) The higher the smile level, the higher the smile level. The surrounding sound is low, and the shooting date and time at this time is 11:21:10 on September 15.

  In this embodiment, as shown in FIG. 2 (a), the number, position, size, and characteristics of the face image, the volume of surrounding audio, and the shooting date / time information are determined every 5 seconds and recorded. Part 4 is recorded. Further, the recording unit 4 also determines and records whether or not it can be a representative image candidate (main subject). In the example of FIG. 2A, two people are shown when 0 second elapses, but both face sizes are the same as D2, and both sides of the screen (positions A4 and A6). A person cannot be said to be a main subject, and a candidate is recorded as “none”.

  When 5 seconds have elapsed since the start of shooting, two people are shown, but the face size is small, D2 on one side and D3 on the other, and both sides of the screen (positions A4 and A6). The candidate is not recorded as a main subject and is recorded as “none”. Next, when 10 seconds have elapsed from the start of shooting, three people are shown, and among these, the specific person PA is shown in the middle (position A5) in the size of D2. Since there is a specific person in the middle, this PA is the main subject, and this PA is recorded as a representative image candidate. At 15 seconds and 20 seconds from the start of shooting, three people are shown, and even at these timings, the specific person PA is shown in the middle (position A5) in the size of D2. Since there is a specific person in the middle, this PA is recorded as a representative image candidate.

  Next, the concept of determining a representative image from a plurality of moving images will be described with reference to FIG. The moving image A21, the moving image B22, and the moving image C23 are moving images recorded in series. Each moving image is a collection of continuous still images. A person 26 and a person 28 are shown in the moving image A21, a person 26 is shown in the moving image B22, and a person 26 and a person 27 are shown in the moving image C23. When it is determined whether or not the persons 26 to 28 appearing here are the same as the persons registered in the face DB 4b, the person 26 corresponds to the B child and the person 27 corresponds to the C child. The person 28 does not have a corresponding specific person, and the child A is registered in the face DB 4b, but does not appear in the moving pictures A21 to C23.

  When characters in such a series of videos A21 to C23 are analyzed, the person 26 (B child) appears in any of the videos 21 to 23, and the person 28 is not registered in the face DB 4b. Furthermore, although the person 27 (C child) appears in the screen center portion 23c, it appears only at the end, and since it is a small image, the person 26 (B child) is determined as the representative image. In the present embodiment, a person who appears in the first and last moving images and has a certain size is determined as a representative image.

  Next, an example in which the determined representative image is displayed for ending will be described with reference to FIG. The examples of the moving image D31, the moving image E32, and the moving image F33 are all moving images taken when traveling. Based on the moving images 31 to 33, the person to be the representative image is determined in the same manner as described with reference to FIG. A person 26 is shown in the moving picture D31, and a person 26 is shown in the moving picture E32. A person 26 is also shown in each of the images 33a, 33b, and 33c constituting the moving image F33. Based on these analyses, in this example, the person 26 who has the highest frequency is determined as a representative image.

  In this way, when a plurality of moving images are continuously reproduced, the person who has the highest frequency is treated as a leading character among the characters, and displayed at the ending 34 at the end of the moving image reproduction. That is, as shown in FIG. 4, the ending 34 displays the determined representative image, that is, the person 26, and the name of the main performer. Here, the name of the main performer is retrieved and displayed based on the face image and name registered in the face DB 4b.

  Next, with reference to FIG. 5, partial playback for playing back a part of a moving image will be described. FIG. 5 is an example of a moving image that captures an event such as a banquet. In such an event, there is a scene where participants laugh and swell, and such a scene can be enjoyed even if played back later. However, if you start from the scene of a big laugh when playing partly, you don't know the reason for the big laugh and you feel abrupt. Therefore, in the present embodiment, reproduction is performed from the question scene that is the basis of the scene of laughter.

  That is, in the example shown in FIG. 5, in the video portion 41, the person 26 asks the person 29 a question, the person 29 answers in the next timing video portion 42, and then in the next timing video portion 43. And everyone laughs. Therefore, when a part 43 laughing is detected, partial playback is performed from the part 41 of the moving image that triggered the laughing. Specifically, the speech analysis unit 1d determines that the word issued before the scene of laughter is “answer” and the word issued before this “answer” is determined as “question”. The scene after this “question” is partially reproduced. Of course, “Tsukumi” may be inserted after “Question” and “Answer”, so it may be determined whether the word ends with a specific “Question”. It may be determined whether or not it contains a phrase unique to the question, such as "What?"

  Next, the case where a plurality of moving images are continuously reproduced will be described with reference to FIG. When playing back a moving image, first, a thumbnail of the moving image is displayed on the display unit 8. In the example shown in FIG. 6, the thumbnails of nine moving images are displayed in a vertical 3 and horizontal 3 manner. FIG. 6A shows a state where a thumbnail of the moving image is displayed, FIG. 6B shows a state where the user has selected by touching the moving image at the upper left corner, and FIG. 6C shows the upper middle. A state of selection by touching a moving image is shown. After that, although not shown, the moving image in the upper right corner is selected.

  When a plurality of moving images are selected (three in the example shown in FIG. 6), the user next touches a reproduction mark displayed on the display unit 8. When the user's playback instruction is detected by a touch panel or the like, the camera 10 starts playback of a plurality of moving images. In addition to the touch panel, the instruction may be changed as appropriate, for example, by moving the cursor with the cross button or the like and confirming with the OK button.

  Next, with reference to FIG. 7, partial reproduction for reproducing one moving image portion will be described. When one moving image is selected on the display unit 8 as shown in FIG. 7 (a) and then playback is selected as shown in FIG. 7 (b), as shown in FIG. 7 (c), A display for selecting either “part” or “full-length” is displayed. The full version is a mode in which the selected video is reproduced and displayed from the beginning to the end, and the part is a scene that becomes a laughing scene of the selected video as described with reference to FIG. It is a mode to play from.

  The flow for executing the operation of the present embodiment described with reference to FIGS. 3 to 7 will be described with reference to the flowcharts shown in FIGS. FIG. 8 is a camera control flow. If the camera control flow is entered, it is first determined whether or not the power is on (S101). In this step, it is determined whether or not a power switch as an operation member of the camera 10 is on. If the power switch is off, the camera control flow is terminated. Even when the camera control flow is finished, the state of the power switch is detected, and when the power switch is turned on, the operation starts from step S101.

  If the result of determination in step S101 is that the power supply is on, it is next determined whether or not it is in shooting mode (S102). If the result of this determination is that the camera is in shooting mode, then live view display is performed (S103). Here, the subject image is displayed in live view on the display unit 8 based on the image data acquired by the imaging unit 2. The user can determine the composition or the photo opportunity based on the live view display and perform the release operation.

  Once the live view display is started, it is next determined whether or not to start recording (S104). Here, the operation determination unit 6 determines whether or not the release button has been operated. In the present embodiment, when the release button is first operated, the recording of the moving image is started, and when the release button is operated again, the recording of the moving image is ended.

  If the result of determination in step S104 is that recording has not started, processing returns to step S101. On the other hand, if the result of determination is that recording has started, shooting, sound collection, recording, and timing are performed (S111). In this step, image data is acquired by the imaging unit 2, and moving image data that has been subjected to image processing by the signal processing and control unit 1 is recorded in the recording unit 4. Also, voice data is acquired by the voice recording unit 7 and output to the voice analysis unit 1d. Further, the clock unit 9 measures the time from the start of moving image recording.

  Subsequently, a face (expression) is detected and laughter is detected (S112). In this step, the face detection unit 3 determines the face position, characteristics, size, and smile level described with reference to FIG. Further, using the voice data acquired by the voice recording unit 7, the voice analysis unit 1d detects whether or not a laughing voice is included in the voice, whether or not the voice is loud, and the like.

  Subsequently, information about the face for each hour is recorded, and information about the laughter for each hour is recorded (S113). Here, in step S112, the detected face (expression) and laughing voice information are recorded in the recording unit 4 for each time. This makes it possible to record information other than the lowest candidate information among the information shown in FIG. Next, it is determined whether or not the snapshot is in the center of the screen (S114). Here, the determination as to whether or not it is a snap photograph is, for example, whether or not the photographing magnification is about 1/50, in other words, for example, when the focal length is 50 mm in terms of 35 mm film, 50 × 50 = 2500. It is determined whether the subject is closer than about 5 m or a photograph of a person image above the waist. If it is a snapshot, it is further determined whether or not the face portion detected by the face detector 3 is in the center of the screen.

  If the result of determination in step S114 is a snapshot at the center of the screen, then the candidate is selected as the main subject and its timing is recorded (S115). Here, among the information shown in FIG. 2A, the information on the lowest candidate is recorded in correspondence with the time measurement. If this recording is performed, or if the result of determination in step S114 is that the screen is not center snap, it is next determined whether or not recording has ended (S116). Here, the operation determination unit 6 determines whether or not the release button has been operated again. If the result of this determination is that recording has not ended, processing returns to step S101.

  If the result of determination in step S116 is that recording has been completed, moving image shooting is terminated, and in step S121 and subsequent steps, preprocessing for generating an ending image is performed. First, it is determined whether there is a main subject candidate (S121). Here, it is determined whether or not the main subject candidate is recorded in step S115. That is, it is determined whether or not a candidate specific person is recorded at the bottom of the information in FIG.

  If the result of determination in step S121 is that a main subject candidate has been recorded, next, the corresponding text is determined in the database (S122), and it is determined whether there is corresponding text (S123). Here, it is determined whether or not the name of the specific person who is a candidate in the face DB 4b is registered as text information.

  If the result of determination in step S123 is that there is a corresponding text, the name of the main subject is recorded as additional information together with moving image data (S124). If the main subject name is recorded or if the result of determination in step S123 is that there is no corresponding text, it is next determined whether or not all candidates have been determined (S125). In this step, it is determined whether or not the processing in steps S121 to S124 has been performed for all of the candidates shown in the lowermost stage of FIG. As a result of this determination, if all the candidates have not been determined, the process returns to step S121. On the other hand, if all the candidates have been determined, the process returns to step S101.

  If the result of determination in step S <b> 102 is not shooting mode, it is next determined whether or not playback mode is in effect (S <b> 131). In this step, it is determined whether or not the playback mode has been selected by the operation determination unit 6. If the result of this determination is not playback mode, processing returns to step S101.

  On the other hand, if the result of determination in step S131 is playback mode, image playback is performed (S132). Here, the selected plurality or one moving image is reproduced. When a plurality of moving images are selected, the moving images are reproduced in order. In the case of reproducing one moving image, it is possible to select the partial reproduction described with reference to FIG. Details of this image reproduction will be described later with reference to FIGS. When the image reproduction ends, the process returns to step S101.

  Next, an image reproduction flow will be described with reference to FIGS. 9 and 10. If the flow of image reproduction is entered, it is first determined whether or not it is finished (S201). In this step, it is determined whether or not the user has instructed to end the reproduction mode by an operation such as pressing the reproduction button again. If the result of this determination is end, image playback ends and the flow returns to the original flow.

  If the result of determination in step S201 is not end, thumbnail display is performed (S202). In this step, as shown in FIG. 6A, thumbnails are displayed on the display unit 8 of the images of the main subject that is the first candidate in each selected moving image. For example, when the image 1 shown in FIG. 2 is selected, since PA is recorded as a candidate when 10 seconds have elapsed since the start of moving image recording, the image when 10 seconds has elapsed is displayed as a thumbnail. To do.

  Once thumbnail display has been performed, it is next determined whether or not it has been selected (S203). Here, it is determined whether or not the user has touched the video that the user desires to play. If the result of this determination is that it has been selected, the selected thumbnail frame is displayed (S211). In this step, as shown in FIGS. 6B, 6C, and 7A, the frame of the selected moving image is displayed with a bold line. In order to indicate that the frame has been selected, it is of course possible to change the display method to various display methods such as changing the color of the frame line or displaying it in flash, in addition to making the frame line thicker.

  Once the selected thumbnail frame is displayed, the playback image order is determined (S212). In the present embodiment, the moving images are played back in the order in which the user touches, so here, the playback order of the moving images is determined in the touched order. Subsequently, the total reproduction image time is determined (S213). Here, the sum of the playback times of the selected moving images is obtained. When the total time of the reproduced image is determined, the process returns to step S203.

  If the result of determination in step S203 is that no selection has been made, it is determined whether or not selection has been completed (S221). That is, every time there is a selection operation in step S203, the process branches to step S211 and step S221 is branched to S201 until the selection operation ends. When the user performs an operation for reproducing a moving image or the like, the process branches from step S221 to S222, and it is determined whether or not a plurality of moving images has been selected (S222). In steps S203 to S213, it is determined whether a plurality of moving images have been selected. As a result of this determination, if a plurality of moving images have been selected, the person appearing in the selected first and last moving images is determined (S223). Here, candidates appearing in the selected first moving image and candidates appearing in the last moving image are extracted. For example, when the image 1 shown in FIG. 2 is selected, three people, PA, PB, and PX, appear, but since PA is recorded as a candidate, Extract PA.

  Next, it is determined whether or not the person determined in step S223 is the main subject (S224). In this step, it is determined whether the person determined as the candidate in the first moving image matches the person determined as the candidate in the last moving image. As a result of this determination, if they match, this candidate is determined to be the main character (S227). In this embodiment, the person appearing in both the selected first and last moving images is used as a star, and display is performed in the ending.

  If the result of determination in step S224 is that the main subject does not match, or if it is determined in step S222 that a plurality of moving images has not been selected, then the person who has appeared the longest is determined ( S225). If there is no main subject common to the beginning and end of a plurality of moving images, the person with the longest appearance time is searched. Even when only one is selected, the person with the longest appearance time is searched. For example, in the example of FIG.

  Once the longest person has been determined, it is next determined whether or not the longest person is the main subject (S226). In the present embodiment, since only the person registered in the face DB 4b is treated as the main subject, it is determined here whether or not the person determined in step S225 is registered in the face DB 4b. As a result of this determination, if it is the main subject, it is determined as the main character in step S227 as described above.

  If it is determined in step S227 that it is the main character, or if the result of determination in step S226 is that the person who has appeared the longest is not the main subject, it is next determined whether or not there is a character other than the main character (S231). ). In this step, it is determined whether or not there is a person appearing in the selected one or more moving images. If the result of this determination is that there is a character, character determination is performed (S232). For example, in the image 1 shown in FIG. 2A, since PA is determined to be the main character, PB and PX other than this character are determined to be characters. In addition, as a character, people other than those registered in the face DB 4b may be used. In this case, if the face detection unit 3 can discriminate between males and females and estimate the age, the information may be acquired together.

  If the character determination is made in step S232, or if the result of the determination in step S231 is that there is no character other than the main character, it is next determined whether or not partial reproduction is performed (S301). As described with reference to FIG. 7C described above, when “part” on the display unit 8 is touched, partial reproduction is executed. Therefore, in this step, it is determined whether or not “part” is touched.

  If the result of determination in step S <b> 301 is that “part” has not been touched, playback is then performed in order (S <b> 302). In this step, the moving image is reproduced according to the order determined in step S212. In reproduction, full-length reproduction is performed, but in addition to this, when each moving picture is long, each digest version may be generated and the digest version may be reproduced. In this case, the image of the leading character is displayed at a good tempo in about 3 to 5 seconds divided into predetermined times, and the characteristic scene such as the first image and the end image of the photographed image is determined, Only the image may be displayed by switching to about 3 seconds.

  When reproduction is started in accordance with the order, it is next determined whether or not reproduction has been completed to the end (S303). Here, it is determined whether or not playback has been completed up to the end of the playback order determined in step S212. If the result of this determination is that playback has not been completed, the process returns to step S302 to continue playback of the moving image.

  If the result of determination in step S303 is that reproduction has been completed, ending is displayed in step S304 and subsequent steps. First, a lead determination is performed (S304). Here, in step S227, it is determined whether or not the main character is determined. If the result of this determination is that the starring determination has been made, an image in which the starring has the highest smile level is then displayed (S305). As shown in FIG. 2 (a), since the detected characters are also determined and recorded for the smile level, search for the image with the highest smile level for the characters determined to be the main characters. To do. In the example shown in FIG. 2A, the starring is PA, and the image with the highest smile level of PA is 15 seconds after the start of photographing. As shown in the moving image 34 of FIG.

  Subsequently, the name of the lead is displayed (S306). In step S124, since the name of the main subject is recorded, in this step, the name of the main character is determined from the image showing the main character and displayed as shown in the moving image 34 of FIG. Therefore, if there is a star, first, the image of the star is displayed in step S305, and then the name of the star is displayed in step S306. If it is determined in step S232 that there are characters, the names of these characters may be displayed. If you don't know the name of the character, you can display the characters “◎”, “one man”, “uncle”, “baby”, etc. in a determinable range, making the screen lively and enjoyable.

  If the result of determination in step S304 is that there is no star, the date and time common to all images is displayed (S307). As shown in FIG. 2A, since the date / time information is also recorded when the moving image is shot, the date / time common to a plurality of moving images is displayed based on the date / time information. If the shooting date is common, that day is displayed. If the shooting date is not common, the shooting month is displayed. If the shooting date is not common, the shooting year is displayed.

  If the date and time common to all images is displayed in step S307 or the name of the main character is displayed in step S306, it is next determined whether or not the reproduction time is 1 minute or longer (S311). Here, the determination is made based on the total reproduction image time obtained in step S213. If the result of this determination is 1 minute or more, the ending display is finished in 10 seconds (S312). On the other hand, if the result of the determination is within 1 minute, the ending display is finished in 3 seconds ( S313). This is because if the playback time is short but the ending is long, the playback becomes boring. In the present embodiment, the ending display time is changed according to the playback time.

  When the ending display is finished in step S312 or S313, the process returns to step 101 (see FIG. 8).

  If the result of determination in step S <b> 301 is partial playback, then file determination is performed according to the order (S <b> 321). Here, the file determination is performed so that the order of touching the thumbnails shown in FIG. 6 or the order specified together with the thumbnail specification is reflected. Subsequently, it is determined whether or not there is a laughter in the designated file (S322). At the time of shooting a moving image, audio data is input by the audio recording unit 7, and the audio analysis unit 1d determines whether or not a laughing voice is included and the magnitude of the voice, as shown in FIG. The determination result is recorded together with the moving image. Based on this record, it is determined in this step whether or not laughter is included.

  If the result of determination in step S322 is that there is a laughing voice, then from the previous dialogue to the laughing voice is reproduced (S323). In this step, as described with reference to FIG. 5, the moving image is reproduced from the dialogue that has caused the laugh. If the time point is not clear, a predetermined time before laughing is reproduced.

  Once playback has been performed in step S323, it is next determined whether playback has been performed up to the last file (S324). If the result of this determination is that playback has not ended up to the last file, processing returns to step S321. On the other hand, as a result of the determination, when the reproduction to the last file is completed, the process jumps to the above-described step S304, displays the ending, and ends the reproduction.

  If the result of determination in step S322 is that there is no laughter in the file, it is next determined whether or not a leading character has appeared (S331). In this step, the determination is made based on whether or not the starring determination has been made in step S227. If the result of this determination is that a leading character has appeared, the portion containing the leading character and having a high sound is reproduced (S332). In this step, the main character appears and the loud part is played for a predetermined time. If the time is too long, it becomes a boring image, so it is not necessary to reproduce all the loud parts. When this reproduction is performed, the process returns to step S324.

  If the result of determination in step S331 is that the leading character has not appeared, it is determined whether or not a person appears (S333). As described above, if a person appears at the time of moving image shooting, it is recorded together with the moving image, and therefore determination is made based on the recording result. If the result of this determination is that a person has appeared, a period in which there are many smiling persons is reproduced (S334). Since the smile level is determined and recorded by the face detection unit 3 at the time of moving image shooting, playback is performed based on the smile level. On the other hand, if the result of determination in step S333 is that no person has appeared, playback is performed at double speed (S335). This is because a person does not appear and the image is a landscape or the like. In this case, there is no sense of incongruity even if the image is reproduced at double speed. For example, when shooting while panning a majestic mountain range, it is not necessary to play over the panning time, and it is possible to experience the majesticness without spending time even at double speed.

  When playback is performed in step S334 or step S335, the process returns to step S324 to determine whether or not playback has been performed up to the last file. If playback has been performed up to the last file, an ending is displayed in step S304 and subsequent steps. End playback.

  As described above, in one embodiment of the present invention, when one moving image is generated and reproduced from a plurality of moving images, characters are determined and recorded at the time of shooting (S115), and common to a plurality of images at the time of reproduction. The characters who appear are starring (S227) and are sequentially reproduced and displayed. In addition, a representative image in which the leading role appears is determined, and this representative image is displayed at the time of ending (S305). For this reason, it is possible to easily and immediately enjoy the captured moving image.

  In one embodiment of the present invention, when selecting a representative image, it is first determined whether or not it is a candidate for a main subject that may become a representative image at the time of shooting (S114 and S115). Therefore, before the start of reproduction, a representative image in which the leading role appears is determined (S227), and this is displayed in the ending. As described above, since processing is performed at the time of shooting and playback, playback can be performed in a short time. If a plurality of videos are gathered to complete a single video, there are videos that shoot other than the main character. In this case, it is not known whether the main character or the supporting character is being filmed. In this embodiment, by selecting and starting a plurality of images at the time of reproduction, the main character is determined from the characters appearing in common. In other words, the starring is accurately selected in two stages of shooting and playback.

  Furthermore, in one embodiment of the present invention, main subject candidates are recorded with reference to the names of persons registered in the face DB 3b (S121 to S124). Further, when displaying the ending, the image of the main character determined from the main subjects is displayed (S305), and the name of the main character is also displayed (S306). Since the name of the starring star is displayed, you can enjoy it even more.

  Furthermore, in one embodiment of the present invention, when a plurality of moving images are reproduced, the person who appears in the first moving image and the last moving image selected in determining the starring (S223) and the main subject. In the case (S224), the main role is taken (S227). Since it appears at the beginning and end, the consistency of the story as the leading role is created and you can enjoy watching it. In addition, in determining the starring, other than this, it may be selected by a method of a main subject or the like that was shown in the largest number of moving images.

  Furthermore, in the embodiment of the present invention, when performing partial reproduction, a scene with a laughing voice is automatically selected and reproduced (S322, S323). Scenes with laughter are lively and can be enjoyed. Furthermore, when playing a scene with laughter, in this embodiment, playback is started from the scene with the question that triggered the laughter, so it is possible to eliminate the abrupt feeling. In addition, you can understand what kind of conversation was exciting and enjoy together.

  Furthermore, in one embodiment of the present invention, when there is no laughter during partial playback, a period when the voice is loud (S332) or a period when there are many smiling people (S334) is played. . For this reason, even when there is no laughter, it is possible to determine an important excitement scene and enjoy it as an effective partial reproduction.

  Furthermore, in one embodiment of the present invention, the leading image is displayed when displaying the ending (S305). For this reason, it is possible to end the moving image without any sudden feeling. At this time, the name of the leading character is displayed based on the registration in the face DB 4b in advance, and can be enjoyed when the moving image is reproduced in the group, and can be clearly shown as the ending image. In addition, the ending image becomes lively with characters, the feeling that the image ends, and the viewer's mood is also raised. Furthermore, by displaying the name, even if you do not know the face of the starring and know only the name, you can understand and enjoy that person was such a person. Like the ending of the movie, it will be a sneak peek at the performers, and the motivation of the performers will increase.

  Furthermore, in one embodiment of the present invention, the ending display time is changed according to the moving image playback time (S311 to S313). For this reason, it can be set as the ending time found in the reproduction time of a moving image.

  Next, a modified example of the thumbnail display in step S202 will be described with reference to FIGS. In one embodiment, the first image among images candidated as the main subject in each moving image is displayed side by side. On the other hand, in this modification, thumbnail display is performed for each person who appears in common in a plurality of moving images.

  The thumbnail display in this modification will be described with reference to FIG. In this display example, it is assumed that three moving images, a moving image G41, a moving image H42, and a moving image J43, are recorded in the recording unit 4. A person 27 and a person 26 appear in the moving picture G41, a person 26 appears in the moving picture H42, and a person 26 and a person 27 appear in the moving picture J43.

  When such a moving image is recorded, the display unit 8 alternately displays a thumbnail display 45 that collects scenes showing the person 26 and a thumbnail display 46 that collects scenes showing the person 27. . In this example, a total of four images are displayed on the display unit 8 in two vertical and horizontal images. Since the person 27 has only three images, the thumbnail display 46 does not display one image.

  Next, an operation for displaying commonly appearing persons as thumbnails will be described with reference to the flowchart shown in FIG. In the thumbnail display flow, first, the top images of the respective moving images are reduced and arranged (S401). Subsequently, a search for common characters in the display thumbnail video is performed (S402). Here, the search is performed based on the facial features of the characters determined by the face detection unit 3 and recorded in the recording unit 4. In the example shown in FIG. 2A, three persons PA, PB, and PX are searched.

  Once the common character search is performed, it is next determined whether or not there is a person who appears in common (S403). If there is a common character as a result of this determination, it is next determined whether or not a predetermined time, 3 seconds in this modified example, has elapsed (S404). As a result of this determination, if the predetermined time has not elapsed, it is determined whether or not to end the thumbnail display (S421). When the user operates the operation member to end the display mode during display, the process returns to the original flow. If the result of determination is not end, processing returns to step S404.

  If the predetermined time has passed as a result of the determination in step S404, then the most commonly appearing person is converted into a representative image and displayed as a thumbnail (S405). Here, like the thumbnail display 45 shown in FIG. 11, an image showing the most frequently appearing person 26 is displayed as a thumbnail. Further, when displaying as a representative image, the first image selected as a candidate is used. Subsequently, the frames that do not appear are darkened (S406). Since there are cases where the image does not appear in all the images, the thumbnail corresponding to the image that does not appear is displayed as black or a predetermined color so as not to appear there.

  Next, it is determined whether or not a predetermined time, in this modification example, 3 seconds has elapsed (S411). As a result of this determination, if the predetermined time has not elapsed, it is determined whether or not the process is completed as in step S421 (S422). If the result of this determination is that it has not ended, processing returns to step S411, whereas if it has ended, processing returns to the original flow.

  If the result of determination in step S411 is that a predetermined time has elapsed, the next common person is represented as a representative image and displayed as a thumbnail (S412). Here, as in the thumbnail display 46 shown in FIG. 11, the next image showing the most frequently appearing person 27 is displayed as a thumbnail. Subsequently, the frames that do not appear are darkened (S413). Similar to step S406, if the number of characters is not enough for the number of thumbnail display frames, the missing frames are darkened. In the thumbnail display 46 shown in FIG. 11, since there are only three frames, the upper right frame is darkened.

  Subsequently, it is determined whether or not a predetermined time, in this modification example, 3 seconds has elapsed (S414). As a result of the determination, if the predetermined time has not elapsed, it is determined whether or not the process is finished as in step S421 (S423). If the result of this determination is that it has not ended, processing returns to step S414, while if it has ended, processing returns to the original flow.

  If the result of determination in step S414 is that a predetermined time has elapsed, processing returns to step S401.

  As described above, the thumbnail display in this modification is configured to display thumbnails centering on a person who appears in common. For this reason, it becomes easy to select from a large number of moving images when performing playback. In this modification, when displaying thumbnails, the characters are displayed as representative images. Since the images selected as candidates are displayed as thumbnails as representative images, an image that is in the center and appears somewhat large is selected, so that it is easy to see when selecting the reproduction display.

  In addition, although the thumbnail display 45 and 46 of FIG. 11 given as an example has two persons appearing in common, it is not limited to this, and of course, three or more persons may be used. In this case, in the flowchart shown in FIG. 12, steps S404 to S406 may be added corresponding to the third and subsequent persons.

  As described above, in one embodiment or modification of the present invention, when a person appearing in a moving image is specified and a thumbnail of a moving image is displayed, an image including the specified person according to the result of specifying the person is displayed. Are displayed as representative images. For this reason, a moving image can be enjoyed instantly and easily. That is, when displaying thumbnails, simply displaying the top image may not show the person who is the subject of the video, but in this embodiment, the identified person is displayed as a representative image. From this, you can easily select a video and enjoy the video instantly. In one embodiment or modification of the present invention, person determination is performed at the time of shooting, and in the case of reproducing a plurality of moving images, a person who appears in common is determined. For this reason, it becomes possible to instantly generate one moving image from a plurality of moving images and enjoy it.

  In the embodiment and the modification of the present invention, the moving image is reproduced after selecting the thumbnail at the time of reproduction. However, the present invention is not limited to this, and the selection based on the thumbnail may be omitted. For example, digest reproduction may be performed consistently with the same character at the time when a plurality of moving images showing the same character are gathered. In this case, a digest playback button is provided on the playback mode screen, and when this is operated, unified movie editing is possible with the characters included in the latest photographed image. If there is a video shot just before the character appears, only the scenes in which the character appears can be gathered together, and several new videos can be generated and played back together. In addition, it is possible to enjoy such that the A child to C child of the face DB 4b shown in FIG. 3 are displayed and the automatically edited moving image is switched depending on which face is selected. Furthermore, even if you do not select a face, you can search for other videos using the characters in the last image in order to create an image with the main character of the person appearing in the last video shot, Several new videos may be generated and played back together.

  Although the image capturing device also serves as a playback device, a playback device may be provided separately, and a moving image captured by the image capturing device may be played back by the playback device. The playback device may be a playback-only machine, but may of course be played back by a personal computer. Further, the playback device may also serve as the photographing device. In this case, when the moving image is reproduced, since the representative image or the like is automatically selected and displayed on the reproducing apparatus side, an unexpected image can be displayed and enjoyed.

  In the embodiment and modification of the present invention, the representative image is selected from the main subject for the sake of explanation. However, the main subject itself may be displayed as the representative image. Furthermore, in one embodiment or modification of the present invention, the names of the main performers and the main performers are displayed in the ending, but they may be displayed other than the ending, for example, at the beginning of the moving image. When displayed at the beginning, you can increase your willingness to watch the video that will start. It can also evoke the feeling of who will see it if it appears. In addition, there is a feeling that something will begin, and to a certain extent, it makes you feel mixed with predictions and expectations.

  Furthermore, in one embodiment or modification of the present invention, partial playback is performed only when one video is selected. However, the present invention is not limited to this, and even when multiple videos are selected, Of course, partial reproduction may be performed.

  Furthermore, in the embodiment and modification of the present invention, the digital camera is used as the photographing device. However, the camera may be a digital single-lens reflex camera or a compact digital camera, such as a video camera or movie camera. It may be a camera for moving images such as a mobile phone, a personal digital assistant (PDA), a game machine, or the like.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 ... Image processing and control part, 1b ... Representative image determination part, 1c ... Composite display part, 1d ... Speech analysis part, 2 ... Imaging part, 3 ... Face detection part, 4 ... Recording unit, 4b ... Face DB, 6 ... Operation determination unit, 7 ... Audio recording unit, 8 ... Display unit, 9 ... Clock unit, 10 ... Camera, DESCRIPTION OF SYMBOLS 12 ... Communication part, 21 ... Moving image A, 22 ... Moving image B, 23 ... Moving image C, 23c ... Center part of a screen, 25 ... Person (A child), 26 ... person (child B), 27 ... person (child C), 28 ... person, 29 ... person, 31 ... movie D, 32 ... movie E, 33 ... movie F, 34 ... Ending, 41 ... Movie G, 42 ... Movie H, 43 ... Movie J, 45 ... Thumbnail display, 46 ... Thumbnail display

Claims (17)

In a photographing device having a photographing mode and a reproduction mode,
A recording unit for recording a shot video during shooting in the shooting mode;
A first determination unit that determines a person appearing in the captured moving image during shooting in the shooting mode and records a determination result in the recording unit;
A second determination unit for determining a person who appears in common in a plurality of shooting operations when collecting and playing back a plurality of the shot videos during playback in the playback mode;
A playback unit that uses the determination result of the second determination unit to take out a part of the moving image and play it back sequentially;
A photographing apparatus comprising: A shooting section for shooting images;
A moving image recording unit that records images continuously acquired from the imaging unit as a moving image;
A person specifying unit for specifying a person appearing in the moving image recorded in the moving image recording unit during the shooting operation from the face image obtained by the imaging unit;
A display control unit for displaying an image including the specified person as a representative image according to the person specifying result when displaying the recorded moving image as a thumbnail;
A photographing apparatus comprising: A character determination unit for determining a person appearing in the video;
An image display unit for displaying a plurality of images side by side;
A character determination unit for determining a person who appears in common in the plurality of images displayed on the image display unit;
A representative image determining unit that determines a representative image of the moving image according to the determination result of the character determining unit;
A display device comprising: An imaging unit that continuously captures a subject and obtains a moving image;
Among the moving images obtained by the imaging unit, a still image selecting unit that selects a still image representing the main character of the moving image;
A display control unit that displays the still image and the additional information representing the main character in association with each other during reproduction of the moving image;
A photographing apparatus comprising:   5. The additional information representing the character is personal name information obtained as a result of collation between an image obtained by the imaging unit and a database recorded in the recording unit. Shooting device.   5. The photographing apparatus according to claim 4, wherein the still image associated with the additional information is displayed as a thumbnail representing reproduction or a moving image at the beginning or end of the moving image. A voice pickup unit that picks up surrounding voices,
A voice determination unit that determines whether or not there is a laughing voice based on the collected voice;
Have
The main character is a character who can pick up a person who has spoken at a timing before the time when the laughing voice is detected by the voice determination unit.
The imaging apparatus according to claim 4, wherein: A first main subject determination unit that determines a main subject included in each of the plurality of videos;
A second main subject determination unit for determining a second main subject from the first main subject group included in the plurality of videos;
A digest reproduction unit that preferentially reproduces a portion where the second main subject appears;
A playback apparatus comprising: A moving image storage unit for storing moving image data;
A determination unit for determining whether or not a main subject exists for the still images constituting the moving image;
A display unit for determining a representative image based on the main subject determined by the determination unit and displaying the representative image;
A display device comprising:   The display device according to claim 9, wherein the display unit displays a main subject as a thumbnail for each of a plurality of moving images.   The display device according to claim 9, wherein the display unit selects a representative image from a plurality of main subjects and displays the representative image.   The display device according to claim 9, wherein the main subject is a person who matches a person image registered in advance. A moving image storage unit for storing moving image data;
A determination unit for determining whether or not there is a person registered in advance for the still images constituting the moving image;
A display unit for determining a representative image based on the person determined by the determination unit and displaying the representative image;
A display device comprising:   The display device according to claim 13, wherein the determination unit determines an image in which a person appearing in common among a plurality of moving images is shown as a representative image. An imaging unit for outputting video image data;
A face detection unit that determines whether a face is included in the image based on the image data;
A face database storing identification data for identifying faces;
Based on the determination result by the face detection unit and the identification data stored in the face database, it is determined whether the face in the image data matches the identification data. A representative image determination unit for determining
A display unit for displaying the representative image determined by the representative image determination unit;
A photographing apparatus comprising: Take a picture,
Record continuously acquired images as movies,
During the shooting operation, identify the person appearing in the recorded video,
When displaying the recorded moving image as a thumbnail, according to the person specifying result, an image including the specified person is displayed as a representative image.
An imaging method characterized by the above. Determine who appears in the video,
Display multiple images side by side,
Determine who appears in common in the multiple images displayed above,
According to the determination result of the characters, a representative image of the video is determined.
A display method characterized by that.