JP2006237791A - Information processing apparatus and program - Google Patents

Abstract

A conventional information processing apparatus has a problem that a predetermined still image cannot be appropriately extracted from a moving image and output.
A content storage unit that stores at least one content including video information having video and audio information having audio, and an utterance that detects an utterance location that is an utterance location based on the audio. A location detection unit 102; a still image extraction unit 104 that extracts one or more still images constituting the utterance location detected by the utterance location detection unit 102; and one or more extracted by the still image extraction unit 104 A predetermined still image can be appropriately extracted from the moving image and output by the information processing apparatus including the output unit 108 that outputs the still image to one or more non-overlapping windows.
[Selection] Figure 1

Description

  The present invention relates to an information processing apparatus that extracts and outputs a predetermined still image from a moving image.

In a conventional moving image summarizing method for summarizing moving images and the like, there is a moving image summarizing method capable of automatically extracting semantically important events in a moving image (for example, see Patent Document 1). The video summarization method includes a feature extractor 40 that extracts video features, a probability model 42 that uses a model such as a hidden Markov model to integrate features and determine boundaries, and commercial and non-commercial slow motion playback segments. A commercial / non-commercial filter 44 and a summary generator 46 that generates a summary based on the detected slow motion playback segment. The feature extractor 40 in this moving image summarization method extracts features from the color histogram in block 50 and extracts three features from the pixel-based differences in block 52. The features extracted at block 52 characterize each component of the playback segment's slow motion, still field, and / or normal speed playback. The feature extracted in block 50 in the moving image summarizing method characterizes the editing effect component.
Further, Non-Patent Document 1 discloses a method for acquiring content including moving images and audio described in the present embodiment.
JP 2002-232840 A (first page, FIG. 1 etc.) Yasuyuki Kaku et al., “Recording and Sharing Experiences in Ubiquitous Environment”, System / Control / Information (Journal of System Control Information Society), November 2004, Vol. 48, no. 1, pp. 458-463

  However, the information processing apparatus that implements the conventional moving image summarization method has a problem that a still image cannot be appropriately cut out from the moving image and output effectively.

  The information processing apparatus according to the first aspect of the invention is a content storage unit that stores at least one content including video information having video and audio information having audio, and a portion that speaks based on the audio An utterance location detection unit that detects an utterance location, a still image extraction unit that extracts one or more still images constituting the utterance location detected by the utterance location detection unit, and a 1 extracted by the still image extraction unit The information processing apparatus includes an output unit that outputs the above still image to one or more windows that do not substantially overlap.

  With this configuration, for example, the actions of those who participated in the event are summarized by automatically extracting still images that make up the scene to be noticed, such as conversation or speaking, and placing them in many windows. An electronic album is available.

  The information processing apparatus according to the second aspect of the invention includes a content storage unit that stores at least one content including video information having video and audio information having audio, and two or more objects reflected in the video. A facing part detection unit that detects that they are facing each other; a still image extraction unit that extracts one or more still images from the video that configures the facing part detected by the facing part detection unit; and the still image extraction The information processing apparatus includes an output unit that outputs one or more still images extracted by the unit to one or more non-overlapping windows.

  With this configuration, for example, you participated in an event by automatically extracting still images that make up the scene you should pay attention to, such as the place where the main character is viewing, facing the object, and placing it in many windows. An electronic album summarizing the behavior of the person.

In the information processing apparatus according to the third aspect of the present invention, in contrast to the first aspect, the still image extraction unit repeatedly extracts one or more still images constituting an utterance location or / and an opposite location from the video. The output unit is an information processing apparatus that outputs the one or more windows while switching the still image extracted by the still image extraction unit.
With this configuration, when a large number of scenes are cut out, a series of actions can be effectively overviewed even on a narrow display.

Further, in the information processing apparatus according to the fourth aspect of the present invention, as compared with the first to third aspects, the output unit includes a still image output unit that outputs the still image extracted by the still image extraction unit, and the still image It is an information processing apparatus including a blowout output unit that outputs a blowout over a picture.
With this configuration, it is possible to view a plurality of cut out still images like a comic.

Further, in the information processing apparatus according to the fifth aspect of the invention, in contrast to the fourth aspect of the invention, the blowout output means analyzes the sound corresponding to the video having the still image, and two or more types according to the analysis result It is an information processing apparatus which distinguishes and outputs the balloon of the shape.
With such a configuration, it is possible to easily grasp what kind of scene it was just by looking at the balloon.

Further, in the information processing apparatus according to the sixth invention, in contrast to the third and fourth inventions, the blowing output means analyzes the voice corresponding to the video having the still image and acquires the length of the utterance. The information processing apparatus outputs a character string having a length corresponding to the length in a balloon.
With such a configuration, it is possible to intuitively grasp the length of conversation or utterance simply by looking at the character string in the balloon.

In addition, the information processing apparatus according to the seventh aspect of the invention further includes a position detection unit that analyzes the video and detects a position of at least the periphery of the object, as compared with the fourth to sixth aspects. The means is an information processing apparatus that outputs a balloon around the position detected by the position detection unit.
With this configuration, it is possible to display a speech balloon as if a person in a still image is talking.

Further, the information processing apparatus according to the eighth aspect of the invention is not a user who analyzes the video and the sound and outputs the sound corresponding to the output still picture in the still picture, as compared with the fourth to seventh aspects of the invention. The non-display user utterance detection unit further detects that the voice is a user's voice, and the blowing output unit detects that the non-display user utterance detection unit is a voice of a user who is not a user who appears in a still image. In this case, the information processing apparatus outputs a speech balloon in such a manner that the speech balloon appears from the outside or corner of the window.
With this configuration, it is possible to intuitively understand that a person who is not shown in the still image is talking.

The information processing apparatus according to the ninth aspect of the invention further includes a position detection unit that analyzes the video and detects at least a peripheral position of the object with respect to the first to sixth and eighth aspects of the invention, The output unit is an information processing apparatus that outputs the still image by changing an output mode between a region around the position detected by the position detection unit and another region.
With this configuration, it is possible to display an object in which attention is desired in a manner that allows attention.
The information processing apparatus according to the tenth aspect of the present invention is the information processing apparatus according to the tenth aspect of the present invention, wherein the output section is a color area around the position detected by the position detection section, and the other area is monochrome. An information processing apparatus that outputs the still image.
With this configuration, appropriate highlight display can be performed.

The information processing apparatus according to the eleventh aspect of the present invention is the information processing apparatus according to the eleventh aspect, wherein the video information has an object identifier for identifying a video and an object appearing in the video, and the audio information is , Having an object identifier for identifying a voice and a voice speaker, the utterance point detection unit is a utterance point where the volume of the voice is equal to or larger than a predetermined size, and is paired with one object identifier. And an utterance part that is a part of a conversation having a voice paired with another object identifier that is substantially continuous.
With this configuration, it is possible to cut out a still image corresponding to the part where the conversation is performed.

The information processing apparatus according to the twelfth aspect of the present invention is the information processing apparatus according to the eleventh aspect, further comprising: a scene type determining unit that analyzes the content that constitutes the utterance location that is the location of the dialogue and determines the type of scene. Further, the output unit is an information processing apparatus in which the shape of the area around the position detected by the position detection unit is different based on the type of the scene.
With this configuration, the type of scene can be intuitively grasped by looking at the highlight shape.

  According to the information processing apparatus of the present invention, an appropriate still image can be extracted from a moving image and overviewed.

Hereinafter, embodiments of an information processing apparatus and the like will be described with reference to the drawings. In addition, since the component which attached | subjected the same code | symbol in embodiment performs the same operation | movement, description may be abbreviate | omitted again.
FIG. 1 is a block diagram of an information processing apparatus according to this embodiment.

The information processing apparatus includes a content storage unit 101, an utterance location detection unit 102, an opposite location detection unit 103, a still image extraction unit 104, a position detection unit 105, a scene type determination unit 106, a non-display user utterance detection unit 107, and an output unit 108. It comprises.
The output unit 108 includes a still image output unit 1081 and a blowout output unit 1082.

  The content storage unit 101 stores one or more contents including video information including video and audio information including audio. The content includes, for example, video information, audio information, and an object identifier that identifies an object captured by the video. The object is, for example, a person, an exhibit, an exhibition panel, or the like. The video information includes, for example, an object identifier that identifies the video and the person who has taken the video. Also, the video information includes, for example, an object identifier that identifies a video and an exhibit in which a camera that captures the video is installed. The voice information has an object identifier for identifying voice and a voice speaker. Such content is acquired by, for example, the following information acquisition device. An example of the information acquisition apparatus is shown in FIG. In FIG. 2, a “CCD camera”, an “infrared ID tag”, and an “infrared sensor” are provided above the ears of the user (object). The “CCD camera” acquires an image. The “infrared ID tag” transmits an infrared signal on which a signal indicating the object identifier of the user is superimposed. The “infrared sensor” receives an infrared signal from the outside. That is, the “infrared sensor” is an “IR tracker”. The “IR tracker” receives a signal transmitted from the “infrared ID tag” of the opposing object, and obtains an object identifier. It has a “microphone” at the mouth and a “throat microphone” at the throat. In addition, an HMD (head mounted display) is provided in front of the eyes. The signal of the “CCD camera” is acquired by the PC carrying the back of the user and transmitted from the PC to the information processing apparatus. Further, the HMD is used to output information on the position of the user, the displayed exhibits and the people (objects) facing the user. Needless to say, the information acquisition apparatus in FIG. 2 is an example of a content method. The content storage unit 101 is preferably a non-volatile recording medium, but can also be realized by a volatile recording medium. It may be a non-volatile recording medium or a volatile recording medium.

  The utterance part detection unit 102 detects an utterance part that is a part where the utterance is made based on the voice. The utterance location is usually a location where the volume of the voice is greater than or equal to a predetermined level, and the voice of a predetermined level or higher is continuously detected for a predetermined time (for example, 5 seconds) or longer. It is a place. The utterance location detection unit 102 is an utterance location where the size of the voice is a predetermined size or more, and a voice that is paired with one object identifier and a voice that is paired with another object identifier that is substantially continuous. An utterance location (also referred to as a dialogue location in this case) may be detected. The utterance point detection unit 102 can be usually realized by an MPU, a memory, or the like. The processing procedure of the utterance location detection unit 102 is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit). In addition, a location is the information for the part for a certain time among the information which has time axes, such as a moving image and an audio | voice, or the information which specifies the information of the said part.

  The facing location detection unit 103 detects a location where two or more objects are facing each other. Here, “facing” does not need to face from the front, but may be in a positional relationship that allows dialogue. In addition, being in a position where people and exhibits face each other is also called opposite. For example, the opposite location detection unit 103 has the object identifier acquired by the “IR tracker” held by the user A as the object identifier of the user B, and the object identifier acquired by the “IR tracker” held by the user B. Is the object identifier of user A, it is determined that user A and user B are facing each other (facing each other). That is, the object identifier transmitted from the information acquisition apparatus having the “IR tracker” of the user A is the object identifier of the user B, and the object identifier transmitted from the information acquisition apparatus having the “IR tracker” of the user B Is the object identifier of the user A, the facing location detection unit 103 determines that the user A and the user B are facing each other. Note that it may be detected that two or more objects are facing each other by analyzing the image of the video. In addition, the algorithm for recognizing “opposing” does not matter. The facing location detection unit 103 can be usually realized by an MPU, a memory, or the like. The processing procedure of the facing part detection unit 103 is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The still image extraction unit 104 extracts one or more still images constituting the utterance location detected by the utterance location detection unit 102 from the video. Further, the still image extraction unit 104 extracts one or more still images from the video that configures the opposite location detected by the opposite location detection unit 103. Still picture extraction part 104 may repeatedly extract one or more still pictures which constitute a speech part or / and an opposite part from a picture. “Extracting a still image from a video” may acquire the still image itself, or may acquire pointer information in the video for acquiring the still image. The still image extraction unit 104 can be usually realized by an MPU, a memory, or the like. The processing procedure of the still image extraction unit 104 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The position detection unit 105 analyzes the video and detects at least a position around the object. The position detection unit 105 detects an infrared transmission position (a place where an infrared ID tag exists) in the video by analyzing two or more still images, and detects the transmission position of an object (such as a person or an exhibit). May be detected. Objects such as people and exhibits hold infrared ID tags that transmit infrared rays. The position detection unit 105 can usually be realized by an MPU, a memory, or the like. The processing procedure of the position detection unit 105 is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The scene type determination unit 106 analyzes the content that constitutes the utterance location that is the location of the dialogue, and determines the type of scene. The type of scene is, for example, the first scene “opposite” that shows that a person (a type of object) in the video is looking at an exhibit (a type of object), or a person (object in the video) The second scene where a person is talking to another person (a kind of object), and another person (a kind of object) is talking to the person (a kind of object) in the video The third scene “speech” is the fourth scene “dialogue” in which a person (a kind of object) and another person (a kind of object) are talking to each other. The scene type determination unit 106 can usually be realized by an MPU, a memory, or the like. The processing procedure of the scene type determination unit 106 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The non-display user utterance detection unit 107 analyzes the content constituting the utterance part or the opposite part, and detects that the voice corresponding to the still image to be output is the voice of the user who is not the user who appears in the still image. The non-display user utterance detection unit 107 can be usually realized by an MPU, a memory, or the like. The processing procedure of the non-display user utterance detection unit 107 is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The output unit 108 outputs the one or more still images extracted by the still image extraction unit 104 to one or more windows that do not substantially overlap. This window is usually a tiled window, but may partially overlap. A window is not necessarily a window in a window system. Any display mode may be used. The output unit 108 preferably displays the still image like a cartoon. In the comic technique, there is a technique in which frames overlap, and it is preferable to display a still image according to such a technique. The output unit 108 includes, for example, a still image output unit 1081 and a blowout output unit 1082. However, the blowout output means 1082 is not essential in the output unit 108.

  The still image output unit 1081 outputs the still image extracted by the still image extraction unit 104. A mode of outputting a still image is not limited. Still image output means 1081 may output audio that is synchronized with video including still images. The still image output unit 1081 may or may not include an output device such as a display. The still image output unit 1081 can be realized by driver software of an output device or driver software of an output device and an output device. Output is a concept that includes display on a display, printing on a printer, sound output, transmission to an external device, and the like.

  The blowout output unit 1082 outputs a balloon superimposed on the still image output from the still image output unit 1081. It is preferable that the blowout output means 1082 analyze the audio corresponding to the video having a still image, and distinguish and output two or more types of balloons according to the analysis result. It is preferable that the balloon output means 1082 analyzes the voice corresponding to the video having a still image, acquires the length of the utterance, and outputs a character string having a length corresponding to the length in the balloon. . It is preferable that the blowout output means 1082 outputs a balloon around the position detected by the position detection unit 105. When the non-display user utterance detection unit 107 detects that the voice of a user who is not a user who appears in a still image is detected, the balloon output unit 1082 preferably outputs a balloon in a manner in which a balloon appears from the outside or corner of the window. It is. The “speech balloon” is a frame for adding comments, comments, explanations, and the like, and is used for comics and the like. The form of “speech balloon” and the output mode such as color are not limited. Also, anything may be displayed in the “speech balloon”, or nothing may be displayed. The blowout output means 1082 may be considered as including or not including a display device. The output unit 108 can be realized by, for example, display device driver software, or display device driver software and a display device. The output is a concept including display on a display, printing on a printer, transmission to an external device, and the like.

  Next, the operation of the information processing apparatus will be described using the flowcharts of FIGS. In addition, the example of this operation | movement assumes that it is the following situations, for example. In other words, a user (a kind of object, hereinafter referred to as “the main character” as appropriate) walks through the venue where exhibits, exhibition panels, etc. are displayed, and while viewing the exhibits, and the exhibitors of the exhibits. Talking about the venue. At that time, the main character wears an information acquisition device as shown in FIG. Further, other visitors, exhibitors, and exhibits in the venue are also equipped with information acquisition devices as shown in FIG. And the content which has the image | video, audio | voice, object identifier, etc. which each object (including a visitor, an instructor, an exhibit, etc.) acquired is stored in the information processing apparatus. An operation of the information processing apparatus in such a case will be described.

(Step S <b> 301) The utterance point detection unit 102 acquires the content accumulated by the information acquisition device worn by the hero from the content storage unit 101. Normally, the information processing apparatus receives an input of an object identifier that identifies a hero and acquires content corresponding to the object identifier.
(Step S302) The utterance location detection unit 102 detects an utterance location based on the content acquired in step S301. Details of the processing for detecting an utterance location will be described with reference to the flowcharts of FIGS.

  (Step S303) The facing part detection unit 103 detects a part in the content acquired in step S301, where two or more objects are facing each other. The details of the processing for detecting the opposite location will be described with reference to the flowchart of FIG.

(Step S304) The output unit 108 determines the frame division at the time of output. Frame division is determination of the arrangement and size of windows on the screen. The frame division determination process is, for example, a process of selecting frame division information that is information indicating a predetermined frame division. In the frame division determination process, for example, the number and size of windows may be freely determined. Details of the frame division determination process will be described with reference to the flowchart of FIG.
(Step S305) The output unit 108 substitutes 1 for the counter i.

  (Step S306) The output unit 108 determines whether or not the i-th location (speech location or opposite location) exists. If the i-th location exists, the process goes to step S307, and if the i-th location does not exist, the process goes to step S309.

(Step S307) The output unit 108 determines a frame for outputting one still image constituting the video of the i-th location. For example, the output unit 108 is preferably arranged in a large-sized frame if the time of the i-th location is long, and is arranged in a small-sized frame if the time of the i-th location is short. A detailed algorithm example in this case is as follows. First, the output unit 108 acquires the number P (for example, 10) of all locations. Then, the output unit 108 acquires N (for example, 5) as the number of frames divided in step S304. Next, N frames are arranged in order of frame size. Then, it is assumed that the times of all the locations are acquired and the time of the i-th location is acquired as the M-th (M is, for example, 2) long location. Next, P, N, and M are used as parameters so that a still image corresponding to a portion having a long time is output to a large-sized frame. When P = 10, N = 5, and M = 2, the output unit 108 switches and displays two still images in one frame. Therefore, the output unit 108 displays the i-th still image. The image is assigned so that it is displayed on the largest frame.
(Step S308) The output unit 108 increments the counter i by one. The process returns to step S306.
(Step S309) The output unit 108 substitutes 1 for the counter i.
(Step S310) The output unit 108 determines whether or not the i-th location exists. If the i-th location exists, the process proceeds to step S311. If the i-th location does not exist, the process ends.

(Step S <b> 311) The output unit 108 acquires one still image among videos included in the content at the i-th location. There is no limitation on the algorithm for acquiring one still image. For example, the output unit 108 acquires the middle still image in the time of the i-th location. Further, the output unit 108 may acquire the first still image of the content at the i-th location, for example.
(Step S312) The output unit 108 performs a balloon process. Details of the balloon processing will be described with reference to the flowchart of FIG.

(Step S313) The output unit 108 generates a still image based on the still image acquired in step S311 and the balloon image acquired in step S312. Details of processing for generating a still image to be output will be described with reference to the flowchart of FIG.
(Step S314) The output unit 108 displays the still image generated in step S313 on the assigned frame (window). The assignment of each still image to a frame is determined in step S307.
(Step S315) The output unit 108 increments the counter i by one. The process returns to step S310.
In the flowchart of FIG. 3, a voice corresponding to the i-th location may also be output. For example, when two or more still images are output, one sound may be selected and output.
In the flowchart of FIG. 3, the process may be repeated without repeating and the process from step S301 may be performed again.

  Next, the detailed operation of the processing for detecting the utterance location will be described with reference to the flowcharts of FIGS. The utterance part includes an utterance part where one object talks unilaterally and a conversation part where two objects talk to each other.

  (Step S401) The utterance part detection unit 102 acquires and marks all the parts of the hero's voice that are substantially continuous for a predetermined time or longer. The audio portion may be audio data itself, or may be information on the start point and end point in the content. In addition, the data structure of the voice part is not limited. Normally, in the content storage unit 101, content is managed in pairs with object identifiers. “Marking” means, for example, that “1” is written in the scene type identifier of location information indicating the location of the audio (for example, the main character has an object identifier, start point and end point information, and a scene type identifier). is there. The scene type identifier is information indicating the type of scene (interaction) at the cut out location. For example, “1” is “speech” (unilateral utterance), “2” is “dialogue” (two-way conversation), “3” is “opposite” (the main character sees the exhibition) It is that you are.

(Step S402) The utterance location detection unit 102 acquires and marks all speech locations that are voices other than the hero and are substantially continuous for a predetermined time or longer. The location information indicating the location of the audio includes, for example, an object identifier other than the main character, information on the start point and end point, and a scene type identifier. Then, “1” (indicating utterance) is written in the scene type identifier.
(Step S403) The utterance part detection unit 102 substitutes 1 for the counters i and j.

(Step S404) The speech part detection unit 102 determines whether or not the voice part of the i-th main character exists among the voice parts acquired in Step S401. If there is a voice part of the i-th main character, the process goes to step S405, and if there is no voice part of the i-th main character, the process returns to the upper function.
(Step S405) The utterance part detection unit 102 acquires the voice part of the i-th main character.

(Step S406) The utterance part detection unit 102 determines whether or not there is a jth other person's voice part among the voice parts acquired in Step S402. If the jth other person's voice part exists, the process goes to step S407, and if the jth other person's voice part does not exist, the process goes to step S414.
(Step S407) The utterance location detection unit 102 acquires the speech location of the jth other person.

  (Step S408) The utterance part detection unit 102 determines whether or not the voice part of the i-th main character and the voice part of the j-th other person are substantially continuous. A method for determining whether two voices are substantially continuous will be described with reference to FIG. “Substantially continuous” refers to the situation of (a), (b), and (c) in FIG. That is, (a) indicates that two sounds (sound 1, sound 2) are output continuously with only a slight overlap (x). In this case, the time interval of x is preferably “3 seconds or less”, for example. Further, (b) is a case where two sounds (sound 1 and sound 2) are continuous without overlapping or gaps. Furthermore, (c) is a case where there is a short time interval (y) between two voices (voice 1, voice 2). For example, “y” is preferably “2 seconds or less”. Further, the modes shown in FIGS. 10D and 10E cannot be determined to be “substantially continuous”. This is because the time during which the two speakers speak twice is a predetermined time or more. In FIG. 10, the line indicates that the user is speaking, and the horizontal axis indicates time (t). Voice 1 and voice 2 are voices uttered by different persons (voices paired with different speaker identifiers).

  (Step S409) The utterance part detection unit 102 groups the voice part of the i-th main character and the voice part of the j-th other person. “Grouping two voice locations” means that the two voice locations can be identified as dialogues, for example, the voice location of the i-th main character and the voice of the j-th other person. “2” is written in the scene type identifier of the location, and the two audio locations are associated with each other.

(Step S410) The utterance part detection unit 102 determines whether or not the voice part of the j-th other person is later in time than the voice part of the i-th main character. If so, go to Step S411, and if not, go to Step S412.
(Step S411) The utterance part detection unit 102 increments the counter i by one. The process returns to step S404.
(Step S412) The utterance part detection unit 102 increments the counter j by 1. The process returns to step S404.
(Step S413) The utterance part detection unit 102 increments the counter j by 1. The process returns to step S406.
(Step S414) The utterance location detection unit 102 increments the counter i by 1.
(Step S415) The utterance part detection unit 102 increments the counter j by 1. The process returns to step S404.
Needless to say, the algorithm for detecting the utterance location is not limited to the above.
Next, the operation | movement which acquires another person's audio | voice part is demonstrated using the flowchart of FIG.

(Step S501) The utterance part detection unit 102 acquires all object identifiers included in the content that is paired with the object identifier of the main character. This object identifier is an object identifier acquired by the information acquisition device held by the main character, and is information for identifying a person, an exhibit, or the like facing the main character.
(Step S502) The utterance point detection unit 102 substitutes 1 for a counter i.

(Step S503) The utterance location detection unit 102 determines whether or not the i-th object identifier exists in step S501. The utterance location detection unit 102 goes to step S504 if the i-th object identifier exists, and returns to the upper function if the i-th object identifier does not exist.
(Step S <b> 504) The utterance location detection unit 102 acquires all speech locations that are equal to or longer than a predetermined time in the speech corresponding to the i-th object identifier (speech uttered by the user of the i-th object identifier).
(Step S505) The utterance part detection unit 102 substitutes 1 for a counter j.

  (Step S506) The utterance part detection unit 102 determines whether or not the jth voice part exists in the voice part acquired in step S504. The utterance part detection unit 102 goes to step S507 if the jth voice part exists, and goes to step S511 if the jth voice part does not exist.

  (Step S507) The utterance location detection unit 102 acquires an object identifier corresponding to the jth speech location. This object identifier is an object identifier acquired by the information acquisition device worn by the other person in the time zone of the sound part. That is, the object identified by this object identifier indicates that it is facing the other person in the time zone of the present audio part.

(Step S508) The utterance location detection unit 102 determines whether or not the main character's object identifier exists among the object identifiers acquired in Step S507. If the main character object identifier exists, the process goes to step S509, and if the main character object identifier does not exist, the process jumps to step S510.
(Step S509) The utterance location detection unit 102 acquires and marks the j-th speech location.
(Step S510) The utterance part detection unit 102 increments the counter j by 1. Go to step S506.
(Step S511) The utterance part detection unit 102 increments the counter i by 1. Go to step S503.
Next, details of an example of processing for detecting a facing portion will be described with reference to the flowchart of FIG.

(Step S601) The opposite location detection unit 103 examines an object identifier included in the content corresponding to the object identifier of the main character, and obtains the object identifier acquired almost continuously for a predetermined time or more, and time information that is information about the time. Get all pairs.
(Step S602) The facing part detection unit 103 substitutes 1 for a counter i.

(Step S603) The facing part detection unit 103 determines whether or not the i-th set of information exists in the set of the object identifier and time information acquired in step S601. If the i-th set of information exists, the process goes to step S604, and if the i-th set of information does not exist, the process returns to the upper function.
(Step S604) The facing part detection unit 103 acquires all the object identifiers at the time indicated by the time information included in the information of the set of the content that is paired with the object identifier included in the i-th set of information.

  (Step S605) The facing part detection unit 103 determines whether the object identifier of the main character satisfies a predetermined condition based on the object identifier acquired in Step S604. This predetermined condition is a condition for recognizing that the object identified by the object identifier included in the i-th set of information and the hero face each other for a predetermined time or more. Here, this condition is, for example, that the object identifier of the main character exists without leaving a predetermined interval among the object identifiers indicated by the time information included in the i-th set of information. If the predetermined condition is satisfied, the process proceeds to step S606, and if the predetermined condition is not satisfied, the process jumps to step S608.

(Step S606) The facing part detection unit 103 determines whether or not the hero and the object facing the hero are uttering at the time indicated by the time information included in the i-th set of information. If not uttered, the process goes to step S607, and if uttered, the process jumps to step S608.
(Step S607) The facing location detection unit 103 acquires and marks the facing location based on the time information included in the i-th set of information.
(Step S608) The facing part detection unit 103 increments the counter i by one. Go to step S603.
Next, an example of the operation for determining the frame division will be described with reference to the flowchart of FIG. Here, it is assumed that the output unit 108 stores two or more pieces of frame division information that is information indicating frame division.
(Step S701) The output unit 108 acquires the number of previously detected locations (speech locations, dialogue locations, opposing locations).
(Step S <b> 702) The output unit 108 acquires information regarding the data size (same meaning as time length) of all the previously detected locations.

(Step S703) The output unit 108 determines frame allocation information based on the number of all locations acquired in step S701 and the data size of each location acquired in step S702. For example, the output unit 108 selects frame allocation information with a large number of frames if the number of all locations is large, and selects frame allocation information with a small number of frames if the number of all locations is small. Further, for example, the output unit 108 selects frame allocation information with a small frame size variation if the data size variation is small, and if the data size variation is large, the output unit 108 selects frame allocation information with a large frame size variation. select. However, the algorithm for determining the frame division information is not limited.
Next, a detailed operation example of the balloon process will be described with reference to the flowchart of FIG.
(Step S801) The blowout output means 1082 substitutes 1 for the counter i.

  (Step S802) The blowout output means 1082 determines whether or not the i-th sound is present at a location (such as a dialogue location) corresponding to the still image acquired at Step S311. If the i-th voice exists, the process goes to step S803, and if the i-th voice does not exist, the process jumps to step S812. Note that the i-th sound covers all sounds of the main character and others. The i-th sound is a sound group that has been emitted for a predetermined time or longer.

  (Step S803) The blowout output unit 1082 determines whether or not the i-th sound is the sound of the object (the person shown in the video) of the object identifier included in the content paired with the object identifier that identifies the main character. To do. If it is the sound of the object, the process goes to step S804, and if it is not the sound of the object, the process jumps to step S811.

(Step S804) The position detection unit 105 acquires the position of the object in the still image. For example, the position detection unit 105 can acquire the position of the object by analyzing a video including a still image and recognizing a position emitting infrared rays.
(Step S805) The blowout output unit 1082 acquires the data length of the i-th sound.

  (Step S806) The blowout output unit 1082 determines the type of the balloon based on the data length acquired in Step S805. The blowout output means 1082 selects, for example, jagged balloon information if the voice data length is short, and information on a substantially oval balloon if the voice data length is short.

(Step S807) The blowout output unit 1082 generates a character string based on the data length acquired in Step S805. The character string is meaningless, for example, and the blowout output unit 1082 generates a character string corresponding to the data length using, for example, a random number. The blowout output means 1082 generates, for example, a random number, converts the random number into a character code, and obtains a character. The blowout output unit 1082 repeats such processing according to the data length to generate a character string.
(Step S808) The balloon output unit 1082 generates a balloon image from the balloon information determined and acquired in Step S806 and the character string generated in Step S807.
(Step S809) The balloon output unit 1082 sets the position of the object as the display position of the balloon image.
(Step S810) The blowout output means 1082 increments the counter i by 1. The process returns to step S802.
(Step S811) The blowout output unit 1082 sets the position of the object outside the frame (window). The outside includes corner points, and may be, for example, (0, 0) of relative position coordinates in the window.
(Step S812) The blowout output means 1082 determines whether or not the counter i is 1. If the counter i is 1, the process returns to the upper function. If the counter i is not 1, the process goes to step S813.

(Step S813) The balloon output means 1082 arranges the one or more balloon images generated in step S808 at the display position set in step S809, and synthesizes the balloon image to be combined with the still image. Return to upper function.
Next, a detailed operation example of the output image generation processing will be described with reference to the flowchart of FIG.

(Step S901) The position detection unit 105 acquires the position of the object on the still image to be output. Only one object may be acquired here, or two or more objects may be acquired. For example, the output unit 108 acquires the position of the object on the still image by recognizing the position where the infrared signal is transmitted.
(Step S902) The output unit 108 determines whether or not the type of the scene where the still image is included is a one-way utterance. If it is an utterance, it will go to step S903, and if it is not an utterance, it will jump to step S906.
(Step S903) The output unit 108 determines the highlight shape for emphasizing the object in the still image as the first shape (for example, a circle) corresponding to “utterance”.

(Step S904) The output unit 108 performs a highlight process on the still image based on the highlight shape determined in step S903 or the like. The output unit 108 uses the position acquired in step S901 as the center, changes the shape determined in step S903 to color, and corrects the still image to monochrome in other areas. That is, normally, the output unit 108 performs a process of converting a color image into a monochrome image with respect to the region excluding the region having the shape determined in step S903 with the position acquired in step S901 as the center. Since the process of converting a color image into a monochrome image is a known technique, detailed description thereof is omitted. The highlighting method is not limited to the above. The output unit 108 may output a still image by changing the output mode between a region around the position detected by the position detection unit 105 and another region.
(Step S905) The output unit 108 composes a still image to be output by combining the still image processed in step S904 with the generated balloon image. Return to upper function.
(Step S906) The output unit 108 determines whether or not the type of the scene where the still image is included is a dialogue. If it is a dialog, it will go to step S907, and if it is not a dialog, it will jump to step S908.
(Step S907) The output unit 108 determines a highlight shape for emphasizing an object in the still image as a second shape (for example, an ellipse) corresponding to “dialog”. Go to step S904.
(Step S908) The output unit 108 determines whether or not the scene type of the location including the still image is opposite. If so, go to Step S909, and if not, go to Step S910.
(Step S909) The output unit 108 determines the shape of the highlight for emphasizing the object in the still image as a third shape (for example, a rectangle) corresponding to “opposite”. Go to step S904.
(Step S910) The output unit 108 determines that the highlight shape for emphasizing the object in the still image is the fourth shape (for example, a star shape). Go to step S904.

  Hereinafter, a specific operation of the information processing apparatus according to the present embodiment will be described. In this specific example, it is assumed that the user including the main character is visiting an exhibition or a museum. Here, the exhibition is a technology exhibition. In addition, users including the main character wear an information acquisition device as shown in FIG. In addition, an information acquisition device as shown in FIG. 2 or an information acquisition device without a microphone, a throat microphone, and an HMD is installed in an exhibit (such as a panel for a technical display) as shown in FIG. Has been. Also, in this specific example, information processing when the hero spends several hours while observing the panels of the exhibition, discussing with other people, or explaining the panel etc. to other people. A processing example of the apparatus will be described.

  FIG. 11 shows raw content in which the information acquisition device held by the main character acquires the acquired video, audio, and object identifier in this specific example. The video in FIG. 11 is information acquired by the CCD camera of the information acquisition device held by the main character. The audio | voice of FIG. 11 is the information which the microphone of the said information acquisition apparatus acquires. The object identifier in FIG. 11 is information acquired by the infrared sensor (IR tracker) of the information acquisition apparatus. The object identifier in FIG. 11 indicates an object (person or exhibit) that the main character faces. In FIG. 11, video, audio, and object identifiers are stored in order of acquired time, and are synchronized with each other. In FIG. 11, an object identifier “35” for identifying the holder of the information acquisition device (here, the main character) is stored as header information. For example, the CCD camera of the information acquisition device transmits an object identifier for identifying the holder of the information acquisition device and the acquired video as a pair. Further, for example, the microphone of the information acquisition device transmits the object identifier for identifying the holder of the information acquisition device and the acquired voice as a pair. Further, for example, the IR tracker of the information acquisition device transmits the object identifier for identifying the holder of the information acquisition device and the acquired object identifier as a pair. The raw content in FIG. 11 is a set of information transmitted from the information acquisition device, for example.

  Next, FIG. 12 shows the content acquired by the information acquisition device worn by the user identified by the object identifier “38”. FIG. 13 shows the content acquired by the information acquisition device installed in the display panel identified by the object identifier “01”. In the content of FIG. 13, there is no audio information.

Further, FIG. 14 is an overview of the display panel identified by the object identifier “01”. This exhibition panel has a display for displaying technical explanations, and an infrared ID tag, an IR tracker, and a CCD camera are installed on the top of the display. The infrared ID tag transmits an infrared signal on which an object identifier “01” for identifying the display panel is superimposed. Further, the IR tracker acquires an object identifier transmitted from an infrared ID tag of an information acquisition device worn by a user who is viewing this display panel. Further, the CCD camera captures images of a user or the like who is looking at the display panel and acquires a video.
The content acquired by each of the information acquisition devices is acquired by means (not shown) (communication means, broadcast means, recording medium, etc.), and the information processing apparatus stores the content in the content storage unit 101. To do.
In this case, the utterance location detection unit 102 of the information processing apparatus acquires the content of the main character (object identifier “35”) (the content of FIG. 11) when the predetermined timing comes or according to a user instruction.

Next, the utterance part detection unit 102 detects the utterance part from the content of FIG. 11 by the above-described operation. That is, the utterance part detection unit 102 acquires “t = 0” to “t = 80”, “t = 350” to “t = 720”, and the like. Then, the utterance part detection unit 102 acquires the object identifier “38” of another person who is paired with the voice of “t = 0” to “t = 80”. Next, it is determined whether or not the utterance is made in a section that is substantially continuous with the section of content “t = 0” to “t = 80” of the object identified by the object identifier “38” of another person. Here, in the sections “t = 75” to “t = 200” that are substantially continuous sections from “t = 0” to “t = 80”, the voice corresponding to the object identifier “38” is uttered. It is judged that Next, the utterance point detection unit 102 determines that the voice corresponding to the object identifier “35” is not uttered in a section substantially continuous from the section “t = 75” to “t = 200”. As described above, the utterance point detection unit 102 acquires “t = 200” (this is called the end point) from the section “t = 0” (this is called the start point). In this section, since the conversation between the two people is made, the scene type determination unit 106 determines the scene type as “dialog”.
Next, the utterance point detection unit 102 acquires the corresponding object identifier “40” from the content of FIG. 11 in the section “t = 350” to “t = 720” detected from the content of FIG.

Then, the utterance part detection unit 102 detects from the content (not shown) corresponding to the object identifier “40” that there is no utterance substantially continuous from the section “t = 350” to “t = 720”. Therefore, the utterance location detection unit 102 acquires the sections “t = 350” (start point) and “t = 720” (end point). Next, the scene type determination unit 106 determines the scene type as “utterance”.
Similarly to the above processing, the utterance part detection unit 102 detects a “dialogue” or “speech” part in which the main character is involved, based on the content of FIG.
Next, the facing location detection unit 103 detects a facing location in the content of FIG. 11 based on the processing described above.

  First, the opposite location detection unit 103 checks the object identifier included in the content corresponding to the object identifier “35” of the main character, and obtains the object identifier acquired almost continuously for a predetermined time or more, and time information that is information about the time. Get all pairs. Here, the facing location detection unit 103 detects, for example, a section “t = 3500” (start point) and “t = 3820” (end point). In FIG. 11, in such a section, the object identifier “01” is continuously present for a predetermined time or more. Therefore, the main character has continuously observed the object (exhibition panel) identified by the object identifier “01” during the section “t = 3500” (start point) and “t = 3820” (end point). I understand. Then, the facing location detection unit 103 acquires information of “t = 3500” (start point) and “t = 3820” (end point). Next, the scene type determination unit 106 determines the scene type as “opposite”. Also in other sections, it is needless to say that the facing portion detection unit 103 detects the “facing” portion based on the content of FIG. 11. In the above section, the opposite location detection unit 103 includes the object identifier “35” of the main character among the object identifiers acquired by the information acquisition device installed on the display panel of the object identifier “01” shown in FIG. It may also be determined whether or not the segment is “opposite” by determining whether or not it is included substantially continuously.

  Through the processing described above, the utterance point detection unit 102 and the opposite point detection unit 103 detect “dialogue”, “speech”, and “opposite” points where the main character is involved based on the content shown in FIG. The seed determination unit 106 determines the scene type. Information indicating the detected location and scene type is shown in the location management table of FIG. The location management table is a table that at least temporarily holds information related to locations detected by the speech location detection unit 102 and the opposing location detection unit 103 and the scene type determined by the scene type determination unit 106. The location management table stores one or more records having “ID”, “object identifier”, “start point (t)”, “end point (t)”, and “scene type”. “ID” is an object identifier for identifying information for identifying a record.

  Next, the output unit 108 determines the frame division at the time of output based on the number “218” of locations to be output and the data size (time length of the interval) of each location. Specifically, the output unit 108 holds a large number of frame allocation information shown in FIG. The frame division information may be window information of the frames constituting the screen (window attribute values [position, size, background color, etc.]), bitmap data indicating each frame division in FIG. Does not matter.

  Further, the output unit 108 holds frame allocation determination information, which is information for determining frame allocation, shown in FIG. The frame allocation determination information holds one or more records having “ID”, “frame allocation identifier”, “number of locations”, and “maximum data length (t)”. The “frame allocation identifier” is information for identifying frame allocation information. When the value is “a”, the frame allocation is determined to be the frame allocation in FIG. The “number of places” is a condition for the number of places to be output. “Maximum data length (t)” is a condition of the maximum data length (unit: t (seconds)) among the data lengths of the portions to be output.

In such a case, the output unit 108 selects frame allocation information with reference to the frame allocation determination information in FIG. 17 based on the location management table in FIG. Here, the output unit 108 acquires the number of places “218” in the place management table of FIG. 15. Then, the maximum data length “320” corresponding to each location in the location management table of FIG. 15 is acquired. The maximum data length “320” is a value of “end point−start point” of the record of “ID = 15” in FIG.
Then, the output unit 108 acquires the frame allocation identifier “a” that satisfies the number of places “218” and the maximum data length “320”. Then, the output unit 108 selects the frame allocation information (a) from FIG.

  Next, the still image output unit 1081 acquires one still image among the videos included in the content of each location in the location management table of FIG. Here, it is assumed that the still image output unit 1081 acquires a still image that is substantially in the middle of the video corresponding to each location, for example. It should be noted that the still image output means 1081 can use any algorithm for acquiring a still image in the video. Here, an example of a still image acquired by the still image output unit 1081 one by one from the video corresponding to each location is shown in FIG. FIG. 18A shows a still image corresponding to the location “ID = 1” in FIG. FIG. 18B is a still image corresponding to the location of “ID = 2” in FIG. FIG. 18C is a still image corresponding to the location “ID = 15” in FIG.

  Next, the balloon output means 1082 constitutes a balloon to be added to each still image. Next, an example of processing that constitutes a balloon will be described. First, the blowout output means 1082 holds a balloon management table shown in FIG. The balloon management table stores one or more records having “ID”, “balloon shape”, and “voice length (t)”. The “balloon shape” is an attribute indicating the shape of the balloon, and the attribute value may be a bitmap or graphical data. “Speech length (t)” is the time (t) in this case, indicating the length of the speech. That is, the shape of the balloon changes according to the length of time during which the utterance is made.

  First, the position detection unit 105 analyzes the acquired still image and the still images before and after that (that is, a video), and detects at least the peripheral position of the object. The object normally transmits infrared rays, and the position detection unit 105 acquires a position (coordinates) where infrared rays are transmitted by image processing. Here, for example, it is assumed that the position detection unit 105 acquires the position coordinates (250, 323) of the displayed object based on the still image of FIG.

Next, the blowout output unit 1082 acquires the audio length “200-0 = 200” corresponding to the acquired still image. Next, the blowout output means 1082 determines the type of the balloon corresponding to the voice length (t) “200” as the balloon of “ID = 2” in FIG.
Next, the blowout output means 1082 generates a character string corresponding to the voice length (t) “200”. Here, it is assumed that a character string “#B ··· XY...
Next, the blowout output unit 1082 generates a balloon image based on the balloon shape of “ID = 2” and the character string “# B · XY ... Z ···” in FIG. .

Next, the balloon output means 1082 determines the display position of the balloon as the position coordinates (250, 323) of the displayed object. Note that it is preferable that the blowout output means 1082 is arranged such that the main body of the balloon (a space in which the character string enters) is arranged on the vacant side of the screen with respect to the position coordinates.
Next, the blowout output means 1082 determines whether or not the main character's voice corresponding to the part having the still image exists, and if the voice exists (spoken), acquires the voice length. .

The blowout output unit 1082 sets the position coordinates of the object to (0,500) outside the frame. The balloon output means 1082 generates a character string based on the voice length, and generates a balloon image in the same manner as described above.
Similarly, the blowout output unit 1082 and the position detection unit 105 perform processing to generate a balloon image corresponding to the still image of FIGS. 18B and 18C.
Then, the still image and the balloon image are combined to generate the images shown in FIGS. 20 (a), (b), and (c).

  Next, the output unit 108 performs highlight processing on a still image (for example, FIGS. 20A, 20B, and 20C) by the following processing. Here, the output unit 108 holds the highlight graphic management table shown in FIG. The highlight graphic management table holds one or more records having “ID”, “scene type”, and “highlight graphic”. “Scene type” indicates the type of scene, and “highlight figure” indicates a highlight figure corresponding to the scene type. The highlight figure here is a circle, a square, or a rectangle, but it goes without saying that other figures may be used. Further, the data structure of the attribute value of “highlight figure” does not matter.

  Next, the position detection unit 105 acquires the position of the object on the still image to be output. Next, the output unit 108 acquires a highlight graphic corresponding to the scene type of the still image. That is, the output unit 108 acquires a square highlighted graphic corresponding to the still image scene type “dialog” in FIG. 20A from the highlighted graphic management table in FIG.

  Next, the output unit 108 performs a square highlight process on the still image of FIG. The size of the square may be determined in advance or may be dynamically changed according to the size of the frame. The highlight process is, for example, a process of making the inside of a highlight figure a color image and the outside a monochrome image. In addition, the highlight processing may be performed such that the inside of the highlight figure is displayed in full color and the outside is displayed in 16-bit color image. That is, the highlight process is a process for making the inside of a highlight figure stand out.

  Similarly, the output unit 108 performs a highlight process on the still image in FIGS. 20B and 20C with the shape of the figure corresponding to the scene type centered on the position of the object. As a result, the output unit 108 obtains the still images shown in FIGS. 22 (a), 22 (b) and 22 (c).

  Then, the output unit 108 displays the frame (window) corresponding to each still image. Note that the output unit 108 is preferably an algorithm that arranges the still image in a large-sized frame if the time corresponding to the still image is long.

Further, the output unit 108 sequentially extracts and generates still images corresponding to the location “218” shown in FIG. 15 by the above processing, and arranges them on the frames included in the frame allocation information of FIG. . That is, the output unit 108 displays the still image in the frame, continues the display for a predetermined time (the time may be random), and then displays another still image by overwriting. Therefore, the user sees the still image. That is, the still image extraction unit 104 repeatedly extracts one or more still images constituting the utterance location and / or the opposite location from the video, and the output unit 108 extracts the one or more windows to the still image extraction unit 104. Output while switching still images.
In this case, the entire display image is shown in FIGS. In FIG. 23, there is a space unique to comics between frames. FIG. 24 shows a mode in which there is no space.
As described above, according to the present embodiment, a still image can be extracted from a moving image and displayed in a comic style. In addition, it is possible to realize an effective display such as switching still images in frames constituting a comic display.
In addition, for example, when visiting various exhibitions such as technology, museums and art museums, and further sightseeing, a suitable electronic album is automatically created from the acquired content (including video, audio, etc.) Can be configured.

  Furthermore, the processing in the present embodiment may be realized by software. Then, this software may be distributed by software download or the like. Further, this software may be recorded and distributed on a recording medium such as a CD-ROM. This also applies to other embodiments in this specification. Note that the software that implements the information processing apparatus according to the present embodiment is the following program. In other words, the program includes, on the computer, an utterance location detecting step for detecting an utterance location that is an utterance location based on the voice of content having video and audio, and the utterance location detected in the utterance location detection step. A still image extraction step for extracting one or more still images constituting the image from the video, and an output step for outputting the one or more still images extracted in the still image extraction step to one or more substantially non-overlapping windows. Program.

  In addition, the program stores a location in the content stored in the computer, an opposing location detection step for detecting a location where two or more objects are opposed, and an opposing location detected in the opposing location detection step. A still image extracting step for extracting one or more still images from the constituting video and an output step for outputting the one or more still images extracted in the still image extracting step to one or more windows that do not substantially overlap each other are executed. It is a program for.

  Further, in the still image extraction step in the program, one or more still images constituting an utterance location or / and an opposite location are repeatedly extracted from the video, and in the output step, the still image is displayed in the one or more windows. It is preferable to output while switching the still images extracted in the extraction step.

In addition, it is preferable that the output step in the program includes a still image output substep for outputting the still image extracted in the still image extraction step, and a blowout output substep for outputting a balloon over the still image. It is.
Further, in the blowout output sub-step in the program, it is preferable to analyze the audio corresponding to the video having the still image and distinguish and output two or more types of balloons according to the analysis result.

  Also, in the blowing output sub-step in the above program, the voice corresponding to the video having the still image is analyzed, the length of the utterance is acquired, and the character string having a length corresponding to the length is output in the balloon. It is preferable to do.

  Further, the program causes the computer to further execute a position detection step of analyzing the video and detecting a position of at least the periphery of the object, and in the blowing output sub-step, the balloon is blown around the position detected in the position detection step. Is preferably output.

  Further, the program analyzes the content constituting the utterance location or the opposite location on a computer, and detects that the audio corresponding to the output still image is the audio of a user who is not a user who appears in the still image. If the non-display user utterance detection step is further executed and it is detected in the blowout output sub-step that the voice of a user who is not a user who appears in a still image is detected in the non-display user utterance detection step, the outside or corner of the window is detected. It is preferable to output the speech balloon in such a manner that the speech balloon appears.

  Further, the program causes the computer to further execute a position detection unit that analyzes the video and detects a position of at least the periphery of the object, and in the output step, a region around the position detected in the position detection step It is preferable to output the still image by changing the output mode with other regions.

  In the above program, the video information includes an object identifier for identifying a video and an object appearing in the video, and the audio information includes an object identifier for identifying a voice and a voice speaker. In the detecting step, the dialogue is an utterance portion where the volume of the voice is a predetermined size or more, and has a voice paired with one object identifier and a voice paired with another substantially continuous object identifier. It is preferable to detect the utterance location that is the location of.

  Further, in the above program, the content constituting the utterance location that is the location of the dialogue is analyzed, and scene type determination for determining the type of scene is further executed, and in the output step, based on the type of scene, It is preferable that the shape of the area around the position detected in the position detection step is different.

  In each of the above embodiments, each process (each function) may be realized by centralized processing by a single device (system), or by distributed processing by a plurality of devices. May be.

  FIG. 25 shows the external appearance of a computer that executes the programs described in this specification to realize the information processing apparatuses according to the various embodiments described above. The above-described embodiments can be realized by computer hardware and a computer program executed thereon. FIG. 25 is an overview of the computer system 250, and FIG. 26 is a block diagram of the system 250.

  25, a computer system 250 includes a computer 251 including an FD (Flexible Disk) drive and a CD-ROM (Compact Disk Read Only Memory) drive, a keyboard 252, a mouse 253, a monitor 254, and a speaker 255. .

  26, in addition to the FD drive 2511 and the CD-ROM drive 2512, the computer 251 includes a CPU (Central Processing Unit) 2513, a bus 2514 connected to the CPU 2513, the CD-ROM drive 2512, and the FD drive 2511, and a boot. A ROM (Read-Only Memory) 2515 for storing programs such as an up program, and a RAM (Random Access Memory) connected to the CPU 2513 for temporarily storing instructions of application programs and providing a temporary storage space 2516 and a hard disk 2517 for storing application programs, system programs, and data. Although not shown here, the computer 251 may further include a network card that provides connection to the LAN.

  A program that causes the computer system 250 to execute the functions of the information processing apparatus according to the above-described embodiment is stored in the CD-ROM 2601 or the FD 2602, inserted into the CD-ROM drive 2512 or the FD drive 2511, and further stored in the hard disk 2517. May be forwarded. Alternatively, the program may be transmitted to the computer 251 via a network (not shown) and stored in the hard disk 2517. The program is loaded into the RAM 2516 when executed. The program may be loaded directly from the CD-ROM 2601, the FD 2602, or the network.

  The program does not necessarily include an operating system (OS) or a third-party program that causes the computer 251 to execute the functions of the information processing apparatus according to the above-described embodiment. The program only needs to include an instruction portion that calls an appropriate function (module) in a controlled manner and obtains a desired result. How the computer system 250 operates is well known and will not be described in detail.

In the above program, the step of outputting information does not include processing performed by hardware, for example, processing performed by a monitor in the outputting step (processing performed only by hardware).
Further, the computer that executes the program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.
The present invention is not limited to the above-described embodiments, and various modifications are possible, and it goes without saying that these are also included in the scope of the present invention.

  As described above, the information processing apparatus according to the present invention has an effect of appropriately extracting and outputting a predetermined still image from a moving image, and is useful as an electronic album apparatus or the like.

Block diagram of an information processing apparatus in an embodiment The figure which shows the example of the same information acquisition device Flow chart for explaining the operation of the information processing apparatus Flowchart explaining processing for detecting the same utterance location Flowchart explaining processing for detecting the same utterance location Flowchart explaining processing for detecting the opposite location Flowchart explaining same frame division determination processing Flowchart explaining the balloon process Flow chart explaining the output image generation processing The figure explaining the judgment method of whether the two said audio | voices are substantially continuous. Diagram showing the same content example Diagram showing the same content example Diagram showing the same content example Overview of the panel Figure showing the same location management table Figure showing the same frame allocation information The figure which shows the same frame allocation decision information The figure which shows the example of the still image which the same image output means acquired Figure showing the balloon management table The figure which shows the example of the picture which synthesizes the still picture and the balloon picture Figure showing the same highlight graphic management table The figure which shows the example of the still image which the output part 108 obtains Figure showing an example of the entire display image Figure showing an example of the entire display image The figure which shows the external appearance of the computer which implement | achieves the information processing apparatus Block diagram of the computer system

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Content storage part 102 Speech part detection part 103 Opposite part detection part 104 Still image extraction part 105 Position detection part 106 Scene type determination part 107 Non-display user utterance detection part 108 Output part 1081 Still picture output means 1082 Outlet output means

Claims (9)

A content storage unit storing at least one content including video information including video and audio information including audio;
Based on the voice, an utterance location detector that detects an utterance location that is an utterance location,
A still image extraction unit for extracting one or more still images constituting the utterance location detected by the utterance location detection unit from the video;
An information processing apparatus comprising: an output unit that outputs one or more still images extracted by the still image extraction unit to one or more windows that do not substantially overlap. A content storage unit storing at least one content including video information including video and audio information including audio;
A facing part detection unit that detects a part in the content and two or more objects are facing each other;
A still image extraction unit that extracts one or more still images from the video that configures the opposite location detected by the opposite location detection unit;
An information processing apparatus comprising: an output unit that outputs one or more still images extracted by the still image extraction unit to one or more windows that do not substantially overlap. The still image extraction unit
One or more still images constituting the utterance location or / and the opposite location are repeatedly extracted from the video,
The output unit is
The information processing apparatus according to claim 1, wherein the still image extracted by the still image extraction unit is output to the one or more windows while being switched. The output unit is
A still image output means for outputting the still image extracted by the still image extraction unit;
The information processing apparatus according to claim 1, further comprising a blowout output unit that outputs a blowout on the still image. Further comprising a position detector for analyzing the video and detecting a position of at least the periphery of the object;
The blowout output means includes
The information processing apparatus according to claim 4, wherein a balloon is output around the position detected by the position detection unit. A non-display user utterance detecting unit that analyzes the content constituting the utterance location or the opposite location and detects that the audio corresponding to the output still image is the audio of a user who is not a user appearing in the still image. And
The blowout output means includes
6. The speech balloon is output in such a manner that a speech balloon appears outside or at a corner of the window when the non-display user utterance detection unit detects a voice of a user who is not a user who appears in a still image. Information processing device. The output unit is
The information processing apparatus according to claim 5, wherein an area around the position detected by the position detection unit is in color and the other area is monochrome, and outputs the still image. On the computer,
An utterance location detection step of detecting an utterance location that is an utterance location based on the audio of content having video and audio,
A still image extraction step of extracting one or more still images constituting the utterance location detected in the utterance location detection step from the video;
A program for executing an output step of outputting one or more still images extracted in the still image extraction step to one or more windows that do not substantially overlap. On the computer,
A facing location detection step of detecting a location in the stored content and a location where two or more objects are facing,
A still image extraction step of extracting one or more still images from the video constituting the opposite location detected in the opposite location detection step;
A program for executing an output step of outputting one or more still images extracted in the still image extraction step to one or more windows that do not substantially overlap.

Images