JP2010181461A - Digital photograph frame, information processing system, program, and information storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a digital photograph frame that achieves display control in which sound information acquired by a sound sensor is reflected, an information processing system, a program and an information storage medium. <P>SOLUTION: The digital photograph frame 300 includes a display part 340 displaying an image, a display controller 318 performing display control of the display part 340, and a sound information acquiring part 304 acquiring the sound information detected by the sound sensor for detecting ambient sound. The display controller 318 performs the display control to change the display form of the image displayed on the display part 340 in accordance with the acquired sound information. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a digital photo frame, an information processing system, a program, an information storage medium, and the like.

  In recent years, a digital photo frame has been spotlighted as a device that can easily reproduce an image taken by a digital camera such as a digital still camera. This digital photo frame is a device in which a portion for putting a photo on a photo stand is replaced with a liquid crystal display, and performs reproduction processing of digital image data (electrophotography) read via a memory card or a communication device. .

  As a conventional technique of a digital photo frame, for example, there is a technique disclosed in Patent Document 1. In this prior art, a telephone line connection device is provided in a digital photo stand, which is a digital photo frame, to form a transmission path between the photo stand and a wired or wireless telephone line.

  However, digital photo frames so far have only a function of reproducing an image taken with a digital camera or the like, and display control reflecting ambient sounds, user utterances, and the like has not been performed. For this reason, the reproduced image is monotonous and various contents cannot be displayed to the user.

JP 2000-324473 A

  According to some aspects of the present invention, it is possible to provide a digital photo frame, an information processing system, a program, an information storage medium, and the like that enable display control that reflects sound information acquired by a sound sensor.

  One aspect of the present invention includes a display unit that displays an image, a display control unit that performs display control of the display unit, and a sound information acquisition unit that acquires sound information detected by a sound sensor that detects ambient sound. The display control unit includes a digital photo frame that performs display control to change a display mode of an image displayed on the display unit in accordance with the acquired sound information.

  According to one aspect of the present invention, sound information detected by a sound sensor is acquired. And display control of the display part of a digital photo frame is performed so that the display mode of the image displayed on a display part changes according to the acquired sound information. Therefore, display control reflecting sound information acquired by the sound sensor is possible, and various image expressions can be realized.

  Moreover, in one aspect of the present invention, an utterance detection unit that detects a user's utterance based on the acquired sound information is included, and the display control unit is connected to the display unit based on a detection result of the user's utterance. Display control for changing the display mode of the displayed image may be performed.

  If it does in this way, a display image will come to change according to a user's utterance, and display control which reflected a user's utterance will be attained.

  In one aspect of the present invention, the display control unit may perform control to display detailed information or related information of display content when a user's utterance is detected.

  In this way, for example, when the user utters interest with respect to the displayed content, the detailed information and related information of the display content will be displayed according to the utterance, which has never existed before A digital photo frame interface environment can be provided.

  According to another aspect of the present invention, the display control unit includes a visual recognition state determination unit that determines a visual recognition state of the user, and the display control unit includes: Display control for changing the display mode of the displayed image may be performed.

  In this way, it is possible to perform display control that reflects not only the user's utterance detection result but also the user's visual recognition state.

  In the aspect of the invention, the display control unit may control the display timing of the content based on the detection result of the user's utterance.

  In this way, the user can change the display timing of the content by speaking and can provide an unprecedented type of interface environment.

  In one aspect of the present invention, the utterance detection unit detects a user's non-utterance state, and the display control unit moves to the next content in the slide display of the content when the user's non-utterance state is detected. The switching process may be performed.

  In this way, the display of the current display content is maintained when the user is speaking with interest in the display content, and when the no-speaking state is detected, the next content is displayed, so the slide display A suitable interface environment can be provided.

  According to another aspect of the present invention, a voice recognition unit that performs voice recognition of a user's utterance based on the acquired sound information is included, and the display control unit displays on the display unit based on a voice recognition result. Display control for changing the display mode of the image to be displayed may be performed.

  In this way, the display mode of the display image of the digital photo frame changes according to the voice recognition result of the user's utterance, and display control that reflects the voice recognition result of the user's utterance becomes possible.

  In one aspect of the present invention, the voice recognition unit extracts a user's utterance keyword by voice recognition, and the display control unit performs control to display content selected based on the extracted utterance keyword. Also good.

  In this way, when the user utters, the content selected by the utterance keyword constituting the utterance is displayed, and display control by the user's utterance keyword becomes possible.

  In one aspect of the present invention, the display control unit may perform control to display content corresponding to the tag keyword when the tag keyword associated with the display content matches the utterance keyword. .

  In this way, if a user who is interested in the displayed content utters an utterance keyword related to the content, and the utterance keyword matches the tag keyword of the display content, the matched tag keyword The content corresponding to is displayed. As a result, the intention of the user can be reflected in the content of the content to be displayed next, and a digital photo frame of an unprecedented type can be provided.

  In one aspect of the present invention, the display control unit is selected by AND search of the tag keyword and the utterance keyword when the tag keyword associated with the display content does not match the utterance keyword. You may perform control which displays a content.

  In this way, when the user utters an utterance keyword with respect to the displayed content and the utterance keyword does not match the tag keyword of the display content, it is selected by AND search of the utterance keyword and the tag keyword. The content will be displayed. As a result, it becomes possible to display the next content while reflecting the user's interests and intentions to some extent, and a variety of content can be displayed with various changes.

  In one aspect of the present invention, the display control unit displays content corresponding to the tag keyword when the tag keyword and the utterance keyword match, and the tag keyword and the utterance keyword are one. If not, control may be performed to display the content selected by AND search of the tag keyword and the utterance keyword.

  In this way, when the tag keyword and the utterance keyword match, the content corresponding to the tag keyword is displayed, and when the tag keyword does not match, the content selected by the AND search is displayed. Various content displays that reflect the user's interests and intentions to some extent are possible.

  In one aspect of the present invention, the display control unit performs control to display content selected by AND search of a plurality of different utterance keywords when a plurality of different utterance keywords are extracted from the user's utterance. Also good.

  In this way, it becomes possible to display the next content by analogizing the content desired by the user from a plurality of utterance keywords constituting the user's utterance.

  Moreover, in one aspect of the present invention, the display control unit includes a visual recognition state determination unit that determines a visual recognition state of the user, and the display control unit determines a result of speech recognition and a determination result of the visual recognition state of the user when the user's speech is detected. Based on the above, display control for changing the display mode of the image displayed on the display unit may be performed.

  In this way, when a user's utterance is detected, display control that reflects both the speech recognition result of the utterance and the user's visual state is possible, and various content displays are possible.

  In one aspect of the present invention, the display control unit is selected by a tag keyword associated with display content when a user's utterance is detected and it is determined that the user is gazing at the display unit. Control may be performed to display the displayed content.

  In this way, when the user speaks to the display content and the user watches the display content, the user thinks that the user is interested in the display content, and the tag keyword of the display content. The content selected by can be displayed.

  In one aspect of the present invention, the display control unit is selected by the tag keyword when it is determined that the user is gazing at the display unit and the tag keyword matches the utterance keyword. You may perform control which displays a content.

  In this way, when the user who is watching the display content utters an utterance keyword that matches the tag keyword of the display content, the content selected by the tag keyword can be displayed as the content to be displayed next. It becomes like this.

  In one aspect of the present invention, the display control unit displays content corresponding to the utterance keyword when it is determined that the user's utterance is detected but the user does not watch the display unit. Control may be performed.

  In this way, when the user utters but does not watch the display content, the user's utterance keyword is given priority, and the content corresponding to the utterance keyword can be displayed as the next content to be displayed. Become.

  In the aspect of the invention, the display control unit may count the number of appearances of the extracted utterance keyword and perform control to display content corresponding to the utterance keyword whose appearance number exceeds a predetermined number.

  In this way, it is possible to prevent a situation in which the content display frequently changes in response to the user's utterance.

  In one aspect of the present invention, the display control unit may display a display object associated with the matched tag keyword when tag keywords associated with the plurality of display objects constituting the content image match the utterance keyword. You may perform control which expands and displays detailed information or related information of the display object concerned.

  In this way, it becomes possible to enlarge and display a display object that the user is interested in, and to display its detailed information and related information.

  In the aspect of the invention, the display control unit may perform control to sequentially enlarge and display the plurality of display objects when the tag keyword does not match the utterance keyword.

  In this way, even when the tag keyword and the utterance keyword do not match, a plurality of display objects are sequentially enlarged and displayed, and an unprecedented type of content can be displayed.

  In the aspect of the invention, the voice recognition unit may recognize a user's speaker from the user's uttered voice, and the display control unit may perform control to display content corresponding to the user who is recognized by the speaker. Also good.

  If it does in this way, the user's speaker recognition will be performed from a user's utterance voice, and the content which reflected the user's hobbies, tastes, etc. can be displayed to a user.

  In one aspect of the present invention, the voice recognition unit recognizes a user's emotional state from the user's uttered voice, and the display control unit performs control to display content corresponding to the recognized emotional state. Also good.

  In this way, it is possible to display content that reflects the emotional state of the user by effectively utilizing the sound information acquired by the sound sensor.

  Moreover, in one aspect of the present invention, a positional relationship determination unit that determines a positional relationship between a user and the display unit, the display control unit, when the distance between the user and the display unit is within a predetermined distance, Display control may be performed to change a display mode of an image displayed on the display unit according to the acquired sound information.

  In this way, display control based on sound information can be executed after the user approaches the digital photo frame and can detect the user's voice.

  In the aspect of the invention, the display control unit may perform control to turn on the display of the display unit based on the acquired sound information.

  In this way, it is possible to realize display on control of the display unit of the digital photo frame by effectively utilizing the sound information acquired by the sound sensor.

  In the aspect of the invention, the display control unit may specify a user operation instruction based on the acquired sound information, and may perform display control according to the specified operation instruction.

  In this way, it is possible to effectively use the sound information acquired by the sound sensor to realize an operation instruction for the digital photo frame.

  Another aspect of the present invention includes a content selection unit that selects content, and a display instruction unit that instructs display of an image displayed on the display unit of the digital photo frame based on the selected content, The display instruction unit relates to an information processing system that issues a display instruction to change a display mode of an image displayed on the display unit according to sound information detected by a sound sensor that detects ambient sounds.

  According to another aspect of the present invention, the display control of the display unit of the digital photo frame is instructed so that the display mode of the image displayed on the display unit changes according to the sound information detected by the sound sensor. The Therefore, display control reflecting sound information acquired by the sound sensor is possible, and various image expressions can be realized.

  Another aspect of the present invention relates to a program that causes a computer to function as each of the above-described units, or a computer-readable information storage medium that stores the program.

1A and 1B are diagrams illustrating examples of digital photo frames. 2 is a configuration example of a digital photo frame according to the present embodiment. 3A and 3B are explanatory diagrams of a display control method based on utterance detection. 4A and 4B are explanatory diagrams of a display control method based on the utterance detection result and the visual recognition state determination result. The flowchart for demonstrating the display control method based on speech detection. 7 is a flowchart for explaining gaze state determination processing. FIGS. 7A and 7B are explanatory diagrams and flowcharts of another example of the display control method based on utterance detection. Explanatory drawing of the display control method by voice recognition. Explanatory drawing of the display control method by voice recognition. Explanatory drawing of the display control method by voice recognition. Explanatory drawing of the display control method by voice recognition. The flowchart for demonstrating the 1st display control method by speech recognition. The flowchart for demonstrating the 2nd display control method by speech recognition. The flowchart for demonstrating the method of counting the frequency | count of appearance of an utterance keyword and selecting display content. FIGS. 15A and 15B are explanatory diagrams of a display control method in the case where a content image is composed of a plurality of display objects. The flowchart for demonstrating the display control method in case a content image is comprised with a some display object. Explanatory drawing of the display control method using a user's registration information. FIG. 18A and FIG. 18B are explanatory diagrams of a display control method using user registration information. 19A and 19B are explanatory diagrams of a display control method using a positional relationship with a user. FIGS. 20A to 20C are explanatory diagrams of a method for detecting the positional relationship with the user. The flowchart for demonstrating the method of turning on a display and specifying an operation instruction | indication based on sound information. The modification of the system configuration | structure of this embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Configuration FIG. 1A shows an example of a digital photo frame 300 (digital photo player, image reproduction apparatus) of this embodiment. FIG. 1A illustrates an example of a so-called photo stand type digital photo frame. The digital photo frame 300 is installed by a user in an arbitrary place such as a house. Then, reproduction processing (image reproduction, sound reproduction) of content information such as digital image data and sound data is executed. The digital photo frame 300 can automatically play back content information (media information) such as an image without an explicit playback instruction from the user, for example. For example, a photo slideshow is automatically executed, or an image is automatically reproduced.

  Although FIG. 1A shows an example of a photo stand type digital photo frame, for example, as shown in FIG. As this wall-mounted digital photo frame, for example, electronic paper realized by an electrophoretic display or the like can be used. The digital photo frame 300 may be provided with a content information reproduction instruction button or the like, or a reproduction instruction may be performed using a remote controller.

  The digital photo frame 300 can include a memory card interface such as an SD card, for example. Alternatively, a wireless communication interface such as a wireless LAN or Bluetooth, or a wired communication interface such as USB can be provided. For example, when a user saves content information in a memory card and attaches it to the memory card interface of the digital photo frame 300, the digital photo frame 300 executes automatic reproduction (slideshow or the like) of the content information saved in the memory card. . Alternatively, when the digital photo frame 300 receives content information from the outside through wireless communication or wired communication, the digital photo frame 300 executes a reproduction process (automatic reproduction process) of the content information. For example, when a portable electronic device such as a digital camera or a mobile phone possessed by a user has a wireless function such as Bluetooth, content information is transferred from the portable electronic device to the digital photo frame 300 using this wireless function. Forward. Then, the digital photo frame 300 executes a reproduction process of the transferred content information.

  FIG. 2 shows a configuration example of a digital photo frame (image display device in a broad sense) 300 according to the present embodiment. The digital photo frame 300 includes a processing unit 302, a storage unit 320, a communication unit 338, a display unit 340, a sensor 350, and an operation unit 360. Various modifications such as omitting some of these components (for example, a communication unit, an operation unit, and a sensor) and adding other components (for example, a speaker) are possible.

  The processing unit 302 performs various control processes and arithmetic processes. For example, the above-described units such as the storage unit 320 and the display unit 340 of the digital photo frame 300 are controlled or overall control is performed. The functions of the processing unit 302 can be realized by hardware such as various processors (CPU and the like), ASIC (gate array and the like), a program stored in the information storage medium 330 connected to the processing unit 302, and the like.

  The storage unit 320 serves as a work area for the processing unit 302, the communication unit 338, and the like, and its functions can be realized by a memory such as a RAM, an HDD (hard disk drive), or the like. The storage unit 320 includes a content information storage unit 322 that stores content information such as images and sounds, a sound information storage unit 324 that stores acquired sound information, a detection information storage unit 325 that stores acquired detection information, Registration keyword storage unit 326 for storing registration keywords (general registration keywords, user registration keywords), user state storage unit 327 for storing specified user states, and users for storing user information such as user registration information and sensitivity model information An information storage unit 328 is included.

  The information storage medium 330 (a computer-readable medium) stores programs, data, and the like, and its function can be realized by a memory card, an optical disk, or the like. The processing unit 302 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 330. That is, in the information storage medium 330, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit). Is memorized.

  The communication unit 338 (communication interface) exchanges information with an external device (for example, a server or a portable electronic device) by wireless or wired communication, and functions as a communication ASIC or communication. This can be realized by hardware such as a processor or communication firmware.

  The display unit 340 is for displaying an image as content information, and can be realized by, for example, a liquid crystal display, a display using a light emitting element such as an organic EL, an electrophoretic display, or the like. The display unit 340 may be configured with a touch panel display (touch screen).

  The sensor 350 (sound sensor, user detection sensor, etc.) is a device that outputs detection information based on the detection result. For example, a sound sensor can be used as the sensor 350. The sound sensor is a sensor that converts sound into an electrical signal or the like, and is a microphone that measures sound pressure, which is a physical quantity of sound. Microphones include moving coil type and ribbon type dynamic microphones, condenser type microphones that detect changes in capacitance due to vibration of audio signals, piezoelectric microphones that use the piezoelectric effect, and carbon type microphones. Etc. can be used.

  As the sensor 350, a user detection sensor can be used. User detection sensors include human sensors such as pyroelectric sensors, imaging sensors such as CCD and CMOS sensors, distance sensors such as ultrasonic sensors, and motion that detects the user's operating state (hand and body movements). A sensor or the like can be used.

  The pyroelectric sensor is a sensor that receives infrared rays generated by a person or the like, converts the infrared rays into heat, and converts the heat into electric charges by the pyroelectric effect of the element. By using this pyroelectric sensor, it is detected whether there is a user (person) in the detection range (detection area), the movement of the user in the detection range, the number of users in the detection range, etc. it can. An imaging sensor (image sensor) is an optical sensor that converts one-dimensional or two-dimensional optical information into a time-series electrical signal. By using this image sensor, it is possible to detect whether or not there is a user in the detection range, user movements in the detection range, the number of users in the detection range, and the like. Further, user authentication can be realized by face image recognition using an imaging sensor. Further, the positional relationship such as the distance between the user and the display unit 340 and the angle of the user's line of sight with respect to the display unit 340 can be detected by face detection using an imaging sensor. Alternatively, it is possible to detect a user's visual state such as whether or not the display unit 340 is in the user's visual field range and whether or not the user is gazing at the display unit 340. Alternatively, whether or not the user is approaching can also be detected.

  The sensor 350 may be the sensor device itself, or may be a sensor device including a control unit, a communication unit, and the like in addition to the sensor device. The detection information may be primary information obtained directly from the sensor, or may be secondary information obtained by processing (information processing) the primary information.

  The sensor 350 may be directly attached to the digital photo frame 300, or a home sensor or the like may be used as the sensor 350. When the sensor 350 is attached to the digital photo frame 300, the sensor 350 can be attached to, for example, a frame portion of the digital photo frame 300 as shown in FIG. Alternatively, the sensor 350 and the digital photo frame 300 may be connected using a wired cable or the like.

  The operation unit 360 is used by the user to input various types of information, and can be realized by, for example, an operation button or a remote control device. The user can use this operation unit 360 to perform user registration and registration of playback content (favorite images) desired by the user. For example, the user can input user registration information using the operation unit 360. When the display unit 340 is configured by a touch panel display, the display unit 340 also functions as the operation unit 360.

  The processing unit 302 includes a sound information acquisition unit 304, a detection information acquisition unit 305, an utterance detection unit 306, a voice recognition unit 307, a user state determination unit 310, a registration processing unit 314, a content selection unit 316, and a display control unit 318. Various modifications such as omission of some of these components (for example, a detection information acquisition unit, a user status determination unit, a registration processing unit, and a content selection unit) and addition of other components are possible. It is.

  The sound information acquisition unit 304 performs processing for acquiring sound information detected by a sound sensor that is one of the sensors 350. For example, when a sound (sound, music, etc.) (ambient sound) is detected by a sound sensor and sound information as a detection result is output, the sound information acquisition unit 304 takes in the sound information. The acquired sound information is stored in the sound information storage unit 324 of the storage unit 320. Similarly, when a user state or the like is detected by a user detection sensor that is one of the sensors 350 and detection information (imaging information or the like) that is a detection result is output, the detection information acquisition unit 305 outputs the detection information. take in. The captured detection information is stored in the detection information storage unit 325.

  When an external sensor such as a home sensor is used as the sensor 350, the communication unit 338 receives sound information and detection information, and the sound information acquisition unit 304 and the detection information acquisition unit 305 receive the received sound information and detection. You will get information.

  The utterance detection unit 306 detects the user's utterance (conversation) based on sound information acquired by the sound sensor which is the sensor 350. For example, a sound having a frequency component or amplitude (power) peculiar to human speech is extracted, and the user's speech is distinguished from noise and detected. Specifically, a user's utterance is detected by performing a filter process or the like that passes a specific frequency band for a sound having a certain amplitude level or higher. Note that the detection target based on the sound information is not limited to the user's utterance, and may be a sound, noise, life sound, or the like, or a cough, sneeze, or applause sound uttered by the user.

  The voice recognition unit 307 performs voice recognition of the user's utterance based on the sound information acquired by the sound sensor. Then, the user's utterance keyword (word) is extracted by voice recognition. Alternatively, the user's speaker recognition is performed from the user's uttered voice or the emotional state of the user is recognized.

  Here, speech recognition is a process of analyzing a speech language spoken by a human (performer, user) by a computer and extracting it as text data. Speech recognition is realized by acoustic analysis, a recognition decoder, an acoustic model, a dictionary, and a language model. In acoustic analysis, human speech is converted into feature information by signal processing such as Fourier analysis. The recognition decoder outputs text data based on the feature amount information. Specifically, sound is converted into text data by comprehensively determining acoustic information and language information. The determination process in the recognition decoder is realized by a statistical method such as a hidden Markov model or a dynamic time expansion / contraction method. A dictionary is a data of words (words) to be recognized, and associates phoneme strings with words. The language model is a data of probabilities for words in the dictionary. Specifically, the appearance probability and connection probability of each word are converted into data. By such speech analysis using the acoustic analysis, recognition decoder, acoustic model, dictionary, and language model, the user's utterance keyword can be appropriately extracted from the sound information acquired by the sound sensor. Note that the utterance detection by the utterance detection unit 306 can be realized by using voice recognition by the voice recognition unit 307. That is, the user's speech is detected by utilizing the function of the voice recognition engine.

  The user state determination unit 310 determines the user state based on the detection information acquired by the detection information acquisition unit 305. For example, based on the acquired detection information, the visual recognition state of the user with respect to the display unit 340, the positional relationship between the user (person) and the display unit 340, and the like are determined. Information on the user state representing the user's visual state and positional relationship is stored in the user state storage unit 327.

  Here, the viewing state is a state of the user's visual field range, a gaze state, or the like. Specifically, whether or not the display unit 340 is included in the user's visual field range (view volume), or the user sets the display unit 340. Whether or not they are gazing. The visual recognition state determination unit 311 determines the visual recognition state of the user. For example, it is determined whether the user is gazing at the display unit 340 as the user's viewing state.

  The positional relationship is the distance between the user and the display unit 340, the user's line-of-sight direction with respect to the display unit 340, and the positional relationship determination unit 312 determines the positional relationship between the user and the display unit 340. For example, as a positional relationship between the user and the display unit 340, a distance (distance information, distance parameter) between the user and the display unit 340 is determined.

  It is assumed that an image sensor that images the user is provided as the sensor 350. In this case, the user state determination unit 310 (positional relationship determination unit) detects the user's face area (rectangular frame area) based on the imaging information from the imaging sensor. Based on the detected size of the face area, the distance between the user and the display unit 340 is determined (estimated). In addition, the user state determination unit 310 sets a measurement area that includes the detected face area and is larger than the face area. That is, a measurement area that overlaps the face area is set. Then, the time during which the face area exists in the measurement area is measured, and based on the measured time, it is determined whether or not the user is gazing at the display unit 340. For example, when the presence time in the measurement area is a predetermined time or more, it is determined that the user is gazing.

  Alternatively, the user state determination unit 310 may determine the distance between the user and the display unit 340 by an automatic focusing process (autofocus function) for the user. For example, when the active method is adopted, a device for emitting infrared rays or ultrasonic waves is provided on the digital photo frame 300 or the like, and an infrared or ultrasonic light receiving sensor is provided as the sensor 350. And what is necessary is just to detect the distance with a user, etc. by detecting the reflected light from a user with a light receiving sensor. When the passive method is adopted, an image sensor is provided as the sensor 350, and the distance to the user or the like may be detected by performing image processing on the captured image using a phase difference detection method or a contrast detection method. .

  The registration processing unit 314 performs user registration processing. For example, user registration information setting processing is performed. Specifically, when the user inputs user registration information using the operation unit 360 or the like on the user registration screen or the like, the input user registration information is associated with the user and stored in the user information storage unit 328. Here, the user registration information can include, for example, a user ID, a password, customization information of an image displayed on the display unit 340, and the like. The display unit 340 displays an image reflecting the user registration information.

  In the user registration process, the user's voice may be registered as user registration information. For example, the feature amount information of the user's voice is stored and registered in the user information storage unit 328 as user registration information. When speaker recognition is performed by the voice recognition unit 307, the sound information acquired by the sound sensor is compared with the registered feature information of the user's voice, and the user who is speaking is the registered user. Determine whether or not. Alternatively, an image sensor may be provided as the sensor 350, the user may be imaged by the image sensor, and the feature amount information of the user's face image may be registered as user registration information. In this case, the image information acquired by the image sensor and the feature amount information of the user's face image are compared, and person authentication is performed to determine whether the imaged user is a registered user.

  The content selection unit 316 performs processing for selecting content to be presented to the user. For example, based on the results of speech detection and speech recognition based on the acquired sound information, the corresponding content information is read from the content information storage unit 322, and the content to be presented to the user is selected. Alternatively, content is selected by receiving content information from an external server such as a home server via the communication unit 338.

  The display control unit 318 performs display control of the display unit 340. For example, when the content selection unit 316 reads content information from the content information storage unit 322 or receives content information by the communication unit 338 and selects content, the display control unit 318 displays an image of the selected content. Control for displaying on the unit 340 is performed.

  In the present embodiment, when the sound information acquisition unit 304 acquires the sound information detected by the sound sensor, the display control unit 318 displays an image displayed on the display unit 318 according to the acquired sound information. Display control for changing the aspect is performed. Here, changing the display mode means changing the content of the image itself, or changing the display timing, display time, display method, and the like of the image. For example, display control performed by the display control unit 318 to change the display mode includes control for changing the display image (content image) itself, control for changing the display timing, display time, and switching speed, and display method (fade, wipe, Control to change the slide display method (such as scrolling), control to change the contrast and color tone of the image and image effects (for example, retro tone), control to change the enlargement ratio and reduction ratio of the image, or change the number of screen divisions Control can be assumed. Alternatively, the display mode of the image is changed according to the speed, tone, power, etc. of the utterance (conversation), the speaker is recognized from the utterance, and the image is displayed in a display mode according to the user who is recognized by the speaker. May be. Or if display control is performed according to the keywords extracted by speech recognition of speech, or if emotions such as joy, anger, tension, and normality are recognized from speech speech, display control according to that emotion may be performed. Good. Alternatively, zoom the screen according to the utterance (conversation), increase the slide switching time when a topic related to the content is detected, slow the display switching when the conversation speed is fast, display time May be lengthened.

  Further, when the utterance detection unit 306 detects the user's utterance based on the sound information acquired by the sound sensor, the display control unit 318 displays based on the detection result of the user's utterance (whether or not the utterance is detected). The display mode of the image displayed on unit 340 may be changed. For example, when the user's utterance is detected, detailed information and related information of the display content that is the content displayed when the utterance is detected is displayed. For example, detailed information on the theme set in the display content and information related to the theme are displayed.

  Further, when the visual recognition state determination unit 311 determines the user's visual recognition state, the display control unit 318 displays an image displayed on the display unit 340 based on the user's utterance detection result and the user's visual recognition state determination result. You may change an aspect. For example, the display mode of the display image is changed according to whether or not the user is speaking and whether or not the user is gazing at the display unit 340.

  The display control unit 318 may control the display timing of the content based on the detection result of the user's utterance. For example, when the speech detection unit (speech recognition unit) 306 detects the user's non-speech state, a process for switching to the next content (slide) is performed in the slide display of the content. Specifically, if it is determined that the user is speaking, the current content (current slide) continues to be displayed, and if the non-utterance period exceeds a predetermined time, the next content (next slide) is displayed. ).

  The display control unit 318 may change the display mode of the image displayed on the display unit 340 based on the voice recognition result when the voice recognition unit 307 performs voice recognition of the user's utterance. For example, when an utterance keyword, which is a keyword that can be extracted from the user's conversation, is extracted by voice recognition, the content selected based on the extracted utterance keyword is displayed. Specifically, the content having the theme of the utterance keyword and the content searched by the utterance keyword are displayed.

  Or the display control part 318 displays the content corresponding to a tag keyword, when the tag keyword which is the keyword provided linked | related with the display content, and an utterance keyword correspond. For example, the content that is the subject of the utterance keyword or the content that is searched by the tag keyword is displayed. On the other hand, if the tag keyword and the utterance keyword do not match, the content selected by the AND search of the tag keyword and the utterance keyword is displayed. For example, the tag keyword is stored in the content information storage unit 322 in association with the display content. When one or more utterance keywords are extracted from the user's utterance, it is checked whether or not there is an utterance keyword that matches the tag keyword in the extracted utterance keywords. When any utterance keyword matches the tag keyword, the content associated with the tag keyword is displayed. On the other hand, if no utterance keyword matches the tag keyword, the content is selected and displayed by AND search of the tag keyword and the utterance keyword. For example, it searches for and displays content with the theme of both tag keywords and utterance keywords. When a plurality of different utterance keywords are extracted from the user's utterance, content selected by AND search of the plurality of different utterance keywords may be displayed. For example, the content having the theme of a combination of these keywords is searched and displayed.

  In addition, when the utterance detection unit 306 detects the user's utterance, the display control unit 318 displays based on the result of speech recognition by the speech recognition unit 307 and the determination result of the user's visual state by the visual state determination unit 311. The display mode of the image displayed on unit 340 may be changed. For example, when the user's utterance is detected and it is determined that the user is gazing at the display unit 340, the content selected by the tag keyword associated with the content displayed when the utterance is detected is displayed. Specifically, when the user is gazing and the tag keyword matches the utterance keyword, the content corresponding to the tag keyword (content searched for by the tag keyword, content related to the tag keyword) is displayed. . On the other hand, when the user's utterance is detected but the user is not gazing at the display unit 340, the content corresponding to the utterance keyword is displayed. For example, the content searched by the utterance keyword or the content AND-searched by the utterance keyword and the tag keyword is displayed.

  Further, it is assumed that the content image is configured by an image including a plurality of display objects, and each tag keyword is associated with each display object included in the image. When the tag keyword associated with the display object matches the utterance keyword, the display control unit 318 enlarges the display object associated with the matched tag keyword, or displays detailed information or related information of the display object. To do. For example, the first to Nth tag keywords are associated with the first to Nth display objects (N is an integer of 2 or more), and the Kth (1 ≦ K ≦ N) tag keywords are used as the utterance keywords. If they match, the Kth display object is enlarged, or detailed information and related information of the Kth display object are displayed. On the other hand, if none of the tag keywords of the first to Nth display objects match the utterance keyword, the first to Nth display objects are enlarged and displayed sequentially (in order).

  In addition, the display control unit 318 may count the number of appearances of the extracted utterance keyword and display content corresponding to the utterance keyword whose appearance number exceeds a predetermined number. For example, a histogram of utterance keywords extracted by voice recognition is created, and utterance keywords whose appearance count (appearance frequency) exceeds a predetermined count are detected. Then, the content corresponding to the detected utterance keyword (content searched by the utterance keyword) is selected and displayed. When there are a plurality of utterance keywords whose appearance count exceeds the predetermined count, for example, content corresponding to the utterance keyword with a high appearance count may be displayed preferentially.

  Further, it is assumed that the voice recognition unit 307 recognizes the user's speaker from the user's uttered voice. Specifically, speaker recognition (user recognition) is performed based on the feature amount information of the user's voice registered in the user information storage unit 328 and the acquired voice information. Then, the display control unit 318 displays content corresponding to the user who is recognized as a speaker. For example, when a user's hobbies / preferences are registered as user registration information, content corresponding to the user registration information is selected and displayed. Alternatively, when the voice recognition unit 307 recognizes the emotional state of the user from the user's uttered voice, the display control unit 318 may display content corresponding to the recognized emotional state. For example, when the user is sad, content that makes the user feel at home is displayed, and when the user is in a stressed state, content that makes the user relax is displayed. Further, when a state where the user is excited or a laughing voice is detected, contents that further increase the user are displayed.

  Further, it is assumed that the positional relationship determination unit 312 determines the positional relationship between the user and the display unit 340. Specifically, it is assumed that the distance between the user and the display unit 340 is determined. Then, when the distance between the user and the display unit 340 is within a predetermined distance, the display control unit 318 changes the display mode of the display image according to the acquired sound information. For example, if the sound information is farther than a predetermined distance, it is assumed that the sound information is detected as noise, and display control based on the sound information is not executed. Based display control is executed. Note that the distance between the user and the display unit 340 may be a parameter equivalent to the distance as well as the distance itself.

  The display control unit 318 may perform control to turn on the display of the display unit 340 based on the acquired sound information. For example, when the detected sound is above a certain level, the display of the display unit 340 is turned on to display the content image. For example, when the silent state period elapses for a predetermined time or longer, the display of the display unit 340 is turned off. Moreover, based on the acquired sound information, a user's operation instruction may be specified, and display control according to the specified operation instruction may be performed. For example, user operation instructions such as “forward”, “return”, and “stop” in slide display of content are specified based on sound information, and slide display is controlled according to the specified operation instruction.

2. Display Control by Utterance Detection In the present embodiment, various display controls can be performed based on the detection result of the user's utterance. For example, FIG. 3A schematically shows an example of the detection result of the user's utterance. That is, as a result of the frequency analysis of the detected sound, when the detected sound frequency is in a predetermined frequency band and the amplitude is equal to or higher than a predetermined level, the detected sound is a sound, noise, etc. It is determined that the voice is not the noise but the user's voice. Then, as shown in FIG. 3B, the detailed information and related information of the content displayed at the time of detecting the utterance are displayed. That is, detailed information about the theme (tag keyword) of the display content and information associated with the theme are displayed. For example, if the displayed content is a picture of Hokkaido (when the theme of the content is Hokkaido), detailed information about Hokkaido (various information such as tourism information and special product information about Hokkaido) and related information (Hokkaido theme) Is displayed in advance).

  Further, when a visual recognition state such as a user's gaze state is determined, display control may be performed by reflecting the visual recognition state determination result in addition to the speech detection result. For example, in FIG. 4A, the user's utterance is detected by the sound sensor which is the sensor 350, and it is determined whether or not the user is gazing at the display unit 340. For example, when the user's speech and the user's gaze are detected within a predetermined time, the detailed information and the related information of the content displayed at that time are displayed. In this way, users who are interested in the display content or who are gazing at the display content can be notified of detailed information and related information about the display content. A photo frame can be realized.

  For example, detection of the user's gaze state can be realized using an image sensor that is the sensor 350. That is, as shown in FIG. 4B, the user's face area FAR is detected based on the imaging information from the imaging sensor. Next, a measurement area SAR corresponding to the detected rectangular face area FAR is set. This measurement area SAR is an area that includes the face area FAR and is larger in size than the face area FAR. This measurement area SAR can be set, for example, by oversizing the face area FAR. Then, the time during which the face area FAR exists in the measurement area SAR is measured, and it is determined whether the user is gazing at the display unit 340 based on the measured time. For example, when the face area FAR is located in the measurement area SAR for a certain time or more, it is determined that the user is gazing at the display unit 340.

  Next, a display control method based on utterance detection will be described with reference to the flowchart of FIG. First, the content is displayed (step S1). For example, content is selected and displayed randomly or based on user information (registered user personal information). Then, it is determined whether or not the user's utterance is detected by the sound sensor. If the utterance is detected, it is determined whether or not the user is gazing at the display unit 340 (digital photo frame, display content) (step). S2, S3). If the user is gazing, the detailed information of the currently displayed content or the content of the related information is selected and displayed (step S4).

  Next, gaze state detection processing will be described with reference to the flowchart of FIG. First, a face area (frame area) is detected by face detection using an imaging sensor (camera) (step S21). Next, a measurement area that includes the detected face area and is larger than the face area is set (step S22). That is, as shown in FIG. 4B, a measurement area oversized from the face area is set. Then, the time during which the face area exists in the measurement area is measured using a timer (step S23). That is, after setting the measurement area, timer measurement is started and the time during which the face area is located within the measurement area is measured. Then, it is determined whether or not a predetermined time or more has elapsed (step S24).

  The gaze state detection method is not limited to the method shown in FIG. For example, the gaze state may be detected by detecting the user's red eyes. Alternatively, the position of the pupil is detected from the brightness and darkness of the image area around the eyes of the user's face image captured by the two cameras (stereo cameras), and the user is detected from the detected center position of the pupil and the center position of the eyeball. May be detected to determine whether the user is in a gaze state.

  Further, the display control method based on the utterance detection is not limited to the method shown in FIG. For example, the display timing of the content may be controlled based on the detection result of the user's utterance. Specifically, when the user's non-speech state is detected, a process for switching to the next content is performed in the slide display of the content.

  For example, in FIG. 7A, the utterance is detected when the content CT1 (slide 1) is displayed. During the period in which the utterance is detected, the current content CT1 is displayed without switching to the next content CT2 (slide 2). Specifically, when the non-utterance period does not exceed a predetermined time, the display of the content CT1 is continued. On the other hand, when the non-utterance period exceeds a predetermined time, the content CT2 is switched to the next content CT2. Even during the display of the content CT2, the non-speech period is counted, and when the non-speech period does not exceed the predetermined time, the display of the content CT2 is continued. On the other hand, when the non-utterance period exceeds the predetermined time, the content is switched to the next content CT3.

  In this way, when the user is interested in the display content and is speaking, the display of the content is maintained, and when the user's speech is interrupted, the next content is switched. As a result, it is possible to realize slide display control of a digital photo frame that has never been achieved.

  FIG. 7B shows a flowchart of the display control method of FIG. First, the slide display content CTi is displayed (step S31). Then, the user's utterance is detected by the sound sensor, and it is determined whether or not the user's non-utterance period (period in which the utterance is not detected) exceeds a predetermined time (steps S32 and S33). If the predetermined time is exceeded, i is incremented by 1 and the next content is switched (step S34).

3. Display Control by Voice Recognition In this embodiment, various display controls can be performed based on the voice recognition result of the user's utterance. For example, the user's utterance keyword is extracted by voice recognition, and the content selected based on the extracted utterance keyword is displayed. That is, in FIG. 8, the user's utterance keywords “Hokkaido” and “E Zoo” are extracted by voice recognition. In this case, detailed information and related information of “E Zoo” in “Hokkaido” are displayed as the content selected based on the utterance keyword.

  Specifically, sound information is acquired by detecting ambient sounds with a sound sensor. Next, the acquired sound information is converted into text information by voice recognition, and a word is extracted from the obtained text information. Then, the extracted word and a registered keyword stored in the registered keyword storage unit 326 (keyword database) are collated to determine whether or not the extracted word is a registered keyword. If the extracted word is a registered keyword, the word is determined to be an utterance keyword. Thereby, the situation where meaningless information and unpleasant information are presented to the user can be prevented.

  Based on the obtained utterance keyword, a content corresponding to (related to) the utterance keyword is searched from the content information storage unit 322 of the digital photo frame or the server (home server, external server). Then, an image of the searched content is displayed.

  In this way, when a user who is interested in the displayed content utters an utterance keyword related to the content, the content corresponding to the utterance keyword is displayed. A type of digital photo frame that is not available.

  Here, the registration keyword includes a general registration keyword and an individual registration keyword. General registration keywords are, for example, commonly used nouns such as things, animals, plants, personal names, and place names. This general registration keyword is registered in the general dictionary database as dictionary registration information. Note that the genre may be individually selected by the user as necessary. On the other hand, personal registration keywords include, for example, names of family members, relatives, friends, user hobbies / preference information, user favorite places / stores, tags attached to personally owned images, and the like. This personal registration keyword is registered in the user dictionary database as user registration information. For example, when an utterance including a general registered keyword “Ii-na, Hokkaido” is recognized, information and images related to Hokkaido are displayed. On the other hand, when an utterance including an individual registration keyword “Aunt A is coming this time” is recognized, a personal photograph with a tag of Aunt is displayed.

  In the present embodiment, content corresponding to the tag keyword may be displayed when the tag keyword associated with the display content matches the utterance keyword.

  For example, in FIG. 9, “Hokkaido”, “seasonal ingredients”, “mochi”, “crab”, and “urchin” are associated as tag keywords for display content. Then, from the user's utterance, the utterance keyword “crab” is extracted by voice recognition, and the utterance keyword “crab” matches the tag keyword of the display content. In this case, it is determined that the user is interested in the information of “crab”, and detailed information and related information about “crab” are displayed as content corresponding to “crab”. In this way, it becomes possible to display detailed information and related information about the contents that the user is interested in among the display contents (themes) of the display contents, and an interface environment suitable for the user can be realized.

  In this embodiment, when the tag keyword associated with the display content and the utterance keyword do not match, the content selected by the AND search of the tag keyword and the utterance keyword may be displayed.

  For example, in FIG. 10, “crab” is associated as a tag keyword of the display content. Then, from the user's utterance, an utterance keyword (general registered keyword) “cooking” is extracted by voice recognition, and the tag keyword “crab” and the utterance keyword “cooking” do not match. In this case, it is assumed that the user is interested in “cooking” of “crab”, and the content selected by AND search of “crab” and “cooking” is displayed. Specifically, detailed information and related information about “crab dishes” are displayed. In this way, content that takes into account both the content of the display content and the content of the user's utterance can be displayed as the next content, and various content displays are possible.

  In the present embodiment, when a plurality of different utterance keywords are extracted from the user's utterance, the content selected by AND search of the plurality of different utterance keywords may be displayed.

  For example, in FIG. 11, a plurality of different utterance keywords “crab” and “cooking” are extracted from the utterances of the user who viewed the display content. Also in this case, it is assumed that the user is interested in “cooking” of “crab”, and the content selected by AND search of “crab” and “cooking” is displayed. Specifically, detailed information and related information about “crab dishes” are displayed. In this way, it becomes possible to select appropriate content from the utterance contents of the user who viewed the display content and display it to the user.

  FIG. 12 shows a flowchart of the first display control method of the present embodiment by voice recognition. First, contents are selected randomly or based on user information (personal information) (step S41). The selected content is displayed (step S42).

  Next, based on the sound information from the sound sensor, it is determined whether or not the user's speech has been detected (step S43). If an utterance is detected, it is determined whether an utterance keyword has been extracted by voice recognition (step S44). If no user utterance is detected or no utterance keyword is extracted, the process returns to step S41.

  If an utterance keyword is extracted in step S44, it is determined whether an utterance keyword matching the tag keyword of the display content has been extracted (step S45). When an utterance keyword that matches the tag keyword is extracted, content corresponding to the tag keyword is selected (step S46). That is, as described with reference to FIG. 9, when the tag keyword and the utterance keyword match, related information and detailed information of the tag keyword are displayed. On the other hand, if an utterance keyword that matches the tag keyword is not extracted, an AND search of the tag keyword and the utterance keyword is performed, and the content extracted by the AND search is selected (steps S47 and S48). That is, as described with reference to FIG. 10, when the tag keyword and the utterance keyword do not match, the content selected by the AND search of the tag keyword and the utterance keyword is displayed.

  As described above, in the first display control method of FIG. 12, when the same utterance keyword as the tag keyword is detected, detailed information or related information (hereinafter referred to as detailed / related information as appropriate) of the tag keyword is displayed. When an utterance keyword different from the tag keyword is detected, an AND search is performed and the content is displayed. As a result, various contents can be displayed.

  FIG. 13 shows a flowchart of the second display control method of the present embodiment by voice recognition. In the second display control method of FIG. 13, content display control is performed based on the determination result of the gaze state, which is the user's visual state, in addition to the result of the user's speech detection and voice recognition.

  First, content is selected randomly or based on user information (step S51). Then, the selected content is displayed (step S52). Next, it is determined whether or not the user's speech has been detected. If not detected, it is determined whether or not the user is gazing at the display unit (display content) (steps S53 and S54). If the user is gazing, content corresponding to one of the tag keywords of the display content is selected (step S55). For example, as shown in FIG. 9, when the tag keywords “Hokkaido”, “Seasonal ingredients”, “Samurai”, “Crab”, “Sea urchin” are associated with the display content, any of these tag keywords Display details and related information about

  If an utterance is detected in step S53, an utterance keyword is extracted by voice recognition (step S56). Then, it is determined whether or not the user is gazing at the display unit. When the user is not gazing, content corresponding to the extracted utterance keyword is selected (steps S57 and S58). For example, when the user's utterance keyword is a registered keyword (such as a general registered keyword) registered in the dictionary database, content related to the registered keyword is selected and displayed.

  If it is determined in step S57 that the user is gazing, it is determined whether or not an utterance keyword matching the tag keyword of the display content has been extracted (step S59). If an utterance keyword that matches the tag keyword is extracted, it is determined whether another registered keyword has been extracted as the utterance keyword (step S60). If another registered keyword is extracted, the content is selected by AND search of the tag keyword that matches the utterance keyword and the extracted registered keyword (step S61). On the other hand, if no other registered keyword is extracted, the content corresponding to the tag keyword that matches the utterance keyword is selected (step S62).

  For example, it is assumed that “crab” and “cooking” are extracted as utterance keywords, and the tag keyword of the display content is “crab”. In this case, the utterance keyword “crab” matches the tag keyword “crab”, and the other registered keyword “cooking” is extracted as the utterance keyword. Therefore, in this case, the content is selected by AND search of “crab” and “cooking”. On the other hand, when only “crab” is extracted as the utterance keyword and no other registered keyword is extracted as the utterance keyword, the detail / related information of the tag keyword “crab” is selected and displayed.

  If an utterance keyword matching the tag keyword is not extracted in step S59, it is determined whether another registered keyword has been extracted as an utterance keyword (step S63). If another registered keyword is extracted, the content is selected by AND search of any tag keyword of the display content and the extracted registered keyword, and if not extracted, any of the display content is selected. The content corresponding to the tag keyword is selected (steps S64 and S65).

  For example, it is assumed that the tag keywords of the display content are “crab”, “mochi”, “urchin”, and “cooking” is extracted as the utterance keyword. In this case, the utterance keyword and the tag keyword do not match, but another registered keyword “cooking” is extracted as the utterance keyword. Therefore, the content is selected by AND search of any one of “crab”, “mochi”, “urchin” and “cooking”. For example, the contents of “crab dish”, “crab dish”, and “urchin dish” are selected. On the other hand, it is assumed that the tag keywords of the display content are “crab”, “鮭”, “uni”, and no other registered keywords are extracted as utterance keywords. In this case, the content corresponding to any of “crab”, “crab”, and “urchin” is selected and displayed.

  As described above, in the second display control method of FIG. 13, when the user's utterance is detected, the display mode of the display image is changed based on the result of the speech recognition and the determination result of the user's visual state. Yes.

  For example, when the user's utterance is detected (step S53) but the user is not gazing (step S57), the content corresponding to the utterance keyword is displayed (step S58). That is, voice recognition of the user's utterance is performed, and the content is selected by the utterance keyword (registered keyword) extracted from the voice recognition regardless of the displayed content.

  On the other hand, when the user's speech is detected (step S53) and the user is gazing (step S57), the content selected by the tag keyword of the display content is displayed. Specifically, when the user is gazing and the tag keyword matches the utterance keyword, the content selected by the matched tag keyword is displayed. For example, when the tag keyword matches the utterance keyword and another registered keyword (general registered keyword) is extracted, the content is selected by performing an AND search of the tag keyword and the registered keyword (step S61). If the tag keyword matches the utterance keyword, but no other registered keyword is extracted, the content corresponding to the tag keyword is selected (step S62).

  If the tag keyword does not match the utterance keyword but other registered keywords are extracted, the content is selected by AND search of any tag keyword of the display content and the registered keyword (step S64). If the registered keyword is not extracted, the detail / related information of any tag keyword of the display content is selected (step S65).

  As described above, intelligent display control reflecting both the user's gaze and the user's utterance keyword can be realized, and various contents display using a digital photo frame can be realized.

  In the present embodiment, the number of appearances of the extracted utterance keyword may be counted, and the content corresponding to the utterance keyword whose appearance number exceeds a predetermined number may be displayed. FIG. 14 is a flowchart for performing such processing.

  First, it is determined whether or not the user's utterance has been detected. If detected, the utterance keyword is extracted by voice recognition (steps S91 and S92). Then, the number of appearances of the utterance keyword is counted (step S93). For example, a histogram of the number of appearances of each utterance keyword is created. Then, it is determined whether or not the number of appearances of the utterance keyword exceeds a predetermined number, and if it exceeds, the content corresponding to the utterance keyword is selected as display content (steps S94 and S95).

  In this way, it is possible to prevent a situation in which the content image frequently changes in response to the user's utterance, and to display a content image that is easy for the user to view.

  The number of appearances of the utterance keyword and the utterance keyword history may be accumulated and recorded, and control may be performed so as to increase the display frequency of the content corresponding to the frequently used utterance keyword.

  Alternatively, an external database server that holds the utterance keyword history of many other users is accessed, and other users who frequently use the same keywords as those frequently used by the user are identified, and the utterances of the identified other users Content may be selected based on the keyword history. By performing such collaborative filtering processing, it becomes possible to realize content selection based on utterance keywords in accordance with user preferences.

4). Display control of a plurality of display objects In the present embodiment, when the content image is configured by an image including a plurality of display objects, it is determined whether the tag keyword and the utterance keyword of each display object match, The display object may be enlarged and displayed.

  For example, in FIG. 15A, a plurality of display objects representing “crabs”, “crabs”, and “urchins” are displayed as content images. These display objects are set with tag keywords “crab”, “crab”, and “urchin”.

  When “crab” is detected as the user's utterance keyword, an enlarged display centering on the display object of “crab” is performed, and detailed / related information about “crab” is displayed. Similarly, when “鮭” is detected as the user ’s utterance keyword, the display object of “鮭” is enlarged or the details / related information about “鮭” is displayed, and “uni” is detected as the utterance keyword. Then, the display object of “urchin” is enlarged and detailed / related information about “urchin” is displayed.

  On the other hand, if an utterance keyword that matches the tag keywords “crab”, “鮭”, and “urchin” is not extracted, the display objects “crab”, “鮭”, and “urchin” are sequentially enlarged and displayed. For example, the display objects are enlarged and displayed in the order of “crab”, “鮭”, and “urchin”.

  In this way, the display object that the user is interested in can be enlarged and the details and related information can be displayed. For example, the contents of a picture book such as a living thing can be realized by a digital photo frame. .

  In addition, when the display object is displayed in an enlarged manner, when the user's utterance is detected, the details / related information of the display object that is displayed in an enlarged form may be displayed. In this case, for example, the details / related information of the display object that is displayed in an enlarged manner may be displayed in response to only the word “Sadaido” such as “Yes, this, this”.

  In addition, various modes can be assumed as the display mode of the content image composed of a plurality of display objects. For example, as shown in FIG. 15B, when a content image is divided into a plurality of divided screens such as “weather information”, “traffic information”, “stock price information”, “time information”, etc. The screen becomes a display object. When “weather” is detected as the user's utterance keyword, a display object that is a divided screen of “weather information” is displayed in an enlarged manner. Similarly, when “traffic”, “stock price”, and “time” are detected as user utterance keywords, display objects that are divided screens of “traffic information”, “stock price information”, and “time information” are enlarged and displayed, respectively. Is done.

  FIG. 16 shows a flowchart of a display control method for a plurality of display objects. First, content is selected randomly or based on user information, and the selected content is displayed (steps S71 and S72).

  Next, it is determined whether or not there are a plurality of content display objects. If there is a single display object instead of a plurality, it is determined whether or not the user's speech has been detected (steps S73 and S74). If an utterance is detected, the content of the details / related information of the display object is selected (step S75). On the other hand, if no utterance is detected, content is selected randomly or based on user information (step S71).

  If there are a plurality of display objects, it is determined whether or not the user's utterance has been detected. If no utterance has been detected, it is determined whether or not the user is gazing (steps S76 and S77). . When the user is gazing, control is performed to sequentially enlarge and display a plurality of display objects (step S78).

  If the user's utterance is detected in step S76, the utterance keyword is extracted by voice recognition (step S79). Then, it is determined whether or not the user is gazing. If not, the content corresponding to the utterance keyword is selected (steps S80 and S81). For example, when the user's utterance keyword is a registered keyword registered in the dictionary database, content related to the registered keyword is selected and displayed.

  If it is determined in step S80 that the user is gazing, it is determined whether or not an utterance keyword matching the tag keyword associated with each display object of the plurality of display objects has been extracted (step S82). If extracted, the enlarged display of the display object associated with the matched tag keyword and the details / related information about the display object are selected (step S83). On the other hand, if not extracted, control is performed to sequentially enlarge and display a plurality of display objects (step S78).

  As described above, in FIG. 16, when the tag keyword associated with the plurality of display objects of the content image matches the utterance keyword, the display object associated with the matched tag keyword is enlarged or displayed in detail. Information or related information is displayed (step S83). On the other hand, if the tag keyword does not match the utterance keyword, control is performed to sequentially enlarge and display a plurality of display objects (step S78). In this way, various content displays reflecting the user's utterances can be realized, and display control of a digital photo frame that has never been possible can be realized.

5). Next, various modifications of the display control method according to this embodiment will be described. For example, in this embodiment, the user's speaker recognition may be performed from the user's uttered voice, and the content corresponding to the user who has been speaker-recognized may be displayed.

  For example, in FIG. 17, a user ID, a password, a user's hobbies / preferences, and the like are input as user registration information. Specifically, it is input that the user's hobbies / preferences are concert appreciation. The input user registration information is stored in the user information storage unit 328 as user information. In addition to hobbies and preferences, user favorite information, screen mode settings, slide display methods, and the like may be input as user registration information.

  After inputting such user registration information, voice registration of the user is performed. Specifically, the user speaks his / her name toward a microphone, which is a sound sensor, and inputs voice to perform voice registration. Then, the feature amount information of the input user's voice is extracted, and the extracted feature amount information is registered in association with the user registration information (hobby / preference information etc.) input in FIG. In this way, the user registration process is realized.

  It is assumed that the user's speech is detected by the sound sensor and the user's speaker recognition is performed with the user registration information registered in the digital photo frame 300 as described above. In this case, the display content of the digital photo frame 300 is selected based on the utterance recognition result and the user registration information. Specifically, in FIG. 17, it is input as user registration information that the user's hobbies / preferences are concert viewing. Accordingly, when the user is recognized as a speaker based on the sound information from the sound sensor, content introducing concert information is selected from a plurality of candidate contents as shown in FIG. Is displayed. Further, for example, it is assumed that cooking is input as a user's hobbies / preferences on the user registration screen of FIG. In this case, when the user is recognized as a speaker, as shown in FIG. 18B, content introducing recipe information of the dish is selected and displayed.

  By doing so, content reflecting the user's hobbies, preferences, etc. is automatically selected and displayed in the digital photo frame, so that convenience for the user can be improved.

  For example, as a technique for the user to acquire such concert information, a technique is conceivable in which concert information of an artist viewed by the user or recipe information of a dish is searched using a PC (personal computer) or the like. However, with this method, the user must perform various operations, which is troublesome for the user.

  In this regard, according to the method shown in FIGS. 17 to 18B, by inputting the user registration information in advance, the content of candidate content corresponding to music information and program information can be obtained without the user being conscious. Therefore, content according to the user's hobbies / preferences is automatically selected and displayed on the digital photo frame 300. Therefore, it is possible to provide the digital photo frame 300 that can improve the convenience for the user and display the content reflecting the user's hobbies and preferences.

  In this embodiment, the user's emotional state may be recognized from the user's uttered voice, and the content corresponding to the recognized emotional state may be displayed. For example, when a laughing voice is detected from the user's utterance (conversation) and it is determined that the emotional state of the user is delighted, an effect display that excites as an effect on the display unit 340 of the digital photo frame 300 I do. On the other hand, when it is determined that the emotional state of the user is in a tension state, content that relaxes the user is selected and displayed.

  Various methods are conceivable as a method for recognizing the emotional state (sensitivity state) of the user. For example, the emotional state of the user is classified into a plurality of states such as a normal state, an excited state, a relaxed state, a joyful state, and a sad state. Then, based on the recognition result of the user's voice, it is determined which of these states the user's emotional state is. Also, the level of the emotional state in each state is determined. Specifically, it is determined whether the state is “very happy” or “a little happy”. For example, the emotion state is quantitatively detected by analyzing the feature amount information of the uttered voice of the user by the rhythm and tone of the voice, and the above-described various determinations are realized.

  Further, in the present embodiment, when the positional relationship with the user is determined and the distance between the user and the display unit 340 is within a predetermined distance, that is, when the distance is close, the display mode of the display image is determined according to the acquired sound information. May be changed.

  For example, as shown in FIG. 19A, when the distance from the user is far, that is, when the distance is greater than or equal to a predetermined distance, display control based on the sound information is not performed, and as shown in FIG. Display control based on the sound information is performed. In this way, the display control based on the sound information can be executed after the user approaches the digital photo frame 300 and the user's voice can be reliably detected. Thus, it is possible to prevent a situation in which the display mode of the display image frequently changes due to noise or the like, and it is possible to display a high-quality image that is easy for the user to see.

  Here, various methods are conceivable as a method for detecting the positional relationship with the user. For example, in FIG. 20A, an imaging sensor (camera) such as a CCD or CMOS sensor is used as the sensor 350. And based on the imaging information from an imaging sensor, the user's face area | region FAR which is a rectangular frame area | region is detected. Further, image recognition processing is performed on the user image shown in the face area FAR, and feature point data of the user face image is extracted. This feature point data is registered in association with user registration information, for example, as a facial image recognition result.

  When detecting the positional relationship between the user and the display unit 340, the size of the face area FAR is obtained based on imaging information from the imaging sensor. Based on the obtained size, the distance between the user and the display unit 340 is determined.

  For example, in FIG. 20B, since the size of the face area FAR is small (below a predetermined size), it is determined that the distance from the user is long. In this case, for example, as shown in FIG. 19A, display control based on sound information is not performed.

  On the other hand, in FIG. 20C, since the size of the face area FAR is large (because it is larger than the predetermined size), it is determined that the distance to the user is short. When it is determined that the user is close to the digital photo frame 300 in this manner, display control based on sound information is performed as shown in FIG. 19B. That is, the display mode of the display image is changed based on the sound information acquired by the sound sensor.

  Here, various methods can be considered as a method for detecting a face region. For example, in order to perform face detection, it is necessary to cut out a face region by distinguishing a place where the face is from another object in a captured image captured by an image sensor. The face is composed of eyes, nose, mouth and the like, and these shapes and positional relationships have almost common characteristics, although there are individual differences. Therefore, using this common feature, the face is identified from other objects and cut out from the screen. Clues for this include skin color, facial movement, shape, size, and the like. In the case of using skin color, a method of converting RGB data into HSV data composed of hue, luminance, and saturation and extracting a human skin hue is adopted.

  Or you may produce the average face pattern produced | generated from the face pattern of many people as a face template. Then, this face template is scanned on the screen of the captured image to obtain a correlation with the captured image, and an area having the highest correlation value is detected as a face area.

  In order to improve the detection accuracy, a plurality of face templates may be prepared as dictionary data, and a face area may be detected using the plurality of face templates. Alternatively, the face area may be detected in consideration of the characteristics of the eyes, nose, mouth, the positional relationship between these, and the contrast in the face. Alternatively, the face area can be detected by statistical pattern recognition using a neural network model.

  20A to 20C, not only can the distance between the user and the display unit 340 be detected based on the size of the face area FAR, but also whether or not the user is looking at the display unit 340 at the same time. There is an advantage that it can be detected. That is, when the user's line of sight is not directed toward the display unit 340, the correlation value with the face template is low, and the face area FAR is not detected. Therefore, the detection of the face area FAR is equivalent to the fact that the user's line of sight is directed toward the display unit 340 and the display unit 340 is within the visual field range of the user. In this state, if the size of the face area FAR is detected, display control is performed based on the sound information, and the content is displayed, the content image can be appropriately displayed to the user viewing the display unit 340. become. Thereby, a digital photo frame 300 of an unprecedented type can be provided.

  In the present embodiment, display control may be performed by turning on the display of the display unit 340 of the digital photo frame 300 or specifying a user operation instruction based on the acquired sound information. For example, when a living sound such as conversation, music, or TV sound is detected, the display is turned on. When an utterance keyword related to the operation instruction is detected, display control is performed according to the utterance keyword. For example, when the “forward” utterance keyword is detected during slide display, the next content is displayed, and when the “return” utterance keyword is detected, the previous content is returned and displayed.

  FIG. 21 is a flowchart for performing such display control. First, it is determined whether or not sound information is detected by the sound sensor. If detected, the display of the display unit 340 of the digital photo frame 300 is turned on (steps S101 and S102). Then, it is determined whether or not an operation instruction is specified by the sound information (step S103). Specifically, speech recognition of the user's utterance is performed, and it is determined whether or not there is an utterance keyword that matches the keyword corresponding to the operation instruction among the extracted utterance keywords. When an operation instruction by the user is specified, display control is performed according to the specified operation instruction (step S104).

6). Modification of System Configuration FIG. 22 shows a modification of the system configuration of the present embodiment in FIG. In the system of this modification, a server 200 (an information processing system in a broad sense and a home server in a narrow sense) is provided. The server 200 includes a processing unit 202, a storage unit 220, and a communication unit 238. Various modifications may be made such as omitting some of these components or adding other components. In addition, about the component similar to FIG. 2, the same code | symbol is attached | subjected and the description is abbreviate | omitted.

  The processing unit 202 performs various processes such as a server management process, and can be realized by a processor such as a CPU or an ASIC. The storage unit 220 is a work area for the processing unit 202 and the communication unit 238, and can be realized by, for example, a RAM or an HDD. The communication unit 238 is for performing wired or wireless communication with the digital photo frame 300 or the external server 600, and can be realized by a communication ASIC or a communication processor. For example, the digital photo frame 300 and the server 200 are communicatively connected via a network such as a wireless LAN.

  The processing unit 202 includes a registration processing unit 214, a content selection unit 216, and a display instruction unit 218. Various modifications may be made such as omitting some of these components or adding other components.

  The registration processing unit 214 performs the user registration processing described with reference to FIG. The user registration information is stored and registered in the user information storage unit 228 of the storage unit 220.

  The content selection unit 216 performs processing for selecting content to be presented to the user. For example, content information is read from the content information storage unit 222 of the storage unit 220, or the content to be presented to the user is selected by accessing the external server 600 and receiving the content information.

  The display instruction unit 218 issues an instruction to display an image displayed on the display unit 340 of the digital photo frame 300 based on the content selected by the content selection unit 216. Specifically, an instruction for displaying an image of the content selected by the content selection unit 216 on the display unit 340 is given. For example, a display instruction for changing the display mode of an image displayed on the display unit 340 is performed in accordance with sound information detected by a sound sensor that detects ambient sounds. Then, the display control unit 318 of the digital photo frame 300 performs display control of the display unit 340 in accordance with an instruction from the display instruction unit 218 of the server 200. Thereby, the display mode of the image displayed on the display part 340 changes according to the acquired sound information.

  According to the modification of FIG. 22, since the server 200 performs content selection processing, user registration processing, and the like, the processing load on the digital photo frame 300 can be reduced. Therefore, even when the processing capability of the processing unit 302 (CPU) of the digital photo frame 300 is low, the processing of this embodiment can be realized. Note that these processes may be realized by distributed processing of the home server 200 and the digital photo frame 300.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described together with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term anywhere in the specification or the drawings. In addition, the configuration and operation of the digital photo frame and the information processing system, the display control method, the voice recognition method, the visual recognition state determination method, and the like are not limited to those described in this embodiment, and various modifications can be made. .

200 servers, 202 processing units, 214 registration processing units, 216 content selection units,
218 display instruction unit, 220 storage unit, 222 content information storage unit,
228 User information storage unit, 238 communication unit,
300 digital photo frame, 302 processing unit, 304 sound information acquisition unit,
305 detection information acquisition unit, 306 utterance detection unit, 307 voice recognition unit,
310 user state determination unit, 311 visual recognition state determination unit, 312 positional relationship determination unit,
314 registration processing unit, 316 content selection unit, 318 display control unit,
320 storage unit, 322 content information storage unit, 324 sound information storage unit,
325 Detection information storage unit, 326 Registration keyword storage unit, 327 User status storage unit, 328 User information storage unit, 330 Information storage medium, 338 Communication unit, 340 Display unit,
350 sensor, 360 operation unit, 600 external server

Claims (27)

A display for displaying an image;
A display control unit that performs display control of the display unit;
A sound information acquisition unit that acquires sound information detected by a sound sensor that detects ambient sound,
The display control unit
A digital photo frame that performs display control to change a display mode of an image displayed on the display unit in accordance with the acquired sound information. In claim 1,
Based on the acquired sound information, including an utterance detection unit for detecting the user's utterance,
The display control unit
A digital photo frame that performs display control for changing a display mode of an image displayed on the display unit based on a detection result of a user's utterance. In claim 2,
The display control unit
A digital photo frame that performs control to display detailed information or related information of display content when a user's utterance is detected. In claim 2 or 3,
Including a visual recognition state determination unit for determining the visual state of the user,
The display control unit
A digital photo frame that performs display control for changing a display mode of an image displayed on the display unit based on a detection result of a user's utterance and a determination result of a user's visual state. In any of claims 2 to 4,
The display control unit
A digital photo frame characterized in that the display timing of content is controlled based on a detection result of a user's utterance. In claim 5,
The speech detection unit
Detects the user's utterance state,
The display control unit
A digital photo frame characterized in that when a user's non-speaking state is detected, a process for switching to the next content is performed in the slide display of the content. In claim 1,
Based on the acquired sound information, including a voice recognition unit that performs voice recognition of the user's utterance,
The display control unit
A digital photo frame characterized by performing display control for changing a display mode of an image displayed on the display unit based on a voice recognition result. In claim 7,
The voice recognition unit
Extract user's utterance keywords by voice recognition,
The display control unit
A digital photo frame characterized by performing control to display content selected based on the extracted utterance keyword. In claim 8,
The display control unit
A digital photo frame, wherein when a tag keyword associated with display content matches the utterance keyword, control is performed to display content corresponding to the tag keyword. In claim 8,
The display control unit
When the tag keyword associated with the display content and the utterance keyword do not match, control is performed to display the content selected by AND search of the tag keyword and the utterance keyword. flame. In claim 8,
The display control unit
When the tag keyword and the utterance keyword match, the content corresponding to the tag keyword is displayed. When the tag keyword and the utterance keyword do not match, the tag keyword and the utterance keyword are displayed. A digital photo frame characterized by performing control to display content selected by AND search. In claim 8,
The display control unit
A digital photo frame, wherein when a plurality of different utterance keywords are extracted from a user's utterance, control is performed to display content selected by AND search of the plurality of different utterance keywords. In claim 8,
Including a visual recognition state determination unit for determining the visual state of the user,
The display control unit
A digital control device that performs display control to change a display mode of an image displayed on the display unit based on a result of voice recognition and a determination result of a user's visual state when a user's utterance is detected. Photo frame. In claim 13,
The display control unit
When the user's utterance is detected and it is determined that the user is gazing at the display unit, control is performed to display the content selected by the tag keyword associated with the display content. Photo frame. In claim 14,
The display control unit
When it is determined that the user is gazing at the display unit and the tag keyword matches the utterance keyword, control is performed to display the content selected by the tag keyword. flame. In claim 14 or 15,
The display control unit
A digital photo frame characterized by performing control to display content corresponding to the utterance keyword when it is determined that the user's utterance is detected but the user is not gazing at the display unit. In any of claims 8 to 16,
The display control unit
A digital photo frame that counts the number of appearances of an extracted utterance keyword and performs control to display content corresponding to the utterance keyword whose appearance number exceeds a predetermined number. In claim 8,
The display control unit
When tag keywords associated with a plurality of display objects constituting a content image match the utterance keyword, control for enlarging the display object associated with the matched tag keyword, or detailed information on the display object or A digital photo frame characterized by performing control to display related information. In claim 18,
The display control unit
A digital photo frame, wherein when the tag keyword does not match the utterance keyword, control is performed to sequentially enlarge and display the plurality of display objects. In claim 7,
The voice recognition unit
Recognize the user's speaker from the user's speech,
The display control unit
A digital photo frame characterized by performing control to display content corresponding to a user who is recognized as a speaker. In claim 7,
The voice recognition unit
Recognize the user's emotional state from the user's speech,
The display control unit
A digital photo frame characterized by performing control to display content corresponding to the recognized emotional state. Any one of claims 1 to 21
A positional relationship determination unit that determines a positional relationship between the user and the display unit;
The display control unit
When the distance between the user and the display unit is within a predetermined distance, display control is performed to change a display mode of an image displayed on the display unit according to the acquired sound information. digital photo frame. In any one of Claims 1 thru | or 22.
The display control unit
A digital photo frame characterized by performing control to turn on display of the display unit based on the acquired sound information. In any one of Claims 1 thru | or 23.
The display control unit
A digital photo frame characterized by specifying a user operation instruction based on the acquired sound information and performing display control according to the specified operation instruction. A content selection unit for selecting content;
A display instruction unit for instructing display of an image displayed on the display unit of the digital photo frame based on the selected content,
The display instruction unit
An information processing system that performs a display instruction to change a display mode of an image displayed on the display unit in accordance with sound information detected by a sound sensor that detects ambient sound. A display control unit that performs display control of a display unit that displays an image;
As a sound information acquisition unit that acquires sound information detected by a sound sensor that detects ambient sound,
Make the computer work,
The display control unit
A program for performing display control for changing a display mode of an image displayed on the display unit in accordance with the acquired sound information.   A computer-readable information storage medium, wherein the program according to claim 26 is stored.

Images