JP2014017791A - Information communication program, information communication device and distribution server - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable instant identification of a part of contents displayed on a paper sheet by a simple operation.SOLUTION: An information communication device comprises: a transmission/reception part 10; a feature data storage part 20 for storing feature data obtained by extracting an apparent feature amount of contents displayed on a paper sheet and identification information of the contents; an imaging part 11 for imaging the contents displayed on the paper sheet; a display part 13 for displaying an imaged live view image; a feature extraction part 22a for extracting feature data from the live view image; a matching part 23 for identifying the contents displayed on the paper sheet by matching with the feature data in the feature data storage part 20; an identification display data request part 24a for causing the transmission/reception part 10 to transmit a request for identification display data based on the identification information of the identified contents, the identification display data for performing the identification display for identifying a part of the contents; and an identification display control part 28b for causing the display part 13 to display, together with the live view image, the identification display data received by the transmission/reception part 10.

Description

  The present invention relates to an information communication program, an information communication apparatus, and a distribution server for acquiring information related to contents posted on a paper such as a broadcast program guide.

  Currently, it is common practice to display an electronic program guide on a TV screen and browse a program column or make a program recording reservation by remote control operation. However, since the display area of the television is limited, there are cases where it is not possible to list programs for all channels. In addition, when the period from the date and time of the displayed program guide to the broadcast date and time of the desired program is long, it is necessary to scroll the screen for a long time. Furthermore, although there is a method of searching by keyword, it takes time and effort to input a keyword and operate a search button.

  On the other hand, a mobile phone such as a smartphone can also be used to browse the electronic program guide and reserve recording of programs. However, in this case, since the screen of the mobile phone is very small, the range that can be displayed on one screen is limited. For this reason, it becomes more difficult to find a desired program. Further, it takes time to search for a keyword as in the case of an electronic program guide on a television.

Japanese Patent Laid-Open No. 2005-110023

  By the way, for browsing the program guide, it is possible to use program guides printed on paper such as newspapers, magazines, and cable television monthly magazines. In particular, in the case of magazines specializing in television broadcasting and monthly magazines on cable television, one day's programs can be listed on two pages. Moreover, if the page of a desired day is opened, the program schedule of the day can be viewed immediately. In other words, the program guide on paper has the advantage that it is easier to browse the entire program guide and access the desired program than the electronic program guide.

  However, a user's desired program cannot be automatically picked up by a paper medium. In addition, when making a recording reservation, the TV remote control must be operated again. In particular, regarding recording reservation, there has been a method of making a reservation by reading a G code or a bar code printed on a paper surface by a remote controller or the like. However, since the area of each program column on the page is very small, the amount of information that can be displayed in the program column is restricted when a G code or bar code is printed in each program column.

Here, a technique using information other than the G code and the bar code is disclosed in Patent Document 1 (paragraphs 0040 to 0045). This technique is as follows.
(1) An image in the program column is acquired by an imaging unit of an information processing apparatus such as a mobile phone.
(2) The image area is specified by the user's key operation.
(3) The information analysis unit performs binarization processing on the image information of the specified area, and identifies the outline of the character image.

(4) The title image is identified as character information by comparing the character image with a database for character identification.
(5) The identified title information and the title information in the EIT of the stored electronic program information are compared one by one.
(6) If title information having the same or a similarity of a certain value or more can be detected from the EIT of the electronic program information, the program ID is displayed.
(7) Program reception and recording reservation are performed based on the program ID.

  In this method, after shooting the program column, the procedure of specifying the image area by the user, binarization processing, character recognition, comparison processing with title information, display of the program ID, reception of the program based on the program ID, and recording reservation are performed. Necessary.

  In this case, the user needs a very troublesome work of specifying a necessary image area while viewing the captured image. Further, in program reception and recording reservation based on the displayed program ID, it cannot be intuitively determined whether or not a desired program is actually selected. Furthermore, since the comparison processing is performed after the character information is once converted, the processing load on the information analysis unit is large and the processing time is also long.

  The present invention solves the problems of the prior art as described above, and an object of the present invention is to provide an information communication program capable of immediately identifying a part of the content displayed on a sheet by a simple operation, It is to provide an information communication device and a distribution server.

  In order to achieve the above object, an information communication program according to the present invention extracts feature data obtained by extracting a feature amount on the appearance of content displayed on a paper surface to a computer connected to a transmission / reception unit, an imaging unit, and a display unit, and its data A feature data storage process for storing content identification information; an imaging process for causing the imaging unit to capture the content displayed on the paper; and a display control process for displaying a live view image captured by the imaging unit on the display unit; , Feature extraction processing for extracting feature data on the appearance of the content displayed on the paper surface from the live view image, feature data stored by the feature data storage processing, and features extracted by the feature extraction processing A matching process for specifying the content displayed on the captured paper by matching with the data, and the internal process specified by the matching process. Based on the identification information, the identification display data request process for transmitting the identification display data request for identification display for identifying a part of the content to the transmission / reception unit, and the request by the identification display data request process An identification display control process for displaying the identification display data received by the transmission / reception unit on the display unit together with the live view image is executed.

  The present invention can also be understood as an information communication apparatus that realizes the functions of the above-described processes.

  According to another aspect, an information request process for causing the computer to transmit a request for information regarding the part of the content that is part of the content and the identification display is made to the transmission / reception unit, and according to a request by the information request process And an information display process for displaying the information received by the transmission / reception unit on the display unit.

  Another aspect is characterized by causing the computer to execute an operation request process for causing the transmission / reception unit to transmit an operation request to an external device for the portion on which the identification display is made.

  In another aspect, the page is a program table of a broadcast program, and the identification display is a display for identifying each program in the program table included in the live view image.

  Another aspect is characterized in that the computer is caused to execute a condition setting process for setting a condition for identifying and displaying which program column in the program table included in the live view image.

  Another aspect is characterized in that the computer is caused to execute a user interface control process in which the area where the identification display is made is used as a user interface for inputting information.

  In another aspect, the computer is caused to execute a mode display control process for performing a mode display including a user interface for performing the information request process.

  In another aspect, the computer is caused to execute a mode display control process for performing a mode display including a user interface for performing the operation request process.

  In another aspect, the identification display data includes a ratio of a part of the content to the entire size of a captured image or a position coordinate of the part of the content calculated from the ratio. To do.

  According to the present invention as described above, it is possible to provide an information communication program, an information communication apparatus, and a distribution server that can immediately identify a part of the content displayed on a sheet with a simple operation. .

Overall configuration diagram showing an information communication system in an embodiment The block diagram which shows the information communication apparatus in embodiment The block diagram which shows the delivery server in embodiment Sequence diagram showing processing procedure of information communication system Flowchart showing processing procedure of search mode of information communication apparatus The flowchart which shows the processing procedure of the apparatus mode of an information communication apparatus Flow chart showing processing procedure of net mode of information communication apparatus The figure which shows the example of a screen of the live view image and identification display of an information communication apparatus The figure which shows the example of a screen which displayed the update information button in the example of FIG. The figure which shows the example of a screen which selected the genre button in search mode The figure which shows the example of a screen which selected the genre button in search mode The figure which shows the example screen of keyword search in search mode The figure which shows the example of a screen which displayed the information icon in search mode The figure which shows the example of a screen which selected the information icon in search mode The figure which shows the example of a screen which displayed the program information in search mode Figure showing a screen example of the device mode The figure which shows the example of a recording reservation screen in apparatus mode The figure which shows the example of a screen of recording reservation completed in apparatus mode The figure which shows the example of the recorded picture in device mode Diagram showing an example of recording / tuning screen in device mode The figure which shows the example of a program selection in apparatus mode Diagram showing an example of device playback / transfer screen in device mode The figure which shows the example of the screen in net mode The figure which shows the display example of the popularity information in net mode The figure which shows the display example of the review in net mode

  A mode for carrying out the present invention (hereinafter referred to as “the present embodiment”) will be specifically described with reference to the drawings. In addition, this embodiment is realizable by controlling information communication apparatuses, such as a smart phone and a tablet terminal as a computer, with a program. Various implementations of the hardware and program in this case can be changed.

  The present invention can also be understood as a program as described above and a computer-readable storage medium storing such a program. In the following description, a virtual block diagram illustrating each function of the present embodiment in blocks is used.

[1. Constitution]
[1-1. Overview]
First, as shown in FIG. 1, the present embodiment is configured by an information communication device P capable of transmitting / receiving information to / from a distribution server S and a recording / playback device V via a network N. The user uses the information communication device P to take an image of the sheet M on which a program guide such as a magazine or newspaper is printed.

  The information communication device P receives the identification display data from the distribution server S via the network N, and displays the identification display for identifying the program column that meets the preset condition, superimposed on the captured program column.

  The information communication device P operates the recording / reproducing device V by transmitting a command such as a recording reservation to the recording / reproducing device V for the program whose identification is displayed by the user's selection.

[1-2. Information communication device]
As illustrated in FIG. 2, the information communication device P includes a transmission / reception unit 10, an imaging unit 11, an audio output unit 12, a display unit 13, and an input unit 14 provided in a mobile phone such as a general smartphone. The transmission / reception unit 10 is a processing unit that transmits / receives information to / from the distribution server S via the network N. For example, the transmission / reception unit 10 has a function of connecting to a wireless LAN such as WiFi and the Internet. The network N may be a local network in the home.

  The imaging unit 11 is a processing unit that captures an image of the sheet M on which the program guide is printed. For example, the imaging unit 11 is a camera having an optical unit 11a such as a lens and an image sensor unit 11b which is a light receiving element such as a CMOS or a CCD. This embodiment uses a live view image captured by the imaging unit 11 as described below.

  The live view image is image data for displaying an electric signal obtained by converting light received by the image sensor unit 11b in real time on the display unit 13 described later. This image data also includes image data temporarily stored in a volatile memory such as a DRAM and subjected to image processing by an image processor.

  The live view image is sequentially overwritten and disappears by processing such as terminating the program or turning off the power. That is, the live view image does not include image data stored in the non-volatile memory due to a process of releasing the shutter of the camera or the like.

  The sound output unit 12 is an output device such as a headphone, an earphone, or a speaker that outputs sound that the user listens to. The audio output unit 12 also includes a transmission device that transmits audio data to headphones, earphones, speakers, and the like wirelessly. The display unit 13 is an output device such as a display that displays an image visually recognized by the user.

  The input unit 14 is an input device through which a user inputs information necessary for processing and operation of the present embodiment. The input unit 14 includes a touch panel configured in the display unit 13 and a switch configured in a housing of the information communication device P. The user can perform an input operation using a smartphone such as a tap or a swipe by using the touch panel configured on the display unit 13. Furthermore, a voice input device including a microphone and a voice recognition unit is also included in the input unit 14. That is, this embodiment can be operated by voice.

  The information communication apparatus P includes a feature data storage unit 20, an analysis unit 22, a matching unit 23, a request unit 24, a condition setting unit 25, a mode selection unit 26, a reproduction control unit 27, and a display control unit 28.

  The feature data storage unit 20 is a processing unit that stores feature data and identification information. This feature data is data obtained by extracting in advance the feature amount on the appearance of the content displayed on the paper surface M. In the present embodiment, the content displayed on the page M is a program guide. The feature amount includes information on the shape, size, and position of characters, figures, colors, and the like that identify the entire program table and each program column constituting the program table. The figure includes a line and a frame. The characters are not text-converted, but the appearance features formed by the characters and their arrangement are the feature quantities. The identification information is information associated with the feature data in order to identify the feature data. For example, the broadcast date of each program guide is used as identification information.

  The feature data is stored in advance by the distribution server S described later in association with the identification information. And the transmission / reception part 10 downloads characteristic data and identification information from the delivery server S regularly. The transmission / reception unit 10 may download the feature data in accordance with the activation of the device or program by the user, the input from the input unit 14, or the like.

  The analysis unit 22 is a processing unit that analyzes the live view image captured by the imaging unit 11. The analysis unit 22 includes a feature extraction unit 22a. The feature extraction unit 22a is a processing unit that extracts feature data of the paper surface M from the live view image. A known technique can be applied to a method such as boundary extraction for each program column. Although not shown, the analysis unit 22 includes a processing unit that performs processing for detecting a positional relationship such as a distance and a direction between the imaging unit 11 and the paper surface M in a three-dimensional space. For example, the positional relationship such as the distance and the direction can be obtained by comparing the outer shape and size of the characteristic portion on the actual paper surface M with the outer shape and size of the characteristic portion in the captured image.

  The matching unit 23 matches the feature data stored in the feature data storage unit 20 with the feature data extracted by the feature extraction unit 22a, so that the date of the program guide on the captured paper M and each program column are displayed. A processing unit to be identified.

  The feature data extracted by the feature extraction unit 22a only needs to be able to specify the program table and the program column by collating with the feature data stored in the feature data storage unit 20. For this reason, it may be all or a part of the feature data for the program guide and the program column. Therefore, it is not always necessary that the entire program guide or program column is contained in the live view image. Further, even if there is a distortion of the page M or a difference in direction, the program column can be specified by collating the position including the position in the entire program table.

  The request unit 24 is a processing unit that outputs various requests to the distribution server S, the recording / playback device V, and the like. Information requested by the request unit 24 includes identification display data, program information, device operation, and shared information. The processing units that request these information constitute an identification display data request unit 24a, a program information request unit 24b, a device operation request unit 24c, and a shared information request unit 24d, respectively.

  The identification display data request unit 24a is a processing unit that requests the distribution server S for identification display data for identifying and displaying the program column on the display unit 13. This request includes the date of the program guide specified by the matching unit 23 and a parameter indicating which mode will be described later.

  The identification display data includes the ratio of each program column in the program table relative to the overall size of the captured image, the position coordinates of each program column calculated from the ratio, the figure and characters indicating the program column according to the mode, Including color and size. Note that when the overall size of the captured image is the size of the entire captured paper surface M, the overall size of the paper surface based on the image in the captured screen area, its feature data, and the positional relationship, This includes the case of obtaining the positional relationship. Icon IDs assigned to various icons set in the application program are also included in the identification display data.

  The program information request unit 24b is a processing unit that requests the distribution server S for program information that introduces and advertises information related to the title of the identified program or the selected program, performers, and other programs. is there. The program information includes text, images, and moving images that introduce the program. In addition, the program information can include magazines including the page M and introduction pages of programs in newspapers.

  The device operation request unit 24c is a processing unit that transmits an operation request to the recording / playback device V. This operation request includes commands for requesting recording reservation, recording, channel selection, reproduction, and transfer of the selected program. The command follows the setting of the recording / playback device V.

  The shared information request unit 24d is a processing unit that transmits a request for shared information, which is a summary of popularity, reviews, recommendation information, and the like, to the distribution server S for the selected program. As shared information, it is possible to use statistical data such as reserved recording and channel selection collected by the distribution server S and data obtained by voting performed on the system. In the case of a movie or the like, data obtained from a review result on a mail order or rental DVD site can also be used. Note that the processing by the program information request unit 24b and the shared information request unit 24d is included in the information request processing in the claims.

  The condition setting unit 25 is a processing unit that sets a condition regarding which program column is identified and displayed. The condition includes a keyword input by the user using the input unit 14 in advance. In addition, genres such as “drama”, “news”, and “new program” set in advance are also included in the conditions.

  The mode selection unit 26 is a processing unit that selects a mode to be displayed on the display unit 13. The mode includes, for example, a search mode, a device mode, and a net mode. The search mode is a mode in which program information is displayed while being identified and displayed in the program column based on the conditions set in the condition setting unit 25 (see FIGS. 8 to 12).

  The device mode is a mode in which operation requests such as recording reservation, recording, channel selection, reproduction, and transfer are input for the identified and displayed program (see FIGS. 16 to 20). The net mode is a mode in which shared information is acquired via the network for the identified and displayed program (see FIGS. 21 and 22).

  For example, by default, the mode selection unit 26 selects the search mode. And according to the input from the input part 14, the mode selection part 26 switches to apparatus mode and net mode. An input from the input unit 14 for selecting a mode includes selection of a tab for displaying each mode (see FIG. 10).

  The reproduction control unit 27 is a processing unit that outputs audio to the audio output unit 12 and displays the moving image on the display unit 13 via the display control unit 28 based on the received audio data and moving image data.

  The display control unit 28 is a processing unit that controls information to be displayed on the display unit 13. This information includes a live view image, identification display, mode display, program information, shared information, and the like. The processing units that control the display of these information constitute a live view image control unit 28a, an identification display control unit 28b, a mode display control unit 28c, a program information control unit 28d, and a shared information control unit 28e, respectively.

  The live view image control unit 28 a controls the display of the live view image based on the information input from the imaging unit 11. When the position and direction of the imaging unit 11 with respect to the subject change, the live view image also changes.

  The identification display control unit 28b controls the identification display of the program column based on the above identification display data. For example, the display position of the identification display can be obtained from the ratio of the program column to the entire size (vertical and horizontal) of the imaged sheet M or the position coordinates of each program column calculated from the ratio. The identification display is a display for identifying a program column in the program table specified by the matching unit 23. Examples of the identification display include a display in which a frame overlapped with a box of the program column is displayed with a line of a predetermined color, and a display of the program column with an icon such as an arrow (see FIG. 8).

  When the program field in the live view image is moved due to a change in the position or angle of the imaging unit 11, the identification display control unit 28b is based on the feature data of the program field and the position coordinates of the program field on the paper M. Follow this.

  The mode display control unit 28 c switches the mode display according to the selection by the mode selection unit 26. Note that interfaces such as tabs and buttons in each mode are displayed at specific positions on the display screen in a layer different from the live view image.

  The program information control unit 28d is a processing unit that causes the display unit 13 to display program information acquired from the distribution server S in response to a request from the program information request unit 24b. This program display may also be displayed at a specific position on the display screen in a layer different from the live view image.

  Further, when a pop-up image including program information is displayed at a specific location in the program column in the live view image and the angle or position of the imaging unit 11 is changed, the pop-up image may be tracked as the program column is moved. A pop-up image displays characters and figures with a color-coded background.

  The program information includes moving images. When displaying a moving image, only a moving image may be displayed, or a live view image and a moving image may be divided or displayed simultaneously by a window.

  The shared information control unit 28e is a processing unit that causes the display unit 13 to display the shared information acquired from the distribution server S in response to a request from the shared information request unit 24d. Similarly to the above, this shared information may be displayed in a layer different from the live view image or may be displayed as a pop-up image. The processing of the program information control unit 28d and the shared information control unit 28e is included in the information display processing in the claims.

  Further, the display control unit 28 includes a user interface (UI) control unit 28f. The UI control unit 28 f is a processing unit that specifies a UI area for inputting a desired operation from the images displayed on the display unit 13. That is, the UI control unit 28f specifies areas such as program columns, tabs, buttons, and icons included in the display screen as areas that can be operated by the touch panel. Thus, the user can perform an operation input by performing a swipe or touch on the area.

  In addition, this embodiment is a new mode of so-called augmented reality, AR (Augmented Reality), from the function of being able to access the program information and operating the recording / reproducing device V based on the live view image of the program guide. be able to.

  The information communication device P can typically be configured by installing an application program (hereinafter simply referred to as an application) on a smartphone. For this reason, settings, arithmetic expressions, parameters, and the like necessary for the processing of each unit described above are included in the application, and these are stored in advance in a built-in or removable memory.

[1-3. Distribution server]
As shown in FIG. 3, the distribution server S is a server managed by a magazine publisher that publishes a program guide, a cable television company that distributes a program guide, and the like. The distribution server S includes a storage unit 110 and a distribution control unit 120.

  The storage unit 110 is a processing unit that stores information for distribution. The storage unit 110 stores the application, feature data, identification display data, program information, shared information, and the like. The application version, feature data, identification display data, program information, and shared information are updated as appropriate. The storage unit 110 stores the feature data in association with the ID of the program guide for each broadcast date.

  The distribution control unit 120 includes a transmission / reception unit 121, an application distribution unit 122, a feature data distribution unit 123, a request acquisition unit 124, an identification display data distribution unit 125, a program information transmission unit 126, a shared information collection unit 127, and a shared information transmission unit 128. And a search unit 129.

  The transmission / reception unit 121 is a processing unit that transmits / receives information to / from the information communication device P and other servers via the network N. For example, the transmission / reception unit 121 has a function of connecting to the Internet.

  The application distribution unit 122 is a processing unit that distributes the application in response to a download request for the application of the present embodiment from the information communication device P.

  The feature data distribution unit 123 is a processing unit that distributes the feature data to the information communication device P at a predetermined timing. This timing can be freely set. For example, first, the feature data distribution unit 123 distributes the feature data together with the distribution of the application.

  After that, the feature data distribution unit 123 outputs the feature data at the time of activation of the information communication device P, at the time of activation of the application at the information communication device P, at the time of update of the application at the information communication device P, at a predetermined period set in advance. To deliver.

  Since the feature data is updated every time the program information is updated, the feature data distribution unit 123 may distribute the feature data accordingly. The feature data distribution unit 123 may check the version of the application transmitted from the information communication device P side and the identification information of the feature data, and may distribute the information only when it is not the latest.

  The request acquisition unit 124 is a processing unit that acquires a transmission request from the information communication device P. The identification display data transmission unit 125 is a processing unit that transmits identification display data in response to the transmission request acquired by the request acquisition unit 124. The program information transmission unit 126 is a processing unit that transmits program information in response to the transmission request acquired by the request acquisition unit 124.

  The shared information collection unit 127 is a processing unit that collects shared information from another server or the like. The shared information transmission unit 128 is a processing unit that transmits shared information in response to the transmission request acquired by the request acquisition unit 124. The search unit 129 is a processing unit that searches for necessary information from the storage unit 110 in response to the received transmission request.

  Note that an analysis unit that analyzes the paper surface M may be set in the distribution server S. When the analysis unit receives the image data of the paper surface M, the analysis unit extracts feature data of the paper surface M from the image. The storage unit 110 stores the feature data of the program guide for each date in association with the broadcast date of the program.

[1-4. Recording / playback equipment]
The recording / playback device V is a recorder that includes a tuner and a recording device and is connected to a television. For example, the recording / playback device V includes a DVD recorder, a Blu-ray recorder, and a hard disk recorder. A personal computer equipped with a tuner and a recording device and a television equipped with a recording / playback function also function as the recording / playback device V.

  The recording / reproducing device V records information received by the tuner and reproduces the recorded information. The recording / playback device V is set to be connected to the network N so as to operate in accordance with a command from the information communication device P.

[2. Action]
An example of the processing of the present embodiment as described above will be described with reference to the sequence diagram of FIG. 4 and the flowcharts of FIGS. Note that a processing method according to the following procedure is also an embodiment of the present invention.

[2-1. Recognition of program guide]
First, it is assumed that the information communication device P has already downloaded and installed an application. In addition, the distribution server S periodically uploads feature data of the program guide, and the storage unit 110 stores the data in association with the broadcast date of the program for each date.

  When the user performs an application activation operation, the application is activated to initialize settings, and the transmission / reception unit 10 connects to the network N, and the imaging unit 11 is activated (step 101).

  The distribution server S distributes the feature data to the information communication device P at the timing as described above. For example, as shown in FIG. 4, when the activated application requests feature data from the distribution server S, the distribution server S transmits the feature data. The feature data is received by the transmission / reception unit of the information communication apparatus P and stored in the feature data storage unit 20 (step 102).

  The user directs the optical unit 11a of the imaging unit 11 to a paper surface M on which a program guide such as a magazine or a newspaper is posted. Then, based on the light received by the image sensor unit 11b, the live view image control unit 28a causes the display unit 13 to display the live view image of the paper surface M (step 103).

  The feature extraction unit 22a extracts feature data of the paper surface M from the live view image (step 104). The matching unit 23 matches the feature data stored in the feature data storage unit 20 with the feature data extracted by the feature extraction unit 22a, so that the program table of the captured paper M, its date, and each program column Is identified (step 105). If there is no corresponding feature data in the feature data storage unit 20, the matching unit 23 cannot specify the program guide (NO in step 106). This process is repeated until the program guide can be specified or until the application ends.

  When the matching unit 23 can identify the date and each program column (YES in step 106), the request unit 24 outputs a request for identification display data to the distribution server S based on the date of the identified program table (step 107). ).

  The transmission / reception unit 10 transmits a request for identification display data to the distribution server S as shown in FIG. In the distribution server S, the request acquisition unit 124 acquires a request for recognition display data received by the transmission / reception unit 121. The search unit 129 searches the storage unit 110 for identification display data corresponding to the requested program schedule on the date. The identification display data transmission unit 125 causes the transmission / reception unit 121 to transmit the retrieved identification display data.

[2-2. Search mode display]
The transmission / reception unit 10 in the information communication device P receives the identification display data (step 108). Then, the mode display control unit 28c causes the display unit 13 to display a mode according to the selection by the mode selection unit 26 (step 109). Here, since the search mode is set by default, the mode display control unit 28c causes the display unit 13 to display the search mode.

  That is, the identification display control unit 28b displays the identification display on the live view image of the display unit 13 in the program column that matches the conditions set in the condition setting unit 25. The transmission / reception unit 10 receives the title of the program identified and displayed as described above, which is transmitted by the program information transmission unit 126 in response to a request from the program information request unit 24b. Then, the mode display control unit 28c displays the title in a layer different from the live view image on the display unit 13.

  Examples of such display are shown in FIGS. In this example, on the paper M included in the live view image, two program fields that meet the conditions are surrounded by a color-coded frame, and an arrow pointing to this is displayed. In addition, the title of the identified program is displayed at the top of the display screen.

  Even if the position and direction of the information communication device P change and the program column in the live view image moves, the identification display control unit 28b uses the feature data of the program column and the position coordinates of the program column on the page M. Based on this, the identification display is made to follow the program column.

  In FIG. 9, a button for requesting program update information is displayed. When this button is selected (YES in step 111), the program information request unit 24b transmits a request for updated program information to the distribution server S via the transmission / reception unit 10 (step 112).

  Then, the update information is transmitted from the program information transmission unit 126 of the distribution server S, which is received by the transmission / reception unit 10 (step 113), and is displayed on the display unit 13 by the program information control unit 28d (step 114). Regardless of whether the button is displayed or not, when the program column identified and displayed is tapped, the program information of the program may be displayed as described above.

  Further, in the examples of FIGS. 8 and 9, an example in which the search mode is not displayed at the time of initial recognition is shown. However, as shown in FIG. 10, in the initial search mode, the mode display unit 13 may be set to display an interface such as a mode selection tab.

  In the example of FIG. 10, a tab for selecting a mode is displayed at the top of the screen. A genre selection button for changing the condition setting by the condition setting unit 25 to a preset genre is displayed at the bottom of the screen.

  For example, as shown in FIG. 11, when a news button is selected, the identification display control unit 28b displays the news program in an identification manner. This genre selection button can also display other genres by scrolling. Further, as shown in FIG. 12, the interface can be changed to a search for a keyword that can be input by the input unit 14.

  Furthermore, the request and display modes of program information can also be as shown in FIGS. That is, as shown in FIG. 13, when there is updated program information, the program information control unit 28d displays an icon on the screen. In the example of FIG. 13, the icon “!” Is displayed at the upper left of the screen.

  As shown in FIG. 14, when this icon is tapped, update information is transmitted from the program information transmission unit 126 of the distribution server S in response to a request from the program information request unit 24b. Then, the transmission / reception unit 10 receives this, and the program information control unit 28d causes the display unit 13 to display it.

  For example, as shown in FIG. 15, program information is displayed in a dialog. When there are a plurality of pieces of program information, they are displayed on the live view image in a scroll screen. When there is information on a plurality of programs, the program is numbered and the corresponding number is displayed in the program column corresponding to the displayed program information.

  As described above, after displaying the program information or when there is no request for the program information (NO in step 110) and the user changes the imaging target to the program table of a different date (YES in step 115), the above-mentioned Steps 104 to 114 are performed.

  If the user does not change the imaging target (NO in step 115), there is no operation on the mode change tab (NO in step 116), and there is no end operation (NO in step 117), the process returns to step 111. If there is an end operation (YES in step 117), the transmission / reception unit 10 disconnects the network (step 118) and ends the process.

[2-3. Device operation mode display]
When a mode change tab operation is performed (YES in step 116) and the device mode is selected (step 119), the mode display control unit 28c sets the display on the display unit 13 to the device mode (step 201 in FIG. 6). Even in this device mode, for example, as shown in FIG. 16, the screen display includes a live view display.

  The UI control unit 28f waits for a tap to select any program column in the program table displayed in live view (NO in step 202). When the UI control unit 28f detects that any one of the program columns has been selected (YES in step 202), the information communication device P, as described above, responds to the request from the program information request unit 24b. Program information is acquired (step 203).

  When the mode display control unit 28c determines that the program is an unbroadcasted program by comparing the broadcast date and time of the program included in the program information with the current date and time (YES in step 204), the mode display control unit 28c A recording reservation button is displayed (step 205). For example, as shown in FIG. 17, a screen display including a recording reservation button is displayed along with the display of the program name, channel, and time.

  Note that the screen example in FIG. 17 includes a detailed setting button and a next program button. The detailed setting button is used when making detailed settings such as making reservations continuously every week or designating one of a plurality of hard disks. The next program button is used in the following cases. That is, in the case of a program with a short time before and after a large frame program column, it is difficult to select by a tap because the program column is narrow. Therefore, if the next program button is selected for the next program after the currently selected program, the next program can be easily identified and displayed.

  The UI control unit 28f waits until the displayed recording reservation button is selected (NO in step 206). When the UI control unit 28f detects that the recording reservation button has been selected (YES in step 206), the device operation request unit 24c causes the transmission / reception unit 10 to transmit a recording reservation command for the program (step 207). The recording / reproducing apparatus V which received this performs the recording reservation setting of the said program.

  As shown in FIG. 18, the identification display control unit 28 b displays a clock mark as a recording-reserved icon along with a frame for a program column for which recording reservation is set. As a result, the user can surely recognize a program that has been scheduled for recording.

  As shown in FIG. 19, when the recording is completed and the recording is completed, the identification display control unit 28b displays ▲ as a mark indicating that the recording has been performed for the program column. As shown in FIG. 19, when the recorded data can be transferred, an arrow is displayed as a mark indicating that transfer is possible.

  The determination as to whether or not the recording is completed is made based on, for example, the recording history stored in the memory of the information communication apparatus P or the data of the recorded program received by the transmission / reception unit 10 when communicating with the recording / reproducing device V. Whether or not transfer is possible depends on the settings of the recording / reproducing device V and the information communication device P.

  When the mode display control unit 28c determines that the selected program is not unbroadcasted (NO in step 204) and is a program being broadcast (YES in step 220), the mode display control unit 28c causes the display unit 13 to display a recording button and a selection button. A station button is displayed (step 221). For example, as shown in FIG. 20, the screen display includes a program name, a channel, and a time, and a recording button and a channel selection button.

  The UI control unit 28f waits until the displayed recording button or channel selection button is selected (NO in step 222). When the UI control unit 28f detects that the recording button or the channel selection button is selected (YES in Step 222), the device operation request unit 24c transmits the recording command or channel selection command of the program by the transmission / reception unit 10. (Step 223).

  The recording / playback device V that has received the command records the program being broadcast in the case of a recording command, and displays the program being broadcast by the recording / playback device V in the case of a channel selection command.

  As shown in FIG. 21, when a recorded program and it can be transferred, a mark is displayed on the program. When the mode display control unit 28c determines that the selected program is not being broadcast (NO in step 220) and has been broadcast and recorded (YES in step 230), the mode display control unit 28c displays a device playback button and A transfer button is displayed (step 231).

  For example, as shown in FIG. 22, the screen display includes a program name, a channel, and a time, and a device playback button and a transfer button.

  The UI control unit 28f waits until the displayed device playback button or transfer button is selected (NO in step 232). When the UI control unit 28f detects that the device playback button or the transfer button is selected (YES in step 232), the device operation request unit 24c causes the transmission / reception unit 10 to transmit the recording command or transfer command of the program. (Step 233).

  The recording / reproducing device V that has received this reproduces the recorded data or transfers it to the information communication device P. When the transmission / reception unit 10 receives the transferred recording data, the reproduction control unit 27 causes the audio output unit 12 and the display unit 13 controlled by the display control unit 28 to reproduce the recording data. Thereby, the recorded program can be viewed on the information communication apparatus P.

  After the above processing or when the selected program is not a recorded program (NO in step 230) and the user changes the imaging target (YES in step 208), the processing returns to step 104 in FIG. If the user does not change the imaging target (NO in step 208), there is no operation on the mode change tab (NO in step 209), and there is no end operation (NO in step 210), the process returns to step 202. If there is an end operation (YES in step 210), the transmission / reception unit 10 disconnects the network (step 211) and ends the process.

[2-4. Net mode display]
When the mode change tab operation is performed (YES in step 116 or step 209) and the net mode is selected (step 119 or step 212), the mode display control unit 28c sets the display on the display unit 13 to the net mode. (Step 301 in FIG. 7). For example, the screen display as shown in FIG.

  In the example of FIG. 23, as in the search mode, frames are displayed in different colors in the program column that meets the set conditions or the program column selected by the user on the paper M included in the live view image. In addition, an information selection button for selecting shared information to be displayed is displayed at the bottom of the screen. For example, buttons such as popularity, review, and recommendation are displayed.

  The UI control unit 28f waits for the information selection button to be selected (NO in step 302). When the information selection button is selected (YES in step 302), the shared information request unit 24d causes the transmission / reception unit 10 to transmit a request for update information to the distribution server S (step 303). Then, the shared information is transmitted from the shared information transmitting unit 126 of the distribution server S, which is received by the transmitting / receiving unit 10 (step 304), and the identification display control unit 28b causes the display unit 13 to display the shared information (step). 305).

  For example, when the popular button is selected, the shared information is displayed for each program identified and displayed as shown in FIG. 1, no. An icon such as 2 is displayed. Also, as shown in FIG. 25, when the review button is selected, a star display with a full score of 5 is displayed. When the recommendation button is selected, the recommended information of the program pops up.

  After the above process, when the user changes the imaging target (YES in step 306), the process returns to step 104 in FIG. If the user does not change the imaging target (NO in step 306), there is no operation on the mode change tab (NO in step 307), and there is no end operation (NO in step 308), the process returns to step 302. If there is an end operation (YES in step 308), the transmission / reception unit 10 disconnects the network (step 309) and ends the process.

  When the mode change tab operation is performed (YES in step 307) and the device mode is selected (step 310), the mode display control unit 28c sets the display on the display unit 13 to the device mode (step 201 in FIG. 6). . When the search mode is selected, the process returns to step 110.

  It should be noted that the display of various types of information by the display control unit 28 can be set in two modes as described above. First, there is a mode in which information is displayed following the position of the specific portion when the image pickup unit 11 is displayed at a specific portion of the program in the program table in the live view image.

  On the other hand, various information is displayed at a specific position on the display screen in a layer different from the live view image. In this aspect, the display of information does not move even when the imaging unit 11 moves. For this reason, there is an advantage that the contents of information can be surely shown to the user and the user can easily read.

  Further, the display control unit 28 can display a moving image on the display unit 13 as described above. In this case, only a moving image may be displayed. However, it is possible to confirm what video is being reproduced by displaying the live view image and the moving image simultaneously by dividing or displaying the window. By displaying other information together with the live view image, such as information exchanged via the network, information about the community between users by social network service (SNS), other page numbers related on paper, etc. Information confirmation and operation can be integrated into one screen.

[3. effect]
The effects of the present embodiment as described above are as follows.
[User side]
The user can immediately identify a desired program simply by pointing the imaging unit 11 of the information communication device P toward the program guide. For this reason, it is possible to quickly recognize a desired program by a very simple and easy-to-understand method without taking time and effort for specifying an area. Since information necessary for operation, such as information necessary for program identification, is input by image recognition by the imaging unit 11, operations that are complicated and complicated only by the GUI can be simplified. That is, as compared with the case where character input is performed according to a preset procedure, an object captured and projected can be used as a selection object as it is, so that it is possible to greatly save the input effort.

  Even when using a GUI, as described above, switching to the optimal mode display according to the required related information and device operation, the so-called analog paper surface specification and the portion with good operation by the touch panel are combined. Man-machine interface can be used.

  Further, the user can obtain information on the program simultaneously with the identification display. Furthermore, the user can easily obtain program information by selecting a program column in the live view image.

  In addition, the user can cause the recording / playback device V to record and play back a desired program by selecting the program field of the live view image in the device mode. For this reason, it does not take time and effort to recognize the G code and bar code, identify the character, and the like. Since the user can select the program column while viewing the program table of the live view image, it is easy to understand which program is selected. Furthermore, since no G code or bar code printing space is required, more program information can be obtained.

  Moreover, by specifying a page from which the user sees information that makes sense in the layout, such as a program guide, as a recognition target, it is possible to specify the coordinates from the ratio to the size of the page obtained by the recognition. It is possible to display related information in accordance with a target with a dynamic position.

[Program guide provider]
Program guides in paper media such as magazines and newspapers are suitable for listing all programs of the day as described above, compared to electronic program guides. In addition to this advantage, by using the present embodiment, it is possible to easily search for a desired program, schedule recording of a program, and the like. For this reason, the provider side can increase the willingness to purchase paper media such as magazines and newspapers. It also leads to re-recognition of the value of paper media compared to the electronic program guide.

[4. Other Embodiments]
The present invention is not limited to the above embodiment. The program guide includes program guides for all broadcast programs. Accordingly, program guides for terrestrial digital, BS, CS, cable TV, and Internet TV are also included. A radio broadcast program guide is also included.

  Further, the content displayed on the page is not limited to the program guide. For example, when the external device is a cooking device such as a microwave oven and a recipe book for cooking is imaged, a command for setting the cooking device according to the contents of the recipe may be transmitted. In this case as well, it is possible to save time and effort for inputting information necessary for the operation such as the cooking mode and time by selecting the video imaged and projected.

  Further, when the contents of the instruction manual are imaged, a command for setting may be transmitted to the target device of the instruction manual. For example, a command for checking a malfunction of the air conditioner and a command for switching the setting of the digital camera can be transmitted. Also in this case, by selecting an image that has been captured and displayed as an object to be selected, it is possible to save time and effort for inputting information necessary for operations such as adjustment settings described on the page.

  About the aspect of the icon as an identification display, it is not limited to said aspect. For example, a three-dimensional display mode or a display mode with a movement that changes in size and position can be used. Furthermore, the size and movement of the icon can be changed depending on the degree of coincidence of the search boundaries, for example, if the actor is a leading role or a supporting role. It is also possible to move the frame itself for identifying the program column etc. or change the color.

M ... paper N ... network P ... information communication device S ... distribution server V ... recording / playback device 10 ... transmission / reception unit 11 ... imaging unit 11a ... optical unit 11b ... image sensor unit 12 ... audio output unit 13 ... display unit 14 ... input unit 20 ... Feature data storage unit 22 ... Analysis unit 22a ... Feature extraction unit 23 ... Matching unit 24 ... Request unit 24a ... Identification display data request unit 24b ... Program information request unit 24c ... Device operation request unit 24d ... Shared information request unit 25 ... Condition setting unit 26 ... mode selection unit 27 ... reproduction control unit 28 ... display control unit 28a ... live view image control unit 28b ... identification display control unit 28c ... mode display control unit 28d ... program information control unit 28e ... shared information control unit 28f ... UI control unit 110 ... storage unit 120 ... delivery control unit 121 ... transmission / reception unit 122 ... application delivery unit 123 ... feature data delivery unit 124 ... request acquisition unit 1 5 ... identification data transmitting unit 126 ... program information transmitting unit 127 ... sharing information gathering unit 128 ... shared information transmitting unit 129 ... search section

Claims (11)

To the computer connected to the transmission / reception unit, imaging unit, and display unit,
A feature data storage process for storing feature data obtained by extracting feature quantities on the appearance of the content displayed on the paper and identification information of the content;
An imaging process for causing the imaging unit to image the content displayed on the paper;
Display control processing for displaying a live view image captured by the imaging unit on a display unit;
A feature extraction process for extracting feature data on the appearance of the content displayed on the paper from the live view image;
A matching process for identifying the content displayed on the imaged paper by matching the feature data stored by the feature data storage process and the feature data extracted by the feature extraction process;
Based on the identification information of the content specified by the matching process, an identification display data request process for transmitting the identification display data request for identifying display for identifying a part of the content to the transmission / reception unit;
Identification display control processing for displaying the identification display data received by the transmission / reception unit in response to the request by the identification display data request processing together with the live view image on the display unit,
An information communication program characterized in that In the computer,
An information request process for causing the transmission / reception unit to transmit a request for information on a part of the content, the part on which the identification display is made,
In response to a request by the information request process, information received by the transmission / reception unit is displayed on the display unit, information display processing,
The information communication program according to claim 1, wherein: In the computer,
The information communication program according to claim 1 or 2, wherein an operation request process for causing the transmission / reception unit to transmit an operation request to an external device is executed for the portion on which the identification display is made. The page is a program table of a broadcast program,
The information communication program according to any one of claims 1 to 3, wherein the identification display is a display for identifying each program in the program table included in the live view image. In the computer,
5. The information communication program according to claim 4, wherein a condition setting process for setting a condition for identifying and displaying which program column in the program table included in the live view image is executed. In the computer,
The information communication program according to any one of claims 1 to 5, wherein a user interface control process is performed in which the area where the identification display is made is used as a user interface for inputting information. In the computer,
The information communication program according to claim 2, wherein a mode display control process for performing a mode display including a user interface for performing the information request process is executed. In the computer,
The information communication program according to claim 3, wherein a mode display control process for performing a mode display including a user interface for performing the operation request process is executed. The identification display data includes a ratio of a part of the content with respect to a total size of a captured image or a position coordinate of a part of the content calculated from the ratio. 9. The information communication program according to any one of items 8. A transmission / reception unit that transmits and receives information via a network;
A feature data storage unit for storing feature data obtained by extracting feature quantities on the appearance of the content displayed on the paper and identification information of the content;
An imaging unit for imaging the content displayed on the paper;
A display unit for displaying a live view image captured by the imaging unit;
A feature extraction unit that extracts feature data on the appearance of the content displayed on the paper from the live view image;
A matching unit for specifying the content displayed on the captured paper by matching the feature data stored by the feature data storage unit and the feature data extracted by the feature extraction unit;
Based on the identification information of the content specified by the matching unit, a request for identification display data for identifying display for identifying a part of the content, an identification display data request unit for transmitting to the transceiver unit,
An identification display control unit that causes the display unit to display the identification display data received by the transmission / reception unit in response to a request from the identification display data request unit, together with the live view image;
An information communication device comprising: A transmission / reception unit connected to the information communication device according to claim 10 via a network;
A feature data distribution unit that causes the transceiver unit to transmit the feature data;
In response to a request from the identification display data request unit, an identification display data transmission unit that causes the transmission / reception unit to transmit identification display data;
A distribution server characterized by comprising:

Images