JP2019091151A - Information processing device, information processing method and program - Google Patents

Abstract

To improve visibility of a screen.SOLUTION: An information processing device comprises: a speech information acquisition unit which acquires speech information that a user inputs; a conversion unit which converts the speech information into text data; an answering unit which determines answering based upon the test data and executes the answering; and a control unit which controls display processing according to contents of the answering so that in a screen consisting of a first display region for displaying an interaction partner image showing an interaction partner of the user and interaction contents that the interaction partner outputs in response to the answering, a second display region for displaying the text data, and a third display region for displaying services provided by the answering, the third display region is arranged between the first display region and the second display region, and the interaction contents are displayed in text representation more on the third display area side than the interaction partner image.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

  In recent years, customer operation type terminals have been installed in various places, and a user can receive various services by operating the customer operation type terminals by himself. For example, a customer-operated terminal installed in a movie theater displays seating information of a movie selected by the user, executes settlement processing, and issues a ticket. However, when the user operates the customer operation type terminal, there is a problem that the user takes time to search for a target service due to diversification of services that can be provided by the customer operation type terminal. If it takes time to search and it is not possible to find the desired service, the user may give up on the way. Further, as an alternative means for the user to search, there is a means for the operator to perform remote control using screen sharing software, but there is a problem that the unmanned operation can not be performed. Therefore, a method capable of operating the customer operation type terminal in an interactive manner has been developed.

  As a method capable of operating a customer operation type terminal in an interactive manner, for example, Patent Document 1 below recognizes a user's uttered content by a voice recognition function, and performs predetermined processing based on information acquired from the user's uttered content. A method for performing is disclosed. According to the method of Patent Document 1, the user can receive the intended service without directly operating the customer operation type terminal by conveying the purpose to the customer operation type terminal by speech.

JP, 2015-076038, A

  However, the invention of Patent Document 1 has a problem that the visibility of the screen is poor. For example, in the screen layout described in Patent Document 1, there is a speech balloon on the right of the avatar image that displays the speech content of the avatar as text, and an area for displaying a service provided to the user is displayed below the avatar. In the above-described layout, when the user views the information provided by the avatar, the amount of movement of the user's line of sight between the balloon and the service content increases.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide a new and improved information processing apparatus and information processing method capable of improving the visibility of a screen. And providing a program.

  To solve the above problems, according to one aspect of the present invention, a voice information acquisition unit for obtaining voice information input by a user, a conversion unit for converting the voice information into text data, and the text data A response unit for determining a response and executing the response, an interactive partner image showing the interactive partner of the user, and a first display area for displaying interactive contents output by the interactive partner according to the response; And a second display area for displaying data and a third display area for displaying a service provided by the response, wherein the third display area is the first display area and the second display area. Control unit for controlling the display processing according to the contents of the response so as to be disposed between the display areas of the display unit and to arrange the interactive contents in the text representation on the third display area side with respect to the interactive partner image It comprises a information processing apparatus is provided.

  The control unit may arrange the first display area above the third display area, and arrange the second display area below the third display area.

  The information processing apparatus further includes a detection device capable of detecting the user, and a screen operation information acquisition unit that acquires screen operation information input by a screen operation of the user, and the control unit When the detection device detects the user, it may stand by in a state in which the voice information and the screen operation information can be received.

  When the screen operation information acquisition unit acquires the screen operation information, the response unit may determine the response based on the screen operation information and execute the response.

  The control unit may display, in the third display area, a service capable of selection by voice input and selection by touch of a screen.

  The control unit adjusts the size of the display area of the conversation partner image according to the size of the first display area when the size of the first display area changes at the time of screen switching. It is also good.

  When there is no service to be displayed in the third display area, the control unit hides the third display area, and sets a display area above the second display area as the first display area. It is also good.

  The control unit may display the dialogue content and the text data in a balloon.

  The control unit may display a user image indicating the user in the second display area, and display the text data between the user image and the third display area.

  The control unit hides the second display area and the third display area when the response unit is executing a process that does not require interaction with the user, and the control unit is configured to hide the second display area and the third display area. The lower display area may be used as a fourth display area.

  The information processing apparatus further includes a payment processing unit that controls payment processing, and a ticketing processing unit that controls ticketing processing, and the control unit is processing by the payment processing unit or the ticket processing unit. In the case, the service information may be displayed in the fourth display area.

  The control unit may be capable of switching the speech recognition function on or off.

  When the voice recognition function is on and the user speaks while displaying the fourth display area, the control unit hides the service information and displays the second display area. It is also good.

  The service may be a seat reservation service.

  In order to solve the above problems, according to another aspect of the present invention, acquiring voice information inputted by a user, converting the voice information into text data, and based on the text data, Determining a response, executing the response, an interactive partner image showing the interactive partner of the user, and a first display area for displaying interactive contents output by the interactive partner according to the response; and the text data The third display area is a screen including the second display area to be displayed and the third display area to display the service provided by the response, wherein the third display area is the first display area and the second display. Controlling the display processing according to the contents of the response so as to be arranged between the areas and to arrange the dialogue contents in the text representation on the third display area side with respect to the dialogue partner image Executed by the processor, information processing method is provided.

  In order to solve the above problems, according to another aspect of the present invention, a computer, an audio information acquisition unit for acquiring audio information input by a user, a conversion unit for converting the audio information into text data, and A response unit that determines a response based on the text data and executes the response, a conversation partner image indicating the conversation partner of the user, and a first content of the dialog content displayed by the conversation partner according to the response The third display area is a screen comprising a display area, a second display area for displaying the text data, and a third display area for displaying a service provided by the response, wherein the third display area is the first display area. A table according to the contents of the response is disposed between the display area and the second display area, and the dialogue contents are arranged in the text representation on the third display area side with respect to the dialogue partner image. To function as a control section for controlling the process, the program is provided.

  As described above, according to the present invention, it is possible to realize improvement in the visibility of the screen.

It is an explanatory view showing the outline of the information processing system concerning the embodiment of the present invention. It is an explanatory view showing an example of appearance appearance of a customer operation type terminal concerning the embodiment. It is a block diagram showing an example of functional composition of a customer operation type terminal concerning the embodiment. It is a block diagram showing an example of functional composition of a WebAP server concerning the embodiment. It is an explanatory view showing an example of composition of a layout of a screen concerning the embodiment. It is an explanatory view showing the example of a display of the conversation partner picture concerning the embodiment. It is an explanatory view showing the example of a display of the initial screen concerning the embodiment. It is an explanatory view showing the example of a display of the screen change by the dialogue concerning the embodiment. It is an explanatory view showing the example of a display of the service concerning the embodiment. It is an explanatory view showing the example of a display of the details of the service concerning the embodiment. It is an explanatory view showing an example of a display of a screen at the time of settlement processing concerning the embodiment. It is an explanatory view showing an example of a display of a screen under ticketing processing concerning the embodiment. It is an explanatory view showing an example of a display of a screen after ticket issue concerning the embodiment. It is a sequence diagram which shows the operation example of the customer operation type terminal at the time of the user detection which concerns on the embodiment. It is a sequence diagram which shows the operation example of the WebAP server at the time of the user detection which concerns on the embodiment. It is a sequence diagram which shows the operation example of the customer operation type terminal at the time of the audio | voice information acquisition which concerns on the embodiment. It is a sequence diagram which shows the operation example of the WebAP server at the time of the audio | voice information acquisition which concerns on the embodiment. It is a sequence diagram which shows the operation example of the customer operation type terminal at the time of screen operation information acquisition which concerns on the embodiment. It is a sequence diagram which shows the operation example of the WebAP server at the time of screen operation information acquisition which concerns on the embodiment. It is an explanatory view showing an example of a display of an initial screen in a modification concerning one embodiment of the present invention. It is an explanatory view showing screen change by dialogue in a modification concerning one embodiment of the present invention. It is an explanatory view showing an example of a display of the 3rd display field in a modification concerning one embodiment of the present invention. It is an explanatory view showing an example of a display of the 3rd display field when screen operation in a modification concerning one embodiment of the present invention is performed. It is an explanatory view showing an example of a display of the 2nd display field at the time of a plurality of user detection in a modification concerning one embodiment of the present invention. It is an explanatory view showing the example of a display of the operation display part in the horizontal screen in the modification concerning one embodiment of the present invention. It is a block diagram showing an example of hardware constitutions of a customer operation type terminal concerning the embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration will be assigned the same reference numerals and redundant description will be omitted.

  Further, in the present specification and the drawings, a plurality of components having substantially the same functional configuration or logical meaning may be distinguished by appending different alphabets after the same symbol. However, when it is not necessary to distinguish each of a plurality of components having substantially the same functional configuration or logical meaning, each of the plurality of components is only given the same reference numeral.

<1. Outline of information processing system>
Embodiments of the present invention relate to an information processing system for providing various services to users. Hereinafter, an information processing system according to an embodiment of the present invention will be described by taking a ticket issuing service of a ticket as an example. Note that the service is not limited to the ticket issue service, and any service may be provided. For example, a seat reservation service may be provided. First, an overview of an information processing system according to an embodiment of the present invention will be described with reference to FIG.

  FIG. 1 is an explanatory view showing an outline of an information processing system according to an embodiment of the present invention. As shown in FIG. 1, the customer operation type terminal 10 is connected to a WebAP (AP: Application) server 30 via a network 14.

(1) Customer operation type terminal 10
The customer operation type terminal 10 is an information processing apparatus for issuing a ticket based on the information input by the user. The customer operation type terminal 10 shown in FIG. 1 can show a response based on the information input by the user. Here, the response is the next conversation content for the user's speech, the service provided for the user's request, data displayed on the screen, and the like. Here, uttering means inputting / outputting dialogue contents as speech, outputting dialogue contents in text notation, and inputting / outputting dialogue contents in speech and text notation. For example, the customer operation type terminal 10 transmits the input information of the user to the WebAP server 30 via the network 14, and receives the content of the response determined by the WebAP server 30 from the WebAP server 30. The customer operation type terminal 10 displays the search result of the ticket on the screen based on the contents of the response received from the Web AP server 30, and performs the settlement process and the ticket issue process of the ticket selected by the user.

  Further, since the customer operation type terminal 10 has a voice recognition function and can recognize the user's voice, the user can input information by speech. Further, the customer operation type terminal 10 can reproduce the next conversational sentence for the user's speech by voice. Therefore, the user can operate the customer operation type terminal 10 by interacting with the customer operation type terminal 10. In addition, the operation means of the customer operation type terminal 10 is not limited to operation by voice input. For example, the user can also input by screen operation.

(2) WebAP server 30
The WebAP server 30 has a function of determining a response to user's input information. The WebAP server 30 includes artificial intelligence (AI: Artificial Intelligence), and the AI analyzes a user's input information to determine a response. For example, the WebAP server 30 receives the input information of the user from the customer operation type terminal 10 via the network 14, and transmits the content of the response determined by the AI based on the input information of the user to the customer operation type terminal 10.

  The outline of the information processing system according to the embodiment of the present invention has been described above with reference to FIG. Subsequently, a configuration example of the customer operation type terminal 10 according to the embodiment of the present invention will be described.

<2. Example of configuration of customer operation type terminal>
FIG. 2 is an explanatory view showing an example of the appearance configuration of the customer operation type terminal 10 according to the embodiment of the present invention. FIG. 3 is a block diagram showing an example of the functional configuration of the customer operation type terminal 10 according to the embodiment of the present invention. In the embodiment of the present invention, although the form using the customer operation type terminal 10 shown in FIG. 2 is described as an example, the terminal used in the present invention is not limited to the customer operation type terminal 10, and any terminal may be used. It may be used. For example, the present invention may be implemented by installing an application on an information display device such as a smartphone, a portable terminal such as a tablet terminal, and a display for touch signage. Hereinafter, a configuration example of the customer operation type terminal 10 according to the embodiment of the present invention will be described with reference to FIGS. 2 and 3.

  As shown in FIGS. 2 and 3, the customer operation type terminal 10 according to the embodiment of the present invention includes the camera 120, the speaker 124, the microphone 128, the communication unit 132, the operation display unit 136, the payment medium unit 140, and the ticket. A ticket issuing unit 144 and a control unit 150 are provided.

(1) Camera 120
The camera 120 is one of detection devices having a function of photographing a person. For example, when shooting a user in front of the customer operation type terminal 10, the camera 120 transmits an image obtained by shooting the user to the camera AP 154. In the embodiment of the present invention, as shown in FIG. 2, one camera 120 is installed on the operation display unit 136, but the number and position of the cameras 120 are not particularly limited, and the customer operation The mold terminal 10 may have any number of cameras 120 at any position.

(2) Speaker 124
The speaker 124 has a function of reproducing sound. For example, the speaker 124 transmits a response to the user's input as a speech to the user by reproducing the audio file acquired by the screen AP 158. In the embodiment of the present invention, as shown in FIG. 2, one speaker 124 is installed on the left and right of the camera 120, but the number and position of the speakers 124 are not particularly limited. 10 may have any number of speakers 124 at any position. The customer operation type terminal 10 may be equipped with a near field communication function such as Bluetooth (registered trademark). The customer operation type terminal 10 may transmit the sound reproduced by the speaker 124 to a wireless device possessed by the user, such as a wireless earphone equipped with the wireless communication function, using the short distance wireless communication function. At this time, the user performs in advance a pairing process for connecting the customer-operated terminal 10 with his / her wireless device. As described above, when the device possessed by the user acquires and reproduces the voice, for example, the user can recognize the voice related to the response even in a noisy environment.

(3) Microphone 128
The microphone 128 is one of detection devices having a function of acquiring audio information. For example, the microphone 128 acquires voice information uttered by the user, and outputs the acquired voice information to the noise removal module 166. In addition, the customer operation type terminal 10 can recognize the arrival direction of the voice by providing the plurality of microphones 128, and can grasp the number of users speaking based on the direction of the voice. is there. In the embodiment of the present invention, as shown in FIG. 2, one microphone 128 is installed on the left and right of the operation display unit 136, but the number and position of the microphones 128 are not particularly limited. The mold terminal 10 may have any number of microphones 128 at any position.

(4) Communication unit 132
The communication unit 132 has a function of communicating various information with the WebAP server 30 via the network 14. For example, the communication unit 132 transmits the input information of the user acquired by the customer operation type terminal 10 via the operation display unit 136 to the Web AP server 30 via the network 14. The communication unit 132 also receives the contents of the response determined by the WebAP server 30 from the WebAP server 30 via the network 14.

(5) Operation display unit 136
The operation display unit 136 has a function as a display unit that displays a response to the user and a screen on which the user can input an operation. Further, the operation display unit 136 has a function as an operation unit for the user to perform an operation input. For example, the function as the display unit is realized by a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, or an OLED (Organic Light Emitting Diode) device. Moreover, the function as an operation part is implement | achieved by the touch panel, for example. In FIG. 2, the customer operation type terminal 10 has a configuration in which the functions of the display unit and the operation unit are integrated, but the functions of the display unit and the operation unit may be separated.

(6) Settlement medium unit 140
The settlement medium unit 140 (settlement processing unit) has a function as an opening through which a transaction medium used for settlement for ticket issuance is delivered. Trade media include credit cards or cash. Further, the settlement medium unit 140 has a function of performing settlement processing according to the transaction medium. For example, if the transaction medium is a credit card, the payment medium unit 140 reads the information recorded on the inserted credit card. In addition, when the transaction medium is cash, the settlement medium unit 140 has a cash transport path, a cash cassette, a discrimination unit for discriminating the legitimacy of cash, and the like. The cash inserted into the payment medium unit 140 at the time of deposit is transported to the discrimination unit via the transport path, and when a normal discrimination result is obtained by the discrimination unit, the cash is transported to the cash cassette according to the denomination. In addition, at the time of dispensing, cash is fed out from the cash cassette, the discrimination unit discriminates the fed banknote, and the cash is transported to the settlement medium unit 140 when a normal discrimination result is obtained. Note that electronic money and virtual currency can be used as a transaction medium, and in this case, the payment medium unit 140 may not be provided in the customer operation type terminal 10.

(7) Ticket issue unit 144
The ticket issue unit 144 (ticket issue processing unit) has a function of issuing a ticket. For example, the ticket issuing unit 144 prints on a sheet of paper the information of the ticket selected from the search result of the ticket displayed on the operation display unit 136 by the user, and the ticket is a sheet of paper on which the information of the ticket is printed. Discharge. The customer operation type terminal 10 may include a printer (not shown) in addition to the ticket issuance unit 144. For example, when the settlement is completed in the settlement medium unit 140, the printer may print the contents relating to the settlement on a paper medium and discharge the paper medium as a receipt. In addition, the printer may discharge the content on paper by printing the content input by the user on the paper. For example, if the service to be provided is a seat reservation service, the printer may print out the seat reservation content input by the user on paper and eject the paper-like medium. The printing process by the printer may be switched between the process of printing the payment contents on the receipt and the process of printing the reservation contents by the control information of the control unit 150. Further, the customer operation type terminal 10 may be separately provided with a printer for receipt issuance and a printer for reservation content printing.

(8) Control unit 150
The control unit 150 has a function of controlling the overall operation of the customer operation type terminal 10. For example, the control unit 150 has a function of controlling the camera 120, the speaker 124, the microphone 128, the communication unit 132, the operation display unit 136, the settlement medium unit 140, and the ticket issuing unit 144. In particular, the control unit 150 according to the embodiment of the present invention has a function of controlling display processing of a screen displayed on the operation display unit 136. In addition, about the detail of the display process of a screen, <4. Display Example of Screen> will be described.

  Further, the control unit 150 has a function of causing the customer operation type terminal 10 to stand by in a state in which both the voice information of the user and the screen operation information input by the user's screen operation can be received when the detection device detects the user. Have. Since the customer operation type terminal 10 can receive both the voice information and the screen operation information, the user can operate the customer operation type terminal 10 by either the operation by the voice input or the screen operation. It is. Although the customer operation type terminal 10 according to the embodiment of the present invention can receive only the operation by voice input at the first operation after the initial screen display, the acceptable operation is limited to the operation by voice input. It may be in a state where it can accept only an operation by an arbitrary input. In addition, at the first operation after the initial screen display, customer operation type terminal 10 is in a state capable of accepting operations by a plurality of means, including operations by input by another means, in addition to operations by voice input. May be

  In order to realize the functions described above, the control unit 150 according to the embodiment of the present invention includes a camera AP 154, a screen AP 158, a WebSocket server 162, a noise removal module 166, and a voice recognition AP 170.

(Camera AP154)
The camera AP 154 has a function of controlling processing relating to human detection. For example, when the camera AP 154 acquires an image of the user taken by the camera 120 from the camera 120, the camera AP 154 analyzes the acquired image and determines whether the user is detected. If it is determined that the user is detected, the camera AP 154 notifies the screen AP 158 and the voice recognition AP 170 that the user has been detected. At this time, the camera AP 154 also notifies information on the number of detected users. The camera AP 154 detects feature points of a person from the captured image, and acquires the number of users included in the image based on the detected feature points. When detecting the feature points, the camera AP 154 detects the feature points by referring to the feature points included in the human model stored in advance in a storage medium (not shown). Here, the feature points included in the person model are feature points that commonly exist in the person. In addition, the control unit 150 uses the customer operation type terminal 10 based on the information on the number of users detected by the camera AP 154 and the number of uttering users that can be grasped from the voice information acquired by the microphone 128 The number of users may be determined. For example, when the number of people detected by the camera AP 154 is two and the number of people detected by the microphone 128 is two, the control unit 150 determines that the number of users using the customer operation type terminal 10 is two. On the other hand, when the number of people detected by the camera AP 154 and the number of people detected by the microphone 128 are different, the control unit 150 may weight the number of uttered users to determine the number of users. For example, when the number of people detected by the camera AP 154 is three and the number of people detected by the microphone 128 is two, the control unit 150 determines that the number of users using the customer operation type terminal 10 is two. Further, when the user has paired the wireless device with the customer operation type terminal 10 by the short distance wireless communication function, the control unit 150 may set the number of paired wireless devices as the number of users.

(Screen AP158)
The screen AP 158 has a function of acquiring information to be displayed on the screen and displaying the acquired information on the operation display unit 136. For example, when the camera 120 detects a user, the screen AP 158 acquires a Web page URL of a screen to be displayed as an initial screen from the business Web AP server 304 of the Web AP server 30, and displays the initial screen on the operation display unit 136. Also, for example, when the screen AP 158 acquires text data of voice information input by the user from the voice recognition AP 170, the screen AP 158 displays the acquired text data on the operation display unit 136. Also, for example, when the screen AP 158 acquires the content of the response from the business Web AP server 304, the screen AP 158 displays the screen data indicated by the screen data URL included in the content of the response and text data on the operation display unit 136.

  Further, the screen AP 158 has a function of notifying of the speech recognition. For example, after displaying the initial screen on the operation display unit 136, the screen AP 158 notifies the voice recognition AP 170 via the WebSocket server that it is in a state of waiting for voice reception from the user. The screen AP 158 also has a function of giving an instruction regarding speech recognition. For example, the screen AP 158 instructs the voice recognition AP 170 to turn on or off the voice recognition function.

(WebSocket server 162)
The WebSocket server 162 has a function as a server that mediates the exchange of information between APs in the terminal. For example, the WebSocket server 162 mediates the exchange of information between the screen AP 158 and the voice recognition AP 170.

(De-noising module 166)
The noise removal module 166 has a function of removing noise from speech information. For example, the noise removal module 166 removes noise from the user's voice information acquired from the microphone 128. After the noise removal processing, the noise removal module 166 outputs the voice information to the voice recognition AP 170.

(Voice recognition AP 170)
The voice recognition AP 170 has a function of acquiring text data corresponding to voice information of the user from the Web AP server 30. For example, when the speech recognition AP receives the speech information from the noise removal module 166, the speech recognition AP transmits the received speech information to the speech recognition engine unit 308 of the WebAP server 30. Thereafter, when the speech recognition AP 170 receives the text data converted by the speech recognition engine unit 308, the speech recognition AP 170 transmits the received text data to the screen AP 158 via the WebSocket server 162.

  Further, the voice recognition AP 170 also has a function of switching ON / OFF of the voice recognition function. For example, when the voice recognition AP 170 receives an instruction to turn on the voice recognition function from the screen AP 158, the voice recognition function is turned on. Further, for example, when the voice recognition AP 170 receives an instruction to turn off the voice recognition function from the screen AP 158, the voice recognition function is turned off.

  The configuration example of the customer operation type terminal 10 according to the embodiment of the present invention has been described above with reference to FIGS. 2 and 3. Subsequently, a configuration example of the Web AP server 30 according to the embodiment of the present invention will be described.

<3. Configuration Example of Web AP Server 30>
FIG. 4 is a block diagram showing an example of a functional configuration of the WebAP server 30 according to the embodiment of the present invention. As shown in FIG. 4, the WebAP server 30 according to the embodiment of the present invention includes a communication unit 302, a business WebAP server 304, a speech recognition engine unit 308, an AI dialogue engine unit 312, a dialogue data storage unit 316, and a speech speech engine. A unit 320 is provided.

(1) Communication unit 302
The communication unit 302 has a function of communicating various information with the customer operation type terminal 10 via the network 14. For example, the communication unit 302 receives user input information from the customer operation type terminal 10 via the network 14. Further, the communication unit 302 transmits the content of the response determined by the Web AP server 30 to the customer operation type terminal 10 via the network 14.

(2) Business Web AP server 304
When the business Web AP server 304 receives a request from the customer operation type terminal 10, it has a function of returning a response to the request. For example, upon receiving a request for a Web page URL from the screen AP 158 of the customer operation type terminal 10, the business Web AP server 304 transmits the Web page URL corresponding to the request to the screen AP 158.

  In addition, the business WebAP server 304 receives a request from the customer operation type terminal 10, acquires a response to the request from the AI dialogue engine unit 312 and the speech voice engine unit 320 of the WebAP server 30, and returns the response to the customer operation type terminal 10. Also have. For example, when the business Web AP server 304 receives text data from the screen AP 158, the business Web AP server 304 transmits the text data to the AI dialogue engine unit 312, and acquires the next conversational sentence for the text data from the AI dialogue engine unit 312. Furthermore, the business Web AP server 304 transmits the next conversational sentence to the speech voice engine unit 320, acquires a speech file of the next conversational sentence from the speech speech engine unit 320, and transmits the acquired speech file to the screen AP158. At this time, if there is screen data that can be returned as a response, the business Web AP server 304 also transmits the screen data URL to the screen AP 158.

(3) Speech recognition engine unit 308
The speech recognition engine unit 308 has a function of converting speech information into text data. For example, when the voice recognition engine unit 308 receives voice information of the user from the voice recognition AP 170 of the customer operation type terminal 10, the voice recognition engine unit 308 converts the voice information into text data and transmits the converted text data to the voice recognition AP 170.

(4) AI dialogue engine unit 312
The AI dialogue engine unit 312 has a function of analyzing text data of voice information of the user and acquiring a conversation sentence (next conversation sentence) as a response to the user's speech. For example, when receiving the text data of the voice information of the user from the business Web AP server 304, the AI dialogue engine unit 312 analyzes the text data, acquires the next conversation sentence from the dialogue data storage unit 316, and acquires the acquired conversation sentence It is transmitted to the business Web AP server 304.

(5) Dialogue data storage unit 316
The dialogue data storage unit 316 has a function of storing the next speech sentence. For example, the dialogue data storage unit 316 stores, in advance, a conversational sentence created on the assumption of a conversation with the user. In addition, the dialogue data storage unit 316 may store a conversation sentence created and learned by the AI while the customer operation type terminal 10 repeats the conversation with the user.

(6) Utterance voice engine unit 320
The speech sound engine unit 320 has a function of generating a speech sound. For example, upon receiving the next conversational sentence from the business WebAP server 304, the speech voice engine unit 320 analyzes the next conversational sentence, generates a voice file of the next conversational sentence, and generates the generated voice file as the business WebAP server 304. Send to

  Heretofore, a functional configuration example of the WebAP server 30 according to the embodiment of the present invention has been described with reference to FIG. Subsequently, a display example of a screen according to the embodiment of the present invention will be described.

<4. Screen display example>
Hereinafter, a display example of the screen displayed on the operation display unit 136 according to the present embodiment will be described. First, an example of the layout of the screen will be described.

[4-1. Configuration example of screen layout]
FIG. 5 is an explanatory view showing a configuration example of a screen layout according to the embodiment of the present invention. As shown in FIG. 5, the layout of the screen displayed on the operation display unit 136 according to the embodiment of the present invention includes three layouts of layout 1, layout 2 and layout 3. The type of layout is not limited to three, and an arbitrary number of layouts may be prepared.

  Each layout is configured by any combination of a first display area, a second display area, a third display area, and a fourth display area. The first display area is an area for displaying an AI image (an interactive partner image) indicating an AI as an interactive partner of the user and an utterance content of the AI. The second display area is an area for displaying an image indicating the user and the content of the user's speech. The third display area is an area for displaying a service. The service displayed in the third display area is a service that can be selected by voice input and can be selected by touching the screen. The fourth display area is an area for displaying process information related to the process, service information, and the like. The detailed configuration of each display area will be described later.

  The layout 1 is configured by a combination of a first display area 404 and a second display area 408. The layout 1 is used for the layout of the screen where the user and the AI interact. For example, layout 1 is used for the layout of the initial screen. It is also used when there is no service to be displayed in the third display area.

  The layout 2 is configured by a combination of a first display area 504, a second display area 508, and a third display area 512. The layout 2 is used for the layout of the screen displaying the service. For example, the layout 2 is used for the layout of the screen displaying the service searched by the AI based on the user's utterance.

  The layout 3 is configured by a combination of a first display area 604 and a fourth display area 608. The layout 3 is used for the layout of the screen in which no interaction with the user occurs. For example, the layout 3 is used for the layout of a screen during settlement processing or ticket issuance processing.

  Although the first display area is used in each layout, the size in which the first display area is displayed is different between the case of being used in layout 1 and the case of being used in layout 2 and layout 3. For example, when used in layout 2 and layout 3, the first display area is displayed smaller than that used in layout 1. At this time, the operation display unit 136 also displays the AI image displayed in the first display area in a smaller size.

  In addition, the operation display unit 136 reduces the round frame of the AI image when the AI image is displayed small, but it is particularly determined whether the image of the person or the like indicating the AI in the round frame is also reduced. It is not limited. For example, as shown in the layout 2A of FIG. 6, the operation display unit 136 may reduce the size of the circle of the AI image 702 of layout 1 and the image of the person and display the AI image 704 as shown. Further, for example, as shown in the layout 2B of FIG. 6, the operation display unit 136 reduces only the round frame of the AI image 702 of layout 1 and displays the person's image like an AI image 706 while maintaining the same magnification. You may

  Moreover, the combination of each display area which comprises each layout is not limited to the combination mentioned above, Arbitrary combinations may be used.

  In the above, the configuration example of the screen layout according to the embodiment of the present invention has been described with reference to FIGS. 5 and 6. Subsequently, a screen using layout 1 will be described.

[4-2. Screen using layout 1]
Hereinafter, a configuration example of a screen using layout 1 and a display example will be described. The layout 1 is used for the layout of the screen where the user and the AI interact. The layout 1 arranges a balloon for displaying the uttered content of AI below the AI image and a balloon for displaying the uttered content of the user above the user image, whereby information indicating the content of the dialogue is displayed at the center of the screen. I am trying to Therefore, the user can reduce the movement amount of the line of sight when viewing the contents of the dialogue.

[4-2-1. Configuration Example of Screen Using Layout 1]
Below, the structural example of the screen using the layout 1 is demonstrated. FIG. 7 is a diagram showing an example in which a screen using layout 1 is displayed on the operation display unit 136. As shown in FIG. As shown in FIG. 7, in the screen displayed on the operation display unit 136, the layout 1 configured by the first display area 404 and the second display area 408 is used, and the second display area is It is arranged under the first display area. The first display area 404 is composed of an AI image 412 and a balloon 416. In the AI image 412, an AI image is displayed, and in the balloon 416, the content uttered by the AI is displayed. Further, the second display area 408 is configured of a user image 420 and a balloon 424. The user image 420 displays a user image, and the balloon 424 displays the contents uttered by the user. The speech contents converted into text data are described in text in each balloon. The speech contents converted into text data are similarly described in the same way in the balloon used in the other layouts.

4-2-2. Display example of screen using layout 1]
Below, the example of a display of the screen using layout 1 is explained. For example, the operation display unit 136 illustrated in FIG. 7 displays an initial screen when the customer operation type terminal 10 detects a user. After the initial screen is displayed, AI, for example, "Welcome. Welcome. We will show you how to transfer trains and sightseeing spots here. Search by destination, search by genre, suggestion of recommended plan. We can guide from three kinds of ". The contents uttered by the AI are displayed in a balloon 416. When the user utters “I want to search from a destination” in response to the AI utterance, the content uttered by the user is displayed in the balloon 424.

  FIG. 8 shows the continuation of the interaction between the AI and the user shown in FIG. When the AI recognizes the utterance “I want to search from the destination” shown in FIG. 7, the AI utters “I am searching from the destination. The uttered content is displayed on the balloon 416. Furthermore, when the user utters “I want to go to the AAA station” in response to the AI utterance, the uttered content is displayed in the balloon 424.

  Hereinabove, the screen using the layout 1 according to the embodiment of the present invention has been described with reference to FIGS. 7 and 8. Subsequently, the screen using the layout 2 will be described.

[4-3. Screen using layout 2]
Hereinafter, a configuration example and a display example of a screen using the layout 2 will be described. The layout 2 is used for the layout of the screen displaying the service searched by the AI based on the user's speech. Layout 2 arranges a speech balloon displaying AI's speech content below the AI image and a speech balloon displaying the user's speech content above the user image, and further, places the service between the AI speech balloon and the user's speech balloon It arranges, and the information showing the service and the contents of dialogue is displayed in the center of the screen. Therefore, when the user operates the operation screen of the service, the user can visually recognize the contents of the dialog without much moving the line of sight from the operation screen. In addition, the user feels that the AI image and the balloon are displayed above the service information, and the user's image and the balloon are displayed below the service, so that the user feels that the user interacts with the AI across the table. And you can get a sense of interacting more naturally.

[4-3-1. Example of Screen Configuration Using Layout 2]
Below, the structural example of the screen using the layout 2 is demonstrated. FIG. 9 is a diagram showing an example in which a screen using layout 2 is displayed on the operation display unit 136. As shown in FIG. As shown in FIG. 9, in the screen displayed on the operation display unit 136, the layout 2 constituted by the first display area 504, the second display area 508, and the third display area 512 is used. There is. The third display area is disposed between the first display area and the second display area, the first display area is disposed above the third display area, and the second display area is It is disposed below the third display area. The first display area 504 is composed of an AI image 516 and a balloon 520. In the AI image 516, an AI image is displayed, and in the balloon 520, the content uttered by the AI is displayed. Further, the second display area 508 is configured by the user image 524 and the balloon 528. The user image 524 displays the user image, and the balloon 528 displays the content uttered by the user.

  In addition, the third display area 512 is configured by a search result 532 and a detail button 536. In the search result 532, screen data obtained by accessing the screen data URL is displayed as a search result. When there are a plurality of search results, the third display area 512 may display a plurality of search results, such as the search results 532A, the search results 532B, and the search results 532C shown in FIG. The details button 536 is a button for displaying the details of the search result 532. As shown in FIG. 9, one detail button 536 is displayed for one search result 532. For example, when the user presses the detail button 536A, the screen of the operation display unit 136 transitions to a screen displaying the details of the search result 532A. Also, for example, when the user presses the detail button 536B, the screen of the operation display unit 136 transitions to a screen that displays the details of the search result 532B. Also, for example, when the user utters "Please display the details of route 3" and the AI recognizes the user's utterance, the screen of the operation display unit 136 transitions to a screen displaying the details of the search result 532C. Do. The contents uttered by the user are displayed in the balloon 528.

[4-3-2. Display example of screen using layout 2]
Below, the example of a display of the screen using layout 2 is explained. For example, the operation display unit 136 shown in FIG. 9 displays the result of the search of the customer operation type terminal 10 based on the speech of the user in FIG. After displaying the search results, for example, AI utters, "To AAA station. You can go by the following three routes." The contents uttered by the AI are displayed in a balloon 520. When the user utters "Please route 1" in response to the AI utterance, the content uttered by the user is displayed in the balloon 528. When the AI recognizes the user's utterance "Please give me Route 1", the operation display unit 136 displays a detail screen of the search result. The operation display unit 136 can display the detail screen of the search result even if the user presses the detail button 536.

  FIG. 10 shows the continuation of the interaction between the AI and the user shown in FIG. When the AI recognizes the utterance "Please make me a request at route 1" shown in FIG. 9, the AI utters, "You are able to buy a ticket for this route. You can buy a ticket for this route." The uttered content is displayed on a balloon 520 shown. The operation display unit 136 also displays the details of the route 1 selected by the user in the search result details 540. Furthermore, when the user utters “I will purchase” in response to the AI utterance, the uttered content is displayed in the balloon 528. When the AI recognizes the user's "purchase" utterance, the operation display unit 136 displays a screen for performing a ticket settlement process. The operation display unit 136 can display a screen for performing a ticket settlement process even if the user presses the purchase button in the third display area.

  The screen using the layout 2 according to the embodiment of the present invention has been described above with reference to FIGS. 9 and 10. Subsequently, a screen using the layout 3 will be described.

[4-4. Screen using layout 3]
Hereinafter, a configuration example of a screen using the layout 3 and a display example will be described. The layout 3 is used for layout of a screen during settlement processing in which no dialog with the user occurs, or in ticket ticket processing. The layout 3 can display service information such as a weather forecast. Thus, the user can spend time without payment processing or ticket processing being completed.

[4-4-1. Configuration Example of Screen Using Layout 3]
Below, the structural example of the screen using the layout 3 is demonstrated. FIG. 11 is a diagram showing an example in which a screen using layout 3 is displayed on the operation display unit 136. As shown in FIG. As shown in FIG. 11, in the screen displayed on the operation display unit 136, Layout 3 configured by the first display area 604 and the fourth display area 608 is used, and the fourth display area is It is arranged under the first display area. The first display area 604 is composed of an AI image 612 and a balloon 616. In the AI image 612, an AI image is displayed, and in the balloon 616, the content uttered by the AI is displayed. In addition, the fourth display area 608 is configured by processing information 620 and processing information 624. In the processing information 620, an image regarding insertion of cash is displayed as an image regarding payment processing, and in the processing information 624, an image regarding insertion of a card is displayed. In addition, the image displayed as process information is not limited to the image regarding insertion of cash, and insertion of a card, Arbitrary images may be displayed. In addition, an animation image may be displayed as the processing information. Further, in FIG. 11, two pieces of processing information are displayed in the fourth display area, but the number of pieces of processing information to be displayed in the fourth display area is not limited, and an arbitrary number of pieces of processing information are displayed Good.

  For example, the operation display unit 136 shown in FIG. 11 displays a screen for the customer operation type terminal 10 to perform payment processing based on the speech of the user in FIG. After displaying the payment processing screen, for example, the AI utters "Please insert cash or card." The content uttered by the AI is displayed in a balloon 616. When the user inserts cash or a card into the payment medium unit 140 for the AI utterance, the AI executes payment processing. When the settlement process is completed, the customer operation type terminal 10 executes a process of issuing a ticket.

[4-4-2. Display example of screen using layout 3]
Below, the example of a display of the screen using layout 3 is explained. For example, the operation display unit 136 illustrated in FIG. 12 displays a screen for indicating that ticketing processing of a ticket is being performed after settlement processing. After the ticket issue processing screen is displayed, for example, the AI utters "We are issuing tickets." The content uttered by the AI is displayed in a balloon 616. Further, the processing information 628 displays a large text indicating that the ticket is being issued, saying "The ticket is being issued. Please wait for a while." Further, the processing information 632 displays service information such as a weather forecast so that the user does not get bored while waiting for a ticket to be issued. When the ticket issue process is completed, the operation display unit 136 displays a screen that warns of taking a ticket, a package, and a card.

  For example, the operation display unit 136 illustrated in FIG. 13 displays a screen for paying attention to the removal of a ticket, a tray, and a card. After the caution screen is displayed, for example, the AI utters, "Please receive a ticket. Please note that you forgot to take a deposit or a card." The content uttered by the AI is displayed in a balloon 616. Further, the processing information 636 displays an image indicating receipt of the ticket. Further, the processing information 640 displays an image indicating receipt of a cash or card. When the user receives a ticket, the operation display unit 136 displays an initial screen.

  In the above, the screen using the layout 3 according to the embodiment of the present invention has been described with reference to FIGS. Subsequently, an operation example according to the embodiment of the present invention will be described.

<5. Operation example>
Below, the operation example of the customer operation type terminal 10 which concerns on embodiment of this invention, and the WebAP server 30 is demonstrated. First, an operation example at the time of user detection will be described.

[5-1. Operation example at the time of user detection]
Hereinafter, an operation example of the customer operation type terminal 10 and the WebAP server 30 at the time of user detection will be described with reference to FIGS. 14 and 15. FIG. 14 is a sequence diagram showing an operation example of the customer operation type terminal 10 at the time of user detection. FIG. 15 is a sequence diagram showing an operation example of the Web AP server 30 at the time of user detection.

  First, as shown in FIG. 14, when detecting a person, the camera AP 154 of the customer operation type terminal 10 notifies the voice recognition AP 170 that the person is detected (step S1000). When the voice recognition AP 170 receives a notification from the camera AP that it has detected a person, the voice recognition AP 170 connects to the WebSocket server 162, and stands by in a state where voice input from the user can be received (step S1004). Also, the screen AP 158 acquires the web page URL from the web AP server 30, and displays the web page as an initial screen (step S1008).

  Here, the details of step S1008 will be described with reference to FIG. As shown in FIG. 15, the screen AP 158 of the customer operation type terminal 10 requests the Web page URL from the business Web AP server 304 of the Web AP server 30 (step S1010). The business Web AP server 304 having received the request transmits the Web page URL to the screen AP 158 (step S1012). The screen AP 158 that has received the web page URL accesses the web page URL and displays the web page as an initial screen (step S1014).

  After the initial screen is displayed, as shown in FIG. 14, the screen AP 158 is connected to the WebSocket server 162, and stands by in a state where the screen operation input from the user can be received (step S1018). Also, the screen AP 158 notifies the voice recognition AP 170 that it is in the voice reception waiting state via the WebSocket server 162 (steps S1022 and S1026). The voice recognition AP 170 that has received the notification notifies the Web AP server 30 that it is in a state of waiting for voice reception (step S1030).

  Here, the details of step S1030 will be described with reference to FIG. As shown in FIG. 15, the screen AP 158 of the customer operation type terminal 10 notifies the voice recognition engine unit 308 of the Web AP server 30 that voice recognition is to be started (step S1032). The voice recognition engine unit 308 that has received the notification stands by in a state where it can receive voice.

  In the above, the operation example of the customer operation type terminal 10 and the WebAP server 30 at the time of user detection has been described with reference to FIGS. 14 and 15. Subsequently, an operation example at the time of acquiring voice information will be described.

5-2. Example of operation when acquiring voice information]
Hereinafter, an operation example of the customer operation type terminal 10 and the WebAP server 30 at the time of acquiring voice information will be described with reference to FIGS. 16 and 17. When the microphone 128 acquires voice information of the user, the customer operation type terminal 10 determines a response based on the acquired voice information, and executes the determined response. FIG. 16 is a sequence diagram showing an operation example of the customer operation type terminal 10 at the time of voice information acquisition. FIG. 17 is a sequence diagram showing an operation example of the WebAP server 30 at the time of acquiring voice information.

  First, as shown in FIG. 16, when the microphone 128 of the customer operation type terminal 10 acquires voice information by the user's speech (step S1100), the microphone 128 outputs the acquired voice to the noise removal module 166 (step S1104). The noise removal module 166 that has acquired the voice performs processing to remove noise from the voice (step S1106), and outputs the voice from which the noise has been removed to the voice recognition AP 170 (step S1108). The voice recognition AP 170 that has acquired the voice transmits the voice to the Web AP server 30, and receives the text data of the voice from the Web AP server 30 (step S1112).

  Here, the details of step S1112 will be described with reference to FIG. As shown in FIG. 17, the voice recognition AP 170 of the customer operation type terminal 10 transmits a voice to the voice recognition engine unit 308 of the WebAP server 30 (step S1114). The voice recognition engine unit 308 having received the voice performs processing of converting the voice into text data (step S1116), and transmits the converted text data to the voice recognition AP 170 (step S1118).

  After receiving the text data, as shown in FIG. 16, the speech recognition AP 170 transmits the text data to the screen AP 158 via the WebSocket server 162 (steps S1122 and S1126). The screen AP 158 that has received the text data displays the text data on the screen (step S1130). After displaying the text data on the screen, the screen AP 158 receives the voice file of the next conversational sentence and the screen data URL from the Web AP server 30, and responds to the user's speech (step S1134).

  Here, the details of step S1134 will be described with reference to FIG. As shown in FIG. 17, the screen AP 158 of the customer operation type terminal 10 transmits text data to the business Web AP server 304 of the Web AP server 30 (step S1136). The business Web AP server 304 having received the text data transmits the text data to the AI dialogue engine unit 312 in order to acquire the next conversational sentence (step S1138). The AI dialogue engine unit 312 having received the text data acquires the next conversational sentence serving as a response to the text data from the dialogue data storage unit 316 (step S1140), and transmits the next conversational sentence to the business Web AP server 304 (step S1142).

  The business Web AP server 304 that has received the next conversational sentence requests the speech voice engine unit 320 for a speech file of the next conversational sentence (step S1144). The speech voice engine unit 320, which receives the request from the business Web AP server 304, transmits the speech file of the next conversation sentence to the business Web AP server 304 (step S1146). The business WebAP server 304 that has received the voice file of the next conversational sentence transmits the voice file of the next conversational sentence and the screen data URL for displaying the service as a response to the user's speech to the screen AP 158 (step S1148). ).

  The screen AP 158 displays the screen data acquired by accessing the text data and the screen data URL (step S1150). Also, the screen AP 158 reproduces the audio file and conveys the content of the response to the user by audio (step S1152).

  In addition, the customer operation type terminal 10 can switch ON and OFF of a speech recognition function. As shown in FIG. 16, when the voice recognition function is turned on, the screen AP 158 transmits an instruction to turn on the voice recognition function to the voice recognition AP 170 (step S1156). The voice recognition AP 170 receives the instruction, turns on the voice recognition function, and transmits to the screen AP 158 that the voice recognition function has been turned on as a response to the instruction (step S1160). When the speech recognition function is to be turned off, the screen AP 158 transmits an instruction to turn off the speech recognition function to the speech recognition AP 170 (step S 1164). The voice recognition AP 170 receives the instruction, turns off the voice recognition function, and transmits to the screen AP 158 that the voice recognition function has been turned off as a response to the command (step S1168).

  When the voice recognition function is ON and the user speaks to the customer operation type terminal 10 during ticket issue processing, the control unit 150 hides the service information displayed on the operation display unit 136, and The second display area may be redisplayed.

  In the above, the operation example of the customer operation type terminal 10 and the WebAP server 30 at the time of voice information acquisition has been described with reference to FIGS. 16 and 17. Subsequently, an operation example at the time of acquiring screen operation information will be described.

[5-3. Example of operation when acquiring screen operation information]
Hereinafter, an operation example of the customer operation type terminal 10 and the WebAP server 30 at the time of acquiring the screen operation information will be described with reference to FIGS. 18 and 19. When the operation display unit 136 acquires the screen operation information of the user, the customer operation type terminal 10 determines a response based on the acquired screen operation information, and executes the determined response. FIG. 18 is a sequence diagram showing an operation example of the customer operation type terminal 10 at the time of acquiring screen operation information. FIG. 19 is a sequence diagram showing an operation example of the Web AP server 30 at the time of acquiring screen operation information.

  First, as shown in FIG. 18, when the operation display unit 136 of the customer operation type terminal 10 acquires the information of pressing the button by the screen operation of the user (step S 1200), the acquired button information is transmitted to the screen AP 158 ( Step S1204). The screen AP 158 which has acquired the button information transmits the button information to the Web AP server 30, receives the voice file of the next conversation sentence and the screen data URL from the Web AP server 30, and responds to the user's button operation (step S1208). .

  Here, the details of step S1208 will be described with reference to FIG. As shown in FIG. 19, the screen AP 158 of the customer operation type terminal 10 transmits button information to the business Web AP server 304 of the Web AP server 30 (step S1210). The business Web AP server 304 that has received the button information requests the speech voice engine unit 320 for a speech file of the next conversational sentence (step S1212). The speech voice engine unit 320, which has received the request from the business Web AP server 304, transmits the speech file of the next conversation sentence to the business Web AP server 304 (step S1214). The business Web AP server 304 that has received the voice file of the next conversational sentence transmits the voice file of the next conversational sentence and the screen data URL for displaying the service as a response to the user's speech to the screen AP 158 (step S1216) ).

  The screen AP 158 displays the screen data acquired by accessing the text data and the screen data URL (step S1218). Also, the screen AP 158 reproduces the audio file and conveys the content of the response to the user by audio (step S1220).

  In the above, the operation example of the customer operation type terminal 10 and the WebAP server 30 at the time of acquiring the screen operation information has been described with reference to FIGS. 18 and 19. Subsequently, the effects of the present invention will be described.

<6. Effect>
The control unit 150 of the customer operation type terminal 10 can change the layout of the screen according to the contents of the response to the user's speech, and displays the contents of the dialogue and the contents of the response in the center of the screen. Can. Also, the control unit 150 of the customer operation type terminal 10 can display the contents of the response to the screen operation input of the user in the same manner. Therefore, the customer operation type terminal 10 can reduce the movement amount of the user's gaze. Further, the customer operation type terminal 10 can give the user a feeling of interacting more naturally by displaying the AI image and the balloon above the user's image and the balloon. Further, the control unit 150 can display the display area of the balloon displaying the content uttered by the AI and the display area of the balloon displaying the content uttered by the user with a predetermined length in the screen width direction. . Furthermore, the control unit 150 may display the image of the AI above the balloon displaying the content uttered by the AI, and may display the image of the user below the balloon displaying the content uttered by the user. it can. Therefore, the control unit 150 can move each balloon to the center of the screen and display it. Furthermore, the control unit 150 displays the balloon in a balloon by increasing the width of the balloon and increasing the amount of information (for example, the number of characters of text) that can be displayed on one line in the balloon to reduce the total number of lines. Can be displayed closer to the center of the screen.

  The embodiments of the present invention have been described above with reference to FIGS. Then, the modification concerning an embodiment of the present invention is explained.

<7. Modified example>
In the following, some variants of the embodiments of the invention will be described. In addition, each modification described below may be independently applied to the embodiment of the present invention, or may be applied to the embodiment of the present invention in combination. In addition, each modification may be applied instead of the configuration described in the embodiment of the present invention, or may be additionally applied to the configuration described in the embodiment of the present invention.

(1) First Modification In the above embodiment, the control unit 150 displays an initial screen using the layout 1 shown in FIG. 5 on the operation display unit 136, but the present invention is not limited to the existing AP. When applied, the control unit 150 may display an initial screen using the layout 2 shown in FIG. 5 on the operation display unit 136. By using the layout 2 in the initial screen, the operation display unit 136 can display an interface for operating the screen of the existing AP in the third display area of the layout 2. When the operation display unit 136 displays the interface for screen operation of the existing AP on the initial screen, the user can input by the screen operation on the initial screen. In addition, even when the interface for screen operation of the existing AP is displayed on the initial screen, the customer operation type terminal 10 is also in a state where the operation by voice input is also possible. For example, as shown in FIG. 20, the operation display unit 136 displays the interface of the existing AP in the third display area 800. When the user presses the button displayed in the third display area, the operation display unit 136 transitions to the next screen shown in FIG.

  Also on the next screen shown in FIG. 21, the operation display unit 136 displays the interface of the existing AP in the third display area 804. By enabling screen operation with the interface of the existing AP from the initial screen, a user who is used to screen operation with the interface of the existing AP can use the customer operation type terminal 10 efficiently without stress. As described above, the control unit 150 can improve the visibility of the operation display unit 136 by enabling the screen operation by the interface of the existing AP on the initial screen.

(2) Second Modified Example In the above-described embodiment, an example in which the control unit 150 displays the layout 2 shown in FIG. 5 on the operation display unit 136 as the layout of the screen capable of screen operation has been described. A display may be made to clarify the place where the screen operation of 2 is possible. For example, as shown in FIG. 22, the control unit 150 displays a finger icon 900 indicating that the detail button can be pressed near the detail button. Specifically, the control unit 150 displays the icon 900A on the detail button of the route 1, the icon 900B on the detail button of the route 2, and the icon 900C on the detail button of the route 3. As described above, the control unit 150 can improve the operability of the operation display unit 136 by performing the display for clarifying the portion where the screen can be operated.

  Note that the display method of the icon 900 is not particularly limited, and may be displayed by any method. For example, the control unit 150 may display the icon 900 simultaneously with the screen display of the layout 2 and set the icon 900 to disappear in several seconds. Also, the control unit 150 may add motion to the icon 900 by animation. Here, the motion by animation is, for example, blinking. By blinking the finger icon 900, the control unit 150 can clearly display the portion where the screen can be operated in a way that is easy for the user to understand.

(3) Third Modification In the above embodiment, the control unit 150 has described an example of transitioning the screen when the user selects an item. However, after highlighting the item selected by the user, the transition of the screen is performed. May be For example, as shown in Highlighted Display 1 of FIG. 23, the control unit 150 may highlight the search result 920 selected by the user by enclosing it in a circle. Also, for example, as shown in highlight 2 in FIG. 23, the control unit 150 highlights the search result 922 selected by the user by increasing the lightness of colors of items other than the search result 922 selected by the user. You may Also, for example, as shown in highlight 3 in FIG. 23, the control unit 150 may highlight the search result 924 selected by the user by blinking. As described above, the control unit 150 can improve the visibility of the operation display unit 136 by highlighting the item selected by the user.

(4) Fourth Modification In the above-described embodiment, the control unit 150 has described the example in which one user image and one balloon are displayed in the second display area. When a user is detected, the control unit 150 may display a plurality of user images and a plurality of speech bubbles in the second display area. For example, when the camera 120 and the microphone 128 detect two users, as shown in FIG. 24, the user image 930 and the balloon 932 of the first user, the user image 934 of the second user and the balloon 938 are displayed. May be As described above, the control unit 150 can improve the visibility of the operation display unit 136 by displaying the user images and the balloon in the second display area by the number of users detected by the customer operation type terminal 10. it can.

(5) Fifth Modification In the above embodiment, the control unit 150 vertically displays the screen on the operation display unit 136. However, according to the direction of the operation display unit 136, the control unit 150 The orientation of the screen and the layout of the screen may be changed. For example, when the operation display unit 136 is in portrait orientation, the control unit 150 displays the orientation of the screen and the layout of the screen described in the above embodiment on the operation display unit 136. Also, for example, when the direction of the operation display unit 136 is horizontal, the control unit 150 displays the screen in the horizontal direction as shown in FIG. Further, the control unit 150 may display each display area on the operation display unit 136 in the order of the first display area 940, the second display area 944, and the third display area 942 along the longitudinal direction of the screen. . As described above, the control unit 150 can improve the visibility of the operation display unit 136 by changing the screen display direction and the screen layout according to the direction of the operation display unit 136.

(6) Sixth Modification In the above-described embodiment, the control unit 150 has described the example of returning the display of the screen to the initial screen when the camera 120 no longer detects the user after issuing the ticket. 150 may display information about the ticket after issuing the ticket. For example, the control unit 150 displays the contents of the issued ticket on the screen, reads out the contents of the ticket and outputs a voice to convey the contents of the ticket to the user so that the user can confirm the contents of the ticket. . When reading out the contents of the ticket, the control unit 150 may make the user understand the read out part by making the read out part of the contents of the ticket displayed on the screen into red. As described above, the control unit 150 can improve the visibility of the operation display unit 136 by displaying the contents of the issued ticket.

(7) Seventh Modification In the above-described embodiment, an example has been described in which the control unit 150 displays service information in the fourth display area of the layout 3 during ticket issue processing of a ticket. When the user confirms the service information, the ticket issue processing speed may be reduced. For example, when detecting that the user operates the screen while displaying the service information, the control unit 150 recognizes that the user is checking the service information, and slows the ticket issue processing speed. As the control unit 150 slows down the ticket issue processing speed, the user can secure a time for checking the service information. As described above, the control unit 150 can improve the visibility of the operation display unit 136 by decreasing the ticket issue processing speed.

  In addition, when the user confirms the service information, the control unit 150 may cause the AI to react according to the content of the information confirmed by the user. For example, when the user confirms the weather forecast as the service information, the control unit 150 may cause the AI to utter, “Are you worried about the weather? Do you want to display a more detailed weather forecast?”. When the user responds that he wants the weather forecast details to be displayed, the control unit 150 displays the weather forecast details on the operation display unit 136.

(8) Eighth Modification In the above embodiment, the screen configuration and operation example of the customer operation type terminal 10 have been described. However, when using an information display device, the control unit 150 uses the time to display service information displayed on the screen. It may be changed with the passage of time. Further, the control unit 150 may change the operation screen of the existing AP displayed in the third display area of the initial screen using the layout 2 with the passage of time. For example, the control unit 150 can improve the visibility of the operation display unit 136 by changing the size and color of characters displayed in the third display area 800 shown in FIG. 20 with the passage of time. . When using the information display device, the control unit 150 may create bar code information corresponding to the operation content input by the user and may display the bar code information on the operation display unit 136. The user can cause the portable terminal to acquire the operation content input by the information display device by causing the portable terminal owned by the user to read the barcode information (or the display screen including the barcode information).

  Hereinabove, the modification according to the embodiment of the present invention has been described. Subsequently, a hardware configuration according to an embodiment of the present invention will be described.

<8. Hardware configuration>
The information processing relating to the provision of the above-described service is realized by the cooperation of software and hardware of the customer-operation type terminal 10 described below.

  FIG. 26 is a block diagram showing an example of the hardware configuration of the customer operation type terminal 10. As shown in FIG. The customer operation type terminal 10 includes a central processing unit (CPU) 101, a read only memory (ROM) 103, and a random access memory (RAM) 105. The customer operation type terminal 10 further includes an input unit 107, a display unit 109, an output unit 111, a storage unit 113, and a network interface 115.

  The CPU 101 functions as an arithmetic processing unit and a control unit, and controls the overall operation in the customer-operated terminal 10 according to various programs. Also, the CPU 101 may be a microprocessor. The ROM 103 stores programs used by the CPU 101, calculation parameters, and the like. The RAM 105 temporarily stores a program used in the execution of the CPU 101, parameters which appropriately change in the execution, and the like. These are mutually connected by a host bus configured of a CPU bus and the like. The CPU 101, the ROM 103, and the RAM 105 can realize the function of the control unit 150 described with reference to FIG. 3 in cooperation with software.

  The input unit 107 generates an input signal based on an input by a user such as a mouse, a keyboard, a touch panel, a button, a microphone, a sensor, a switch, and a lever, and an input by the user. It comprises a control circuit and the like. The user of the customer operation type terminal 10 can input various data to the customer operation type terminal 10 and instruct processing operations by operating the input unit 107.

  The display unit 109 includes, for example, a display device such as a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, a projector device, an OLED (Organic Light Emitting Diode) device, and a lamp. Further, the output unit 111 includes an audio output device such as a speaker and headphones.

  The storage unit 113 is a device for storing data. The storage unit 113 may include a storage medium, a recording device that records data in the storage medium, a reading device that reads data from the storage medium, and a deletion device that deletes data recorded in the storage medium. The storage unit 113 is configured of, for example, a hard disk drive (HDD) or a solid storage drive (SSD), a memory having an equivalent function, or the like. The storage unit 113 drives the storage and stores programs executed by the CPU 101 and various data.

  The network interface 115 is, for example, a communication interface configured of a communication device or the like for connecting to a network. Such a communication interface may be, for example, a short distance wireless communication interface such as Bluetooth (registered trademark) or ZigBee (registered trademark), a wireless local area network (LAN), Wi-Fi (registered trademark), or a mobile communication network (LTE, It is a communication interface such as 3G). The network interface 115 may also be a wired communication device that performs wired communication.

  The example of the hardware configuration of the customer operation type terminal 10 has been described above with reference to FIG.

<9. End>
As described above, according to the embodiment of the present invention, the customer operation type terminal 10 acquires voice information input by the user by speech, and based on text data converted from the speech information, responds to the user's speech It can be decided. In addition, the operation display unit 136 of the customer operation type terminal 10 includes a first display area for displaying the AI image and the utterance content of the AI, a second display area for displaying the utterance content of the user, and the customer operation type terminal 10 A screen configured of a third display area for displaying a service provided to the user can be displayed. In addition, the control unit 150 of the customer operation type terminal 10 arranges the utterance content of AI between the AI image and the service, and arranges the utterance content of the user between the user image and the service according to the content of the response. Can. As described above, the customer operation type terminal 10 can reduce the moving amount of the user's line of sight by displaying the information necessary for the user close to the center of the screen and improving the visibility of the screen. It is possible.

  As described above, it is possible to provide a new and improved information processing apparatus, an information processing method, and a program capable of improving the visibility of a screen.

<10. Supplement>
Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is obvious that those skilled in the art to which the present invention belongs can conceive of various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also fall within the technical scope of the present invention.

  The steps in the processing of the customer-operation-type terminal 10 and the WebAP server 30 in the present specification do not necessarily have to be processed chronologically in the order described in the sequence diagram. For example, each step in the processing of the customer operation type terminal 10 may be processed in an order different from the order described in the sequence diagram, or may be processed in parallel. Also, additional processing steps may be employed and some processing steps may be omitted.

  Further, the customer operation type terminal 10 and the WebAP server 30 described in the present specification may be realized as a single device, or some or all may be realized as separate devices. For example, as a customer operation type terminal 10 shown in FIG. 3 and a WebAP server 30 shown in FIG. 4, it may be realized as a single device having functions separately. Further, for example, all functions of the WebAP server 30 illustrated in FIG. 4 may be provided in the customer operation type terminal 10, and may be realized only by the customer operation type terminal 10. Also, some of the functions may be provided in an apparatus such as a server connected via a network or the like.

  In addition, in order to cause hardware such as the CPU, ROM, and RAM incorporated in the customer operation type terminal 10 and the WebAP server 30 to exhibit functions equivalent to the respective configurations of the customer operation type terminal 10 and the WebAP server 30 described above. Computer programs can also be created. There is also provided a storage medium storing the computer program.

DESCRIPTION OF SYMBOLS 10 Customer operation type terminal 30 WebAP server 120 Camera 124 Speaker 128 Microphone 132 Communication unit 136 Operation display unit 140 Payment medium unit 144 Ticket issue unit 150 Control unit 154 Camera AP
158 screen AP
162 WebSocket server 166 noise reduction module 170 speech recognition AP
302 communication unit 304 business WebAP server 308 speech recognition engine unit 312 AI dialogue engine unit 316 dialogue data storage unit 320 speech utterance engine unit

Claims (16)

A voice information acquisition unit that obtains voice information input by a user;
A converter for converting the voice information into text data;
A response unit that determines a response based on the text data and executes the response;
A first display area for displaying a dialog partner image showing the user's dialog partner and the dialog contents outputted by the dialog partner by the response; a second display area for displaying the text data; and the response The third display area for displaying the service to be displayed, the third display area being disposed between the first display area and the second display area, and the dialogue contents being the dialogue A control unit configured to control display processing according to the contents of the response so as to be disposed in text description on the side of the third display area with respect to the partner image;
An information processing apparatus comprising:   The information according to claim 1, wherein the control unit arranges the first display area above the third display area, and arranges the second display area below the third display area. Processing unit. The information processing apparatus is capable of detecting the user;
A screen operation information acquisition unit that acquires screen operation information input by the user's screen operation;
And further
The information processing apparatus according to claim 1, wherein the control unit stands by in a state where the voice information and the screen operation information can be received when the detection apparatus detects the user.   The information processing apparatus according to claim 3, wherein, when the screen operation information acquisition unit acquires the screen operation information, the response unit determines the response based on the screen operation information and executes the response.   The information processing apparatus according to claim 1, wherein the control unit displays, in the third display area, a service capable of selection by voice input and selection by touch of a screen.   The control unit adjusts the size of the display area of the conversation partner image according to the size of the first display area when the size of the first display area changes at the time of screen switching. The information processing apparatus according to claim 1.   The control unit hides the third display area when there is no service to be displayed in the third display area, and displays the display area above the second display area as the first display area. The information processing apparatus according to claim 1, wherein:   The information processing apparatus according to claim 1, wherein the control unit displays the dialogue content and the text data in a balloon.   The said control part displays the user image which shows the said user on a said 2nd display area, The said text data are displayed between the said user image and a said 3rd display area. Information processing device.   The control unit hides the second display area and the third display area when the response unit is executing a process that does not require interaction with the user, and the control unit is configured to hide the second display area and the third display area. The information processing apparatus according to claim 1, wherein a lower display area is a fourth display area. The information processing apparatus is a payment processing unit that controls payment processing.
A ticketing processing unit that controls ticketing processing;
And further
The information processing apparatus according to claim 10, wherein the control unit displays service information in the fourth display area when the payment processing unit or the ticket issuance processing unit is processing.   The information processing apparatus according to claim 1, wherein the control unit is capable of switching a voice recognition function on or off.   The control unit hides the service information and displays the second display area when the voice recognition function is on and the user speaks while the fourth display area is displayed. The information processing apparatus according to claim 11.   The information processing apparatus according to claim 1, wherein the service is a seat reservation service. Obtaining voice information input by the user;
Converting the voice information into text data;
Determining a response based on the text data and executing the response;
A first display area for displaying a dialog partner image showing the user's dialog partner and the dialog contents outputted by the dialog partner by the response; a second display area for displaying the text data; and the response The third display area for displaying the service to be displayed, the third display area being disposed between the first display area and the second display area, and the dialogue contents being the dialogue Controlling the display processing according to the contents of the response so as to arrange in text description on the side of the third display area with respect to the other party image;
An information processing method to be executed by a processor, including: Computer,
A voice information acquisition unit that obtains voice information input by a user;
A converter for converting the voice information into text data;
A response unit that determines a response based on the text data and executes the response;
A first display area for displaying a dialog partner image showing the user's dialog partner and the dialog contents outputted by the dialog partner by the response; a second display area for displaying the text data; and the response The third display area for displaying the service to be displayed, the third display area being disposed between the first display area and the second display area, and the dialogue contents being the dialogue A control unit configured to control display processing according to the contents of the response so as to be disposed in text description on the side of the third display area with respect to the partner image;
A program to function as.

Images