JP2018129068A - Information processing device, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device, information processing method, and program that automatically generate a message to be notified to a user's terminal device on the basis of notification information.SOLUTION: An information processing device includes a learning unit, a generation unit, and a notification unit. The learning unit learns a reinforcement learning model for message generation for a user by reinforcement learning on the basis of the history of responses of the user to messages notified in the past. The generation unit generates a message to be notified to a terminal device of the user by using the reinforcement learning model for message generation. The notification unit notifies the terminal device of the message generated by the generation unit on the basis of information for providing a push notification to the user.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

  2. Description of the Related Art Conventionally, an information processing apparatus that controls insertion behavior of a dynamic dialog node using reinforcement learning when a dynamic dialog node is inserted between fixed dialog nodes transmitted to a user terminal device is known ( Patent Document 1).

JP2015-125198A

  In the information processing apparatus, a dynamic dialog node having a high similarity probability with respect to a sentence of a fixed dialog node is selected and inserted as the scenario progresses.

  In the information processing apparatus, a dynamic dialog node is inserted along with the progress of the scenario, but a message such as a dialog node may be notified from the information processing apparatus to the user terminal device based on the notification information. In other words, push-type notification may be performed from the information processing apparatus to the user terminal device.

  In the case of performing push-type notification, there is no scenario that exists in the information processing apparatus, and there is no message corresponding to the fixed dialog node, so it is difficult to generate a message to be notified to the user terminal device.

  The present application has been made in view of the above, and an object thereof is to provide an information processing apparatus, an information processing method, and a program that automatically generate a message to be notified to a user terminal device based on notification information. And

  An information processing apparatus according to the present application includes a learning unit, a generation unit, and a notification unit. A learning part learns the reinforcement learning model for message generation with respect to the said user by reinforcement learning based on the response history of the user with respect to the message notified in the past. A generation part produces | generates the message notified to a user's terminal device using the reinforcement learning model for message generation. The notification unit notifies the terminal device of the message generated by the generation unit based on information for performing push notification to the user.

  According to one aspect of the embodiment, it is possible to provide an information processing apparatus, an information processing method, and a program that automatically generate a message to be notified to a user terminal device based on notification information.

FIG. 1 is an explanatory diagram of information processing according to the embodiment. FIG. 2 is a diagram illustrating a configuration example of the information processing system. FIG. 3 is a diagram illustrating a configuration example of the information processing apparatus. FIG. 4 is a diagram illustrating an example of a score. FIG. 5 is a flowchart illustrating an example of a notification message generation process according to the embodiment. FIG. 6 is a hardware configuration diagram illustrating an example of a computer that implements the functions of the information processing apparatus.

  Hereinafter, an information processing apparatus, an information processing method, and a form for implementing a program (hereinafter referred to as “embodiment”) according to the present application will be described in detail with reference to the drawings. Note that the information processing apparatus, the information processing method, and the program according to the present application are not limited by this embodiment.

[1. Information processing]
An example of information processing according to the embodiment will be described. FIG. 1 is an explanatory diagram of information processing according to the embodiment. Here, information processing is executed by the information processing apparatus 1.

  The information processing apparatus 1 selects a keyword using a reinforcement learning model (a message generation reinforcement learning model, a learning model) (step S1).

  The reinforcement learning model, which will be described in detail later, is a keyword having a large integrated reward value (hereinafter referred to as a score) when generating a message (hereinafter referred to as a notification message) that is pushed to the terminal device 2 of the user. Is a model to select.

  The reinforcement learning model is learned so that a keyword score related to a notification message with a good user response is increased with respect to the notification message transmitted to the terminal device 2 of the user.

  For example, if the user responds better to the notification message related to the keyword “soccer” than the notification message related to the keyword “baseball”, the score of the keyword “soccer” is larger than the score of the keyword “baseball”. The reinforcement learning model is learned.

  The information processing apparatus 1 generates a notification message using the dialogue model based on the selected keyword (step S2). Note that the notification message may be generated at the timing of performing the push notification based on the notification information, or may be generated before the timing of performing the push notification. The notification information is information for performing push notification to the user, and is a preset time such as noon or 19:00.

  The dialogue model is a model learned by pairing a keyword and a notification message in the keyword. The notification message used for learning may be generated manually, for example, or may be collected from text obtained from the web, Twitter (registered trademark), or the like. The dialogue model is a model that generates a notification message as a natural sentence.

  The information processing apparatus 1 generates a notification message related to the keyword using the dialogue model. For example, when the selected keyword is “soccer”, a notification message such as “Do you know the result of the soccer game?” Regarding “soccer” is generated using the dialogue model.

  The information processing apparatus 1 transmits the generated notification message to the user terminal device 2 when it is time to make a push notification based on the notification information (step S3).

  As described above, the information processing apparatus 1 can automatically generate a notification message for push notification to the user terminal device 2 using the reinforcement learning model. In addition, since the notification message is a content related to the keyword based on the reinforcement learning model, the satisfaction level of the user can be improved.

[2. Configuration of information processing system 5]
FIG. 2 is a diagram illustrating a configuration example of the information processing system 5. As illustrated in FIG. 2, the information processing system 5 according to the embodiment includes an information processing device 1, a terminal device 2, a speech recognition server 3, and a speech synthesis server 4.

  The terminal device 2, the speech recognition server 3, the speech synthesis server 4, and the information processing device 1 are connected to be communicable with each other wirelessly or via a network N. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

  The terminal device 2 is realized by a smartphone, a tablet terminal, a desktop PC (Personal Computer), a notebook PC, a PDA (Personal Digital Assistant), or the like.

  The speech recognition server 3 is a device that performs natural language processing on speech information and converts speech data into text data. When the speech recognition server 3 receives speech speech data from the terminal device 2, the speech recognition server 3 converts the speech data into text data. The voice recognition server 3 transmits text data obtained by converting the voice data to the information processing apparatus 1.

  The voice synthesis server 4 converts text data such as a notification message generated by the information processing apparatus 1 into voice data. The voice synthesis server 4 transmits the voice data obtained by converting the text data to the terminal device 2.

  The information processing device 1 generates text data of a notification message to be transmitted to the terminal device 2. Further, the information processing apparatus 1 generates text data of the notification message based on the text data transmitted from the terminal apparatus 2 or the text data obtained by converting the voice data via the voice recognition server 3. The information processing device 1 transmits the generated notification message and text data of other messages to the speech synthesis server 4 and the terminal device 2.

  Note that the speech recognition server 3 and the speech synthesis server 4 may be configured integrally with the information processing apparatus 1. When the terminal device 2 has a voice recognition function or a voice synthesis function, the voice data and the text data may be converted using these functions.

[3. Configuration of information processing apparatus 1]
Next, the information processing apparatus 1 according to the embodiment will be described with reference to FIG. FIG. 3 is a diagram illustrating a configuration example of the information processing apparatus 1.

  The information processing apparatus 1 includes a reception unit 10, a transmission unit 20, a storage unit 30, and a processing unit 40.

  The receiving unit 10 receives operation information from the user terminal device 2 via the network N. The operation information includes user response history information (response history) and action history information.

  The response history information is information related to the user's response history with respect to the notification message. The response history information includes the presence or absence of a user response to the notification message. Further, the user's response to the notification message includes a response message based on the user's utterance to the notification message, and an operation (click information or touch information) of the user's terminal device 2 with respect to the notification message. The response history information includes response history information for the current notification message and response history information for past notification messages.

  The action history information includes user utterance history information, user search history information, user browsing history information, and the like.

  The user's utterance history information is information regarding the contents of conversations and utterances that the user has performed with other users via the terminal device 2 and the contents of conversations and utterances that have been performed with the information processing apparatus 1. . Here, the user's utterance history information does not include the user's response message to the notification message.

  The user search history information is word information input by a search performed by the user via the terminal device 2, content information clicked by the user, for example, word information in an article clicked by the user.

  The user browsing history information is information on web pages browsed by the user with the terminal device 2 and information on the user's product purchase history.

  The receiving unit 10 receives data from a server or the like installed outside via the network N.

  The storage unit 30 includes a dialogue model storage unit 31, a reinforcement learning model storage unit 32, and a word storage unit 33. The storage unit 30 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk.

  The dialogue model storage unit 31 stores a dialogue model. The interaction model may be newly acquired and updated via the network N.

  The reinforcement learning model storage unit 32 stores a reinforcement learning model for selecting a keyword. The reinforcement learning model is generated and learned by reinforcement learning, for example, Q-learning.

  Reinforcement learning is a learning method that preferentially selects an action that gives a maximum reward in the future when the action is given to a certain state. Various actions are tried for the state, and the reinforcement learning model is updated by receiving a reward corresponding to the tried action.

  In the reinforcement learning according to the embodiment, the word set in the latest action history information is “state”, the keyword is “action”, and the user's response to the notification message related to the keyword is “reward”.

  For example, the reward when the user responds to the notification message is “+1”. For example, the reward when the user does not respond to the notification message is “0”. In addition, for example, when the user utters a notification message and the utterance content is a positive content for the notification message, the reward is “+1”, and the negative content is The reward is “−1”. Also, the reward when the dialogue continues for a predetermined number of times or more with respect to the notification message is “+5”. The predetermined number of times is a preset number of times.

  The latest action history information is action history information within a first predetermined time before performing a push notification. The first predetermined time is a preset time, for example, 24 hours, one week.

  In the reinforcement learning model, a score that is an integrated value of rewards for each keyword is attached, and when generating a message by push notification, a word extracted by the latest action history information, and the keyword with the highest score is selected. Selected.

  Note that the initial value of the score in the reinforcement learning model is set based on, for example, a general keyword ranking or a user response rate in a system that responds to a certain keyword. For example, the initial score for a keyword with a high ranking is increased.

  Thereby, it can suppress that the notification message regarding a keyword with low topicality is produced | generated also in the initial stage of reinforcement learning, and it can suppress that such a notification message is transmitted to a user's terminal device 2. FIG.

  The word memory | storage part 33 memorize | stores temporarily the word extracted by the analysis part 41 mentioned later from the latest action history information. Specifically, the word storage unit 33 stores the words extracted by the analysis unit 41 for a second predetermined time. The second predetermined time is a preset time, may be the same time as the first predetermined time, or may be a time longer than the first predetermined time.

  The processing unit 40 includes an analysis unit 41, a selection unit 42, a generation unit 43, and a learning unit 44.

  The analysis unit 41 determines whether there is a user response to the notification message. The analysis unit 41 responds to the notification message within the third predetermined time, for example, when the operation information of the terminal device 2 for the notification message or the response message is received, there is a user response to the notification message. It is determined that The analysis unit 41 determines that there is no user response to the notification message when there is no response to the notification message within the third predetermined time. The third predetermined time is a preset time.

  Further, when the analysis unit 41 receives a response message for the notification message, the analysis unit 41 analyzes the response message with respect to the text data of the response message using morphological analysis or the like, and a word group included in the response message To extract. Then, the analysis unit 41 specifies whether the response message has a positive content or a negative content with respect to the notification message.

  Moreover, the analysis part 41 extracts the word contained in action history information from a user's action history information. For example, when the user utters “Yesterday's soccer was fun”, the analysis unit 41 extracts words such as “Yesterday” and “soccer”.

  The selection unit 42 selects a keyword using the reinforcement learning model when it is time to perform push notification based on the notification information. Using the reinforcement learning model, the selection unit 42 selects a word having the highest score from the words stored in the word storage unit 33 as a keyword. That is, the selection unit 42 selects a word having the highest score in the reinforcement learning model as a keyword from words extracted from the latest action history information.

  For example, when the score obtained by reinforcement learning is the score shown in FIG. 4 and “sports”, “soccer”, and “baseball” are stored in the word storage unit 33, the selection unit 42 selects “soccer As a keyword. FIG. 4 is a diagram illustrating an example of a score.

  The generation unit 43 generates a notification message using the dialogue model based on the keyword selected by the selection unit 42. That is, the generation unit 43 generates a notification message using a reinforcement learning model and further using a dialogue model. The generation unit 43 generates a natural sentence related to the keyword as a notification message for the keyword.

  For example, when the keyword selected by the selection unit 42 is “soccer”, using the dialogue model, “Do you know the result of the soccer game?” A notification message such as

  Note that the generation unit 43 may generate a plurality of notification message candidates for a keyword using a dialogue model. In this case, the generation unit 43 selects a notification message with a predetermined probability from a plurality of notification message candidates. The predetermined probability is a preset probability, for example, the same probability.

  The transmission unit 20 transmits the notification message generated by the generation unit 43 to the terminal device 2 or the voice synthesis server 4 (see FIG. 2) via the network N based on the notification information, and notifies the push notification. .

  The learning unit 44 rewards the keyword based on the user's response to the notification message, and learns the reinforcement learning model. The learning unit 44 learns a reinforcement learning model based on a user response history with respect to notification messages transmitted in the past.

  For example, when there is a user response to the notification message, the learning unit 44 gives a reward “+1” to the keyword selected when the notification message is generated, and learns the reinforcement learning model. When there is no user response to the notification message, the learning unit 44 gives a reward “0” to the keyword selected when the notification message is generated, and learns the reinforcement learning model. In addition, when a predetermined number of dialogues or more continue for the notification message, the learning unit 44 gives a reward “+5” to the keyword selected when the notification message is generated, and learns the reinforcement learning model. .

[4. Notification message generation process]
Next, the notification message generation process will be described with reference to FIG. FIG. 5 is a flowchart illustrating an example of a notification message generation process according to the embodiment.

  The selection unit 42 selects a keyword using the reinforcement learning model from the words stored in the word storage unit 33 when it is time to make a push notification based on the notification information (step S10).

  The generation unit 43 generates a notification message using the dialogue model based on the keyword selected by the selection unit 42 (step S11).

  The transmission part 20 transmits the notification message produced | generated by the production | generation part 43 to a user's terminal device 2 (step S12).

  The receiving unit 10 receives a user response to the notification message (step S13), and the analyzing unit 41 analyzes the user response (step S14).

  The learning unit 44 rewards the keyword selected by the selection unit 42 based on the response content of the user to the notification message, updates the score, and learns the reinforcement learning model (step S15).

[5. Modified example]
In addition to the above-described embodiment, the following modifications can be applied.

  In the above embodiment, the “state” of the reinforcement learning model is set as a word set in the latest action history information, and “behavior” is set as a keyword in the word set. However, “action” is prepared separately. A keyword selected from the keyword set may be used.

  In this case, the dialogue model is a model that generates a notification message from a sentence related to a keyword. For example, when the information processing apparatus 1 makes a push notification about a news article, the conversation model is learned by pairing the news article with the summary of the news article. That is, the dialogue model in this case is a model that selects a summary of the news article from the news of the selected keyword and generates it as a notification message.

  Note that news related to keywords may be selected in order of popularity, or may be selected using an existing recommendation engine. By notifying the summary of a news article as a notification message, a notification message with improved readability can be displayed in a narrow notification area of the terminal device 2.

  The selection unit 42 selects a keyword having the highest score in the reinforcement learning model from a separately prepared keyword set in accordance with user behavior history information.

  The generation unit 43 generates a notification message using the dialogue model based on the keyword selected by the selection unit 42.

  For example, in a news article, there is “Yoshinori, 30% return for the sixth hit of the season! Raise 25th place with 3000 hits in total!”, And “baseball” is selected as a keyword by the selection unit 42. Based on the dialogue model, the generation unit 43 has a 30% batting average for the above-mentioned news article summary “Yoshinori“ Strike ”! Is generated as a notification message. And the transmission part 20 carries out push notification of the produced | generated notification message.

  Note that the dialogue model may be learned so that a specific keyword (for example, “Yoshinori”) is included in the notification message.

  Moreover, you may use a RNN Encoder-Decoder model instead of an interaction model. The RNN Encoder-Decoder model is a model in which a news article is converted into a real vector by an encoding RNN model and then a summary of the news article is generated as a notification message by a decoding RNN model using the vector as an input.

  It should be noted that “action” may be a category selected from a separately prepared category set, for example, “baseball”, “idol”, “stock”, etc. in the case of a news article.

  In addition, “action” may be a buzzword extracted from Twitter or the like, and the generation unit 43 may generate an excitement summary of Twitter or the like as a notification message using a dialogue model.

  As described above, the information processing apparatus 1 can automatically generate a notification message related to the keyword selected from the keyword set. Therefore, user satisfaction with the notification message can be improved.

  In the above embodiment, a keyword is selected using a reinforcement learning model. However, an interaction model that generates a notification message may be selected by reinforcement learning.

  In the information processing apparatus 1, the dialogue model storage unit 31 stores a plurality of dialogue models, and the selection unit 42 learns by reinforcement learning and is a reinforcement learning model (hereinafter referred to as a reinforcement dialogue model) for selecting a dialogue model. ) To select a call message. That is, the dialogue model becomes “action” in reinforcement learning. The enhanced interaction model is stored in the storage unit 30.

  The interaction model storage unit 31 stores, for example, a model that mainly generates a notification message asking a question or a model that generates a notification message that directly notifies a result or the like as a type of interaction model. In such a dialogue model, for example, a sentence with a question content and a sentence with a content that directly notifies a result are used as a sentence when generating the model.

  The generation unit 43 generates a notification message using the keyword selected by the reinforcement learning model and the interaction model selected by the reinforcement interaction model.

  In the information processing apparatus 1, different notification messages can be generated for the same keyword depending on the conversation model used.

  For example, when the keyword is “soccer”, if you use an interactive model that mainly generates an inquiry message, “Do you know the soccer game result?” When the dialog model for notification is used, “XX team won!” Is generated as a notification message.

  Based on the user's response to the notification message, the learning unit 44 gives a reward to the keyword to learn the reinforcement learning model, and further gives a reward to the dialogue model to learn the reinforcement dialogue model.

  For example, if the notification message “Do you know the result of the soccer game?” Is better than the notification message of “XX Team has won!” The score for the dialogue model that generates “Do you know the result of the soccer game?” Increases.

  The information processing apparatus 1 can generate a notification message with a good user response rate and the like for the keyword. Therefore, user satisfaction with the notification message can be improved.

  Note that the information processing apparatus 1 may send a notification message to another user and give a reward to the dialogue model based on the response. That is, the information processing apparatus 1 may learn a strengthened dialogue model based on response history information of other users.

  Thereby, the information processing apparatus 1 can generate a notification message with a good response rate of many users.

  The selection unit 42 may select a dialogue model using the reinforcement dialogue model including the keyword selected by the reinforcement learning model. That is, the selection unit 42 may select a dialogue model by keyword or category to which the keyword belongs.

  Accordingly, the information processing apparatus 1 can automatically generate a notification message with a good user response rate in accordance with a keyword or the like. Therefore, user satisfaction with the notification message can be improved.

  In addition to the above-described modification examples, the following modification examples can be applied.

  The interaction model may be learned to generate a notification message with the content of the question.

  Thereby, the information processing apparatus 1 can increase a user's response rate.

  The selection unit 42 may select, as a keyword, for example, a word that is a topic on Twitter or the like.

  Thereby, the information processing apparatus 1 can diversify the notification message. In addition, the information processing apparatus 1 can obtain a user response to a notification message related to a new keyword, and can learn a reinforcement learning model based on the obtained response.

  The selection unit 42 may periodically increase the keyword score. For example, in the Olympic year, the score of keywords related to the Olympics is increased.

  Thereby, the information processing device 1 can automatically generate a topical notification message and transmit it to the user terminal device 2.

  Further, the learning unit 44 may learn a dialogue model based on a sentence obtained by crowdsourcing.

  Thereby, the information processing apparatus 1 can diversify the notification message.

  Note that the learning unit 44 may be provided in an apparatus outside the information processing apparatus 1, and the information processing apparatus 1 may receive and use the reinforcement learning model generated by a different apparatus by the reception unit 10 via the network N.

  The information processing apparatus 1 may apply a gamification system. For example, when the user's response count is “point” and the response count is greater than a preset value, the “level” is increased. And the production | generation part 43 changes a notification message according to a level. For example, when the level increases, the generation unit 43 generates a more intimate notification message, for example, personal content as the notification message. Further, the information processing apparatus 1 may increase the number of push notifications as the level increases. Note that the points may be increased when the number of consecutive conversations or the response content is positive.

  Thereby, the intimacy between the user and the information processing apparatus 1 can be improved, and the response rate of the user to the notification message can be increased.

  The notification information may include user action history information. The information processing device 1 generates a notification message at a timing based on the content of the user's utterance and transmits the notification message to the user's terminal device 2.

  For example, when the user utters “played soccer on a holiday”, the information processing apparatus 1 selects “soccer” as a keyword using the reinforcement learning model at that timing. The notification message is generated, and the notification message is transmitted to the terminal device 2 of the user.

  Thereby, the information processing apparatus 1 can transmit a notification message to the user's terminal device 2 at a timing when the user is interested based on the user's behavior history, and can perform a natural dialogue with the user.

  Further, the notification information may be information related to timing based on the user context. The context is information such as user attribute information, user position information, and user exercise state information.

  For example, the information processing device 1 transmits a notification message at the timing when the user returns home based on the position information obtained from the user's terminal device 2.

  Thereby, the information processing apparatus 1 can transmit a notification message at a timing when the response rate of the user increases.

  Further, the notification information may be information related to the timing of transmitting a message that responds to the user's utterance.

  In this case, the generation unit 43 may use an RNN Encoder-Decoder model that generates a corresponding notification message using the user's utterance as an input sentence as the conversation model. The RNN Encoder-Decoder model is a model in which an input sentence is converted into a real vector by an encoding RNN model, and then an output sentence is generated by a decoding RNN model using the vector as an input. The production | generation part 43 produces | generates the notification message containing a keyword using a RNN Encoder-Decoder model.

  For example, when the user's utterance is “Today's topic?” And the keyword is “Soccer”, the generation unit 43 uses the RNN Encoder-Decoder model with “What is today's topic?” As an input sentence. Notification message candidates are generated in descending order of generation probability. An approximate solution such as a beam search may be used to generate the notification message candidate.

  Then, the generation unit 43 selects only a message including the keyword “soccer” from the generated notification message candidates, and further selects a message having a generation probability equal to or higher than a preset threshold value as a notification message. For example, the generation unit 43 selects a notification message with a predetermined probability from a plurality of notification messages. In this way, the generation unit 43 may generate a notification message.

  The information processing apparatus 1 can automatically generate a message that responds to the user's utterance using the reinforcement learning model.

  Moreover, you may use the distributed expression which shows the word extracted from action log | history information, and a keyword by a real vector. In this case, the generation unit 43 may generate a notification message using a learning model in which the reinforcement learning model and the interaction model are combined into one. The generation unit 43 selects, for example, a distributed representation notification message having a high cosine similarity with respect to the distributed representation of the keyword, and generates a notification message.

  The learning unit 44 learns the learning model so that the distributed expression in the keyword and the distributed expression in the notification message exist close together in the distributed expression space. The learning model is learned and generated using, for example, an RNN (Recurrent Neural Network) technique equipped with LSTM (Long Short-Term Memory).

  Thereby, the information processing apparatus 1 automatically generates a notification message for a similar keyword even when, for example, the word in the latest action history information of the user does not have a word that matches the keyword in the reinforcement learning model. can do.

  Further, the generation unit 43 may generate a notification message using a language model. The language model is a model that statistically summarizes the appearance rate of general messages. When a plurality of notification message candidates are generated based on the keyword by the generation unit 43, the probability that a notification message candidate that becomes a general message is selected as a notification message is reduced using a language model.

  Thereby, the information processing apparatus 1 can suppress that a general message is selected as the notification message, and can prevent the general message from being transmitted to the user terminal device 2. Therefore, it is possible to suppress the user's satisfaction with the notification message from decreasing.

[6. effect]
The information processing apparatus 1 includes a generation unit 43 and a transmission unit 20. The production | generation part 43 produces | generates the notification message transmitted to a user's terminal device 2 using the reinforcement learning model learned by reinforcement learning. The transmission unit 20 transmits the notification message generated by the generation unit 43 to the user terminal device 2 based on the notification information.

  Thereby, the information processing apparatus 1 can generate | occur | produce the notification message transmitted to a user's terminal device 2 automatically using a reinforcement learning model. In addition, the information processing apparatus 1 can automatically generate a notification message with a good user response rate and the like.

  The learning unit 44 learns a reinforcement learning model based on response history information for notification messages transmitted in the past.

  Thereby, the information processing apparatus 1 can automatically generate a notification message with a good response rate of the user.

  The learning unit 44 learns the reinforcement learning model based on the presence / absence of a user response to a notification message transmitted in the past.

  Accordingly, the information processing apparatus 1 can automatically generate a user response, for example, a notification message with a high user response rate.

  The learning unit 44 learns the reinforcement learning model based on the number of conversations based on the notification message transmitted in the past.

  Thereby, for example, the information processing apparatus 1 can automatically generate a notification message with a high response rate of the user by increasing the keyword score regarding the notification message in which the conversation has continued.

  The generation unit 43 generates a notification message using the dialogue model based on the keyword selected using the reinforcement learning model.

  As a result, the information processing apparatus 1 can automatically generate a natural sentence related to a keyword as a notification message.

  The learning unit 44 learns the dialogue model by reinforcement learning based on the response history of other users.

  Thereby, the information processing apparatus 1 can automatically generate a notification message with a good response rate of the user.

  The transmission unit 20 transmits a notification message based on the user context.

  Thereby, the information processing apparatus 1 can increase the response rate of the user with respect to the notification message.

  The transmission unit 20 pushes the notification message to the user terminal device 2.

  Thereby, the information processing apparatus 1 can improve the utilization rate of the application which has notification functions, such as news in the terminal device 2, for example.

  The production | generation part 43 produces | generates a notification message using the reinforcement learning model which selects a word as a keyword with respect to the word extracted from the latest action history information.

  Thereby, the information processing apparatus 1 can generate, for example, a notification message that is highly relevant to the keyword word by selecting the same word as the word extracted from the latest action history information as the keyword. .

  The generation unit 43 generates a notification message for the vector corresponding to the word using a learning model.

  Thereby, the information processing apparatus 1 can generate a notification message for a similar keyword even when, for example, the word in the latest action history information of the user has no word that matches the keyword in the learning model. .

[7. Hardware configuration]
The information processing apparatus 1 according to the above-described embodiment is realized by a computer 1000 configured as shown in FIG. 6, for example. FIG. 6 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing apparatus 1. The computer 1000 includes a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

  The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each unit. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, a program depending on the hardware of the computer 1000, and the like.

  The HDD 1400 stores programs executed by the CPU 1100, data used by the programs, and the like. The communication interface 1500 receives data from other devices via the network N and sends the data to the CPU 1100, and transmits data determined by the CPU 1100 to other devices via the network N.

  The CPU 1100 controls an output device such as a display and a printer and an input device such as a keyboard and a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. Further, the CPU 1100 outputs the determined data to the output device via the input / output interface 1600.

  The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads such a program from the recording medium 1800 onto the RAM 1200 via the media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. Etc.

  For example, when the computer 1000 functions as the information processing apparatus 1 according to the embodiment, the CPU 1100 of the computer 1000 implements the function of the processing unit 40 by executing a program loaded on the RAM 1200. The CPU 1100 of the computer 1000 reads these programs from the recording medium 1800 and executes them. However, as another example, these programs may be acquired from other devices via the network N.

  As described above, some of the embodiments and modifications of the present application have been described in detail with reference to the drawings. It is possible to carry out the present invention in other forms that have been modified and improved.

[8. Other]
In addition, among the processes described in the above-described embodiments and modifications, all or a part of the processes described as being automatically performed can be manually performed, or are described as being performed manually. All or part of the processing can be automatically performed by a known method. In addition, the processing procedures, specific names, and information including various data and parameters shown in the document and drawings can be arbitrarily changed unless otherwise specified. For example, the various types of information illustrated in each drawing is not limited to the illustrated information.

  Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured.

  In addition, the above-described embodiments and modifications can be combined as appropriate within a range that does not contradict processing contents.

  In addition, the “section (module, unit)” described above can be read as “means” or “circuit”. For example, the generation unit 43 can be read as generation means or a generation circuit.

DESCRIPTION OF SYMBOLS 1 Information processing apparatus 2 Terminal apparatus 10 Reception part 20 Transmission part (notification part)
30 storage unit 40 processing unit 41 analysis unit 42 selection unit 43 generation unit 44 learning unit

Claims (10)

A learning unit that learns a reinforcement learning model for message generation for the user by reinforcement learning based on a user's response history to a message notified in the past;
Using the message generation reinforcement learning model, a generation unit that generates a message to be notified to the user terminal device;
An information processing apparatus comprising: a notification unit that notifies the terminal device of the message generated by the generation unit based on information for performing a push notification to the user. The learning unit
The information processing apparatus according to claim 1, wherein the message generation reinforcement learning model is learned based on a response history including presence / absence of a response to the message notified in the past. The learning unit
The information processing apparatus according to claim 1, wherein the reinforcement learning model for message generation is learned based on a response history including the number of dialogues with respect to the message notified in the past. The generator is
The information processing apparatus according to any one of claims 1 to 3, wherein the message is generated using a message generation model based on a keyword selected using the reinforcement learning model for message generation. . The learning unit
The information processing apparatus according to claim 4, wherein the message generation model is learned by reinforcement learning based on a response history including a response history of another user. The notification unit
The information processing apparatus according to any one of claims 1 to 5, wherein the message generated by the generation unit is notified to the terminal device based on a context of the user. The generator is
7. The message to be notified to the terminal device is generated for a word generated and input by the reinforcement learning using a learning model for outputting the word. The information processing apparatus described in 1. The generator is
The message to be notified to the terminal device is generated using a learning model that outputs an output vector for an input vector that is generated by the reinforcement learning and corresponding to an input word. 6. The information processing apparatus according to any one of 6. An information processing method executed by an information processing apparatus,
A learning process for learning a reinforcement learning model for message generation for the user by reinforcement learning based on a user's response history to a message notified in the past;
Using the message generation reinforcement learning model, generating a message to be notified to the terminal device of the user,
A notification step of notifying the terminal device of the message generated by the generation step based on information for performing a push notification to the user. A learning procedure for learning a reinforcement learning model for message generation for the user by reinforcement learning based on a user's response history to a message notified in the past;
Using the message generation reinforcement learning model, a generation procedure for generating a message to be notified to the user terminal device;
A program causing a computer to execute a notification procedure for notifying the terminal device of the message generated by the generation procedure based on information for performing a push notification to the user.

Images