JP2025138445A - Information processing device, information processing method, and information processing program - Google Patents

Abstract

[Problem] Take screenshots on the server side and perform actions based on the screenshots.
[Solution] The information processing device of the present application comprises an acquisition unit that acquires screenshots and question text from a user, an estimation unit that infers the intention of the user's question from the screenshot and question text, a selection unit that selects an expert to answer the question from the screenshot and the intention of the user's question, and a provision unit that provides the screenshot and question text to the expert to make an inquiry, wherein the acquisition unit acquires answers from the experts and the provision unit provides the answers from the experts to the user.
[Selected Figure] Figure 3

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

With the spread of social networking services (SNS), technology has been disclosed that allows users to share screenshots they have created with other users (see Patent Document 1).

Japanese Patent Application Laid-Open No. 2023-033386

However, with the above-mentioned conventional technology, screenshots are shared directly between users, leaving little room for server-side involvement.

This application was made in consideration of the above, and aims to capture screenshots on the server side and perform actions triggered by the screenshots.

The information processing device according to the present application comprises an acquisition unit that acquires a screenshot and a question from a user, an estimation unit that estimates the intent of the user's question from the screenshot and the question, a selection unit that selects an expert to answer the question from the screenshot and the intent of the user's question, and a provision unit that provides the screenshot and the question to the expert to make an inquiry, wherein the acquisition unit acquires an answer from the expert, and the provision unit provides the answer from the expert to the user.

According to one aspect of the embodiment, screenshots can be captured on the server side and actions can be taken based on the screenshot.

FIG. 1 is an explanatory diagram illustrating an overview of an information processing system according to an embodiment. FIG. 2 is an explanatory diagram showing an overview of comment display caused by a screenshot. FIG. 3 is an explanatory diagram showing an outline of inquiries to experts caused by screenshots. FIG. 4 is a diagram illustrating an example of the configuration of a terminal device according to the embodiment. FIG. 5 is a diagram illustrating an example of the configuration of a server device according to the embodiment. FIG. 6 is a flowchart showing a first processing procedure according to the embodiment. FIG. 7 is a flowchart showing a second processing procedure according to the embodiment. FIG. 8 is a diagram illustrating an example of a hardware configuration.

Below, detailed explanations will be given of the information processing device, information processing method, and information processing program (hereinafter referred to as "embodiments") according to the present application, with reference to the drawings. Note that the information processing device, information processing method, and information processing program according to the present application are not limited to these embodiments. Furthermore, identical components in the following embodiments will be designated by the same reference numerals, and duplicate explanations will be omitted.

[1. Overview of the information processing system]
First, an overview of an information processing system according to an embodiment will be described with reference to Fig. 1. Fig. 1 is an explanatory diagram showing an overview of an information processing system according to an embodiment. As shown in Fig. 1, an information processing system 1 according to an embodiment includes a terminal device 10 and a server device 100. These various devices are connected to each other via a network N in a wired or wireless manner so as to be able to communicate with each other. This allows the terminal device 10 to cooperate with the server device 100. The network N is, for example, a local area network (LAN), a wide area network (WAN), the Internet, etc.

The terminal device 10 is an information processing device used by a user U. For example, the terminal device 10 may be a smart device such as a smartphone or tablet, a mobile phone such as a feature phone (Gala-ke/Gala-ho), a personal computer (PC), a personal digital assistant (PDA), a game console or AV device with communication functions, an information appliance or digital appliance, a car navigation system, or a wearable device such as a smartwatch, a head-mounted display, or smart glasses. The terminal device 10 may also be a home or building, a car, a home appliance, or an electronic device that is compatible with the Internet of Things (IoT).

In this embodiment, terminal device 10 is a smart device such as a smartphone or tablet used by user U, and is a portable terminal device capable of communicating with any server device via a wireless communication network such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation: fifth generation mobile communication system), Bluetooth (registered trademark), or wireless LAN. Terminal device 10 also has a screen such as an LCD display with touch panel functionality, and accepts various operations on displayed data such as content, such as tapping, sliding, and scrolling, performed by user U with a finger or stylus. Note that operations performed on an area of the screen where content is displayed may be considered operations on the content. Terminal device 10 may not only be a smart device, but also an information processing device such as a desktop PC (Personal Computer) or laptop PC.

The server device 100 is, for example, a computer such as a PC or a blade server, or a mainframe or workstation. The server device 100 may also be implemented using cloud computing.

In this embodiment, the server device 100 is an information processing device that works in conjunction with the terminal device 10 of each user U and provides API (Application Programming Interface) services for various applications (hereinafter referred to as apps) and various data to each user U's terminal device 10, and is realized by a computer, cloud system, etc.

Furthermore, the server device 100 may be an information processing device that provides some kind of online service to the terminal device 10 of each user U. For example, the server device 100 may provide services such as internet connection, search services, SNS (Social Networking Service), electronic commerce (EC), electronic payments, online games, online banking, online trading, accommodation and ticket reservations, video and music distribution, news, maps, route searches, route guidance, line information, operation information, and weather forecasts as online services. In practice, the server device 100 may work in conjunction with various servers that provide the above-mentioned online services, and act as an intermediary for the online services or be responsible for processing the online services.

The server device 100 can acquire user information about the user U. For example, the server device 100 acquires information (attribute information) about the attributes of the user U, such as the gender, age, and area of residence of the user U. The server device 100 can also acquire information about the attributes of the user U, such as demographic attributes, psychographic attributes, geographic attributes, and behavioral attributes. The server device 100 may also acquire, as user information, the segment or persona (personality) to which the user U belongs in the marketing field. The server device 100 then stores and manages the information about the attributes of the user U (attribute information) along with identification information (user ID, etc.) that identifies the user U.

The server device 100 also acquires various historical information (log data) indicating user U's behavior from user U's terminal device 10 or from various servers based on the user ID, etc. For example, the server device 100 acquires user U's location history, which is a history of user U's location and date and time, from the terminal device 10. The server device 100 also acquires user U's search history, which is a history of search queries entered by user U, from a search server (search engine). The server device 100 also acquires user U's browsing history, which is a history of content viewed by user U, from a content server. The server device 100 also acquires user U's purchase history (payment history), which is a history of product purchases and payment processes, from an e-commerce server or a payment processing server. The server device 100 may also acquire user U's listing history and sales history, which are a history of user U's listings on the marketplace, from the e-commerce server or the payment processing server. The server device 100 also acquires user U's posting history, which is a history of user U's posts, from a posting server or SNS server that provides a word-of-mouth posting service. Note that the various servers described above may be the server device 100 itself. In other words, the server device 100 may function as the various servers described above.

Furthermore, the number of devices included in the information processing system 1 shown in Figure 1 is not limited to that shown. For example, in Figure 1, only one terminal device 10 is shown to simplify the illustration, but this is merely an example and is not limiting; there may be two or more devices.

[2. Use of Screenshots]
In recent years, as a means of sharing online information among young people, screenshots, which place less of a burden on the other party's packet communication, have become popular, replacing the issuance of links (presentation of link destinations) using share buttons and the like. However, currently, screenshot sharing is performed between user terminals, and the server side has little opportunity to intervene. In this embodiment, the server device 100 intervenes in the sharing of screenshots between terminal devices 10 and takes action in response to the screenshots.

[2-1. Group formation due to screenshots]
The server device 100 groups users who have taken screenshots of the same news article and forms a closed community (such as an open chat) that only users in the same group can participate in. In other words, the server device 100 may group users based on the commonality or similarity of the news articles they have taken screenshots of, and form a closed community that only users in the same group can participate in.

The server device 100 may also provide a service that allows users in the same group to post comments on the contents of a screenshot (the news article in question), or a service that allows users in the same group to exchange information.

For example, as shown in FIG. 1, the server device 100 acquires (collects) screenshots of news articles on a news site/app from the user U's terminal device 10 via the network N (step S1). At this time, the server device 100 acquires screenshots of parts of the news articles that interest the user U (or that are related to the user U's doubts, worries, or questions). At the time the screenshot is taken, it is assumed that the user U is quite interested in the content of that part of the news article (the focused part).

In practice, when user U takes a screenshot of a news article on terminal device 10, the screenshot may be automatically sent from terminal device 10 to server device 100 using a function of an app or the like, or user U may be asked whether or not to send the screenshot to server device 100, and the screenshot may be sent to server device 100 if user U gives permission.

Next, the server device 100 determines whether the screenshots of the news articles obtained from each user's terminal device 10 are identical (step S2).

Next, the server device 100 groups users whose screenshots of news articles are determined to be identical (step S3).

Next, the server device 100 creates a closed community (such as an open chat) that only users within the same group can participate in (step S4).

Next, the server device 100 provides an information sharing service within the closed community (step S5).

[2-2. Displaying comments due to screenshots]
Next, comment display caused by a screenshot will be described with reference to Fig. 2. Fig. 2 is an explanatory diagram showing an overview of comment display caused by a screenshot.

For example, as shown in FIG. 2, the server device 100 acquires a screenshot of a news article on a news site/app that allows comments to be posted from the terminal device 10 of the user U via the network N (step S11).

Note that news sites/apps are just one example. In reality, they could be social networking sites, Q&A sites, life advice sites, specialized field sites, video/music distribution sites, novel/illustration posting sites, or other sites/apps that allow users to post comments.

Next, the server device 100 extracts comments that are relevant to the content of the news article from among the comments posted to the news article on the news site/app as candidate comments (step S12). In other words, the server device 100 eliminates comments that are unrelated to the content of the news article (and comments that are only vaguely relevant) from among the comments posted to the news article.

In this case, when the server device 100 acquires a screenshot of a news article, it may identify the news site/app that published the news article. In other words, the server device 100 may identify the news site/app that is open (running) on the terminal device 10 of user U when the screenshot is taken. If the server device 100 is providing news articles to the terminal device 10 of user U, it is easy to identify the source of the news article.

The server device 100 may also use AI (Artificial Intelligence) to learn the news site/app and the form and format of the news article as learning data, and use this AI to estimate the news site/app that published the news article. Alternatively, the server device 100 may obtain, via the network N, from the terminal device 10 of the user U, a screenshot of the news article along with identification information indicating the news site/app that published it.

The server device 100 may also select other users (experts, experienced people, etc.) who are presumed to be able to make appropriate comments on the news article as respondents, send screenshots to the selected respondents, and request them to post comments, with the resulting comments being used as candidate comments.

Next, the server device 100 provides (presents) the candidate comments to the terminal device 10 of the user U via the network N (step S13). At this time, the user U selects an appropriate comment from the candidate comments (such as a comment that matches the user U's intentions). For example, the user U selects a comment related to their interests, concerns, worries, questions, etc.

Next, the server device 100 generates content that can be commented on by merging a screenshot of the news article with a comment (selected comment) selected by the user U from the candidate comments (step S14). Note that there may be multiple selected comments. The server device 100 may merge the screenshot of the news article with the selected comment to create a single image. This allows comments from experts and other users to be added to the screenshot.

In this case, the server device 100 may display the comment in the content at a position related to the screenshot of the news article. For example, the server device 100 may generate content in which the comment is placed in the position of the text or image (such as the object indicated by the comment) that corresponds to the comment in the screenshot of the news article.

The server device 100 may also estimate which parts of the screenshotted news article are likely to be of particular interest to the user U (such as parts that mention specific themes or keywords), and may highlight those parts in the content and provide information about those parts.

Next, the server device 100 provides the generated content to other users designated by user U (step S15). This allows user U to share the generated content with desired users. At this time, the server device 100 may provide the content to users in the same group as user U.

[2-3. Inquiry to an expert due to screenshots]
Next, an inquiry to an expert caused by a screenshot will be described with reference to Fig. 3. Fig. 3 is an explanatory diagram showing an overview of an inquiry to an expert caused by a screenshot.

For example, as shown in FIG. 3, the server device 100 acquires a screenshot of a news article from a news site/app that allows comments to be posted, and a question text, from the terminal device 10 of the user U via the network N (step S21).

Next, the server device 100 estimates the intention of user U's question from the acquired screenshot and question text (step S22). In other words, the server device 100 estimates which part of the screenshotted news article user U was interested in and what intention he or she had in asking the question.

At this time, the server device 100 may further clarify user U's points of interest and empathy in the news article based on user U's past searches, articles viewed, article ratings, etc., and infer the intention of user U's question.

The server device 100 may also infer the intent of the question using generative AI such as a Generative Pretrained Transformer (GPT). For example, the server device 100 may input a prompt containing the acquired screenshot and question text into the GPT, and obtain the intent of the question inferred by the GPT.

Next, the server device 100 selects an appropriate expert as the respondent based on the content of the screenshot and the intent of the question (or the question text) (step S23). In other words, the server device 100 matches experts in the specialized fields (areas) related to the screenshotted portion. Note that multiple experts may be selected. For example, if the screenshotted portion (or the portion thereof that is expected to be of particular interest to user U) relates to multiple specialized fields, the server device 100 may match experts for each specialized field.

The server device 100 may also determine whether to ask an expert or to provide an answer using GPT based on the content of the screenshot and the intent of the question (or the question text). If the server device 100 determines to provide an answer using GPT, it may input a prompt including the content of the screenshot and the intent of the question (or the question text) into GPT and obtain an answer generated by GPT.

The server device 100 may also have the predictive AI learn pairs of screenshot content, question intent, and expert as learning data, and use this predictive AI to select an expert based on the screenshot content and question intent.

In addition, experts may be associated in advance with the content of the screenshot (field, theme, etc.). Furthermore, the server device 100 may select an expert in a field corresponding to a pair of screenshot and question text. Furthermore, the server device 100 may select an expert in a field corresponding to a portion (part) screenshotted by multiple users (or a predetermined number of users or more).

Next, the server device 100 provides the screenshot and question text to the selected expert via the network N and obtains an answer from the expert (step S24).

At this time, the server device 100 may simultaneously send the content and link of the original text of the screenshot (such as a news article) to the expert along with the screenshot and question.

The server device 100 may also pay a reward (points, etc.) to the expert who provides the answer. The expert receives a reward (points, etc.) when providing an answer.

Next, the server device 100 provides the answer to the terminal device 10 of the user U via the network N (step S25). At this time, the server device 100 may present to the user U, along with the answer, information about the expert who provided the answer, such as the expert's name, field of expertise, and rank of expertise (the answer's credibility rank). The expert who provided the answer can appeal to the user U about the high level of their expertise and increase their recognition as an expert.

Next, the server device 100 merges the screenshot and the response to create an article (step S26). That is, the server device 100 generates content by merging the screenshot and the response. At this time, the server device 100 may provide the generated content to other users designated by user U. If the content is made public to an unspecified number of users, the expert who provided the response can demonstrate their high level of expertise to users who view the content and increase their recognition as an expert.

Furthermore, when generating the content, the server device 100 may generate the content using the same method as the comment display described in 2-2 above. In this case, the answers from the experts are treated as posted comments.

[2-3-1. Supplementary Information]
The server device 100 may store sets of screenshots, question sentences, and answers as history information (log data). In this case, the history information may be stored for a fixed period of time, and the oldest history information may be deleted after the fixed period has elapsed.

Furthermore, when acquiring a screenshot and question text, the server device 100 may refer to history information of previously (already) acquired screenshots and question texts to determine whether the acquired screenshot and question text are the same as those previously acquired. If the acquired screenshot and question text are the same as those previously acquired, the server device 100 may extract an answer to the same screenshot and question text from the past history information and provide the extracted answer to the terminal device 10 of the user U via the network N to avoid repeatedly asking the same question to the expert.

Furthermore, the server device 100 may estimate the suitability of user U from a combination of the screenshot, the source of the screenshot (such as a news article), and the question text, and may not allow user U to inquire of an expert if it is estimated that user U is not suitable.

The server device 100 may also infer from the screenshot and question text whether user U intended to ask a serious question. If the server device 100 determines that user U did not intend to ask a serious question, but rather that he or she was asking the question with malicious intent, such as to prank, troll, or cause trouble, the server device 100 may send a warning about this behavior to user U's terminal device 10 via the network N, discard the screenshot and question text, and terminate processing.

The server device 100 may also determine whether the screenshotted portion matches the content of the question (whether the degree of match exceeds a threshold), and if there is a match (the degree of match exceeds a threshold), allow the user to make an inquiry to an expert. The server device 100 may also use image recognition and natural language processing (NLP) to determine whether the screenshotted portion matches the content of the question. For example, the server device 100 may use image recognition and natural language processing to vectorize the text of the screenshotted portion and the question, and determine whether there is a match based on the cosine similarity of the vectors.

The server device 100 may also determine whether the number of users with identical screenshot content exceeds a threshold (e.g., 30 or more users in the most recent past), and if so, permit the user to inquire with an expert. Alternatively, the server device 100 may determine whether the number of times the screenshotted portion (part) is identical exceeds a threshold, and if so, permit the user to inquire with an expert.

The server device 100 may also determine whether there are multiple experts (two or more) whose areas of expertise match the area of expertise related to the screenshotted portion, and if there are multiple experts (two or more), allow the user to inquire about the experts.

The server device 100 may also determine whether the original screenshot (such as a news article) was published within 24 hours, and if so, allow inquiries to be made to experts.

Note that if the server device 100 is unable to obtain screenshots of news articles from the terminal device 10 due to regulations, specifications, etc., some (or all) of the above-mentioned server device 100 processing that directly uses screenshots may be performed (on behalf of) the user U's terminal device 10 by launching an app or the like that has the relevant function.

3. Example of terminal device configuration
Next, the configuration of the terminal device 10 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the terminal device 10 according to the embodiment. As shown in Fig. 4, the terminal device 10 includes a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, a sensor unit 20, a control unit 30 (controller), and a storage unit 40.

(Communication unit 11)
The communication unit 11 is connected to the network N via a wired or wireless connection, and transmits and receives information to and from the server device 100 via the network N. For example, the communication unit 11 is realized by a NIC (Network Interface Card), an antenna, or the like.

(Display section 12)
The display unit 12 is a display device that displays various information such as position information. For example, the display unit 12 is a liquid crystal display (LCD) or an organic electroluminescent display (OLED). The display unit 12 is also a touch panel display, but is not limited to this.

(Input unit 13)
The input unit 13 is an input device that accepts various operations from the user U. For example, the input unit 13 has buttons for inputting letters, numbers, etc. The input unit 13 may be an input/output port (I/O port), a USB (Universal Serial Bus) port, etc. In addition, when the display unit 12 is a touch panel display, a part of the display unit 12 functions as the input unit 13. In addition, the input unit 13 may be a microphone that accepts voice input from the user U. The microphone may be wireless.

(Positioning unit 14)
The positioning unit 14 receives signals (radio waves) transmitted from satellites of the GPS (Global Positioning System), and acquires, based on the received signals, location information (e.g., latitude and longitude) indicating the current location of the terminal device 10. That is, the positioning unit 14 measures the location of the terminal device 10. Note that the GPS is merely an example of a GNSS (Global Navigation Satellite System).

In addition, the positioning unit 14 can determine the position using various methods other than GPS. For example, the positioning unit 14 may determine the position using various communication functions of the terminal device 10 as an auxiliary positioning means for position correction, etc., as described below.

(Wi-Fi positioning)
For example, the positioning unit 14 uses the Wi-Fi (registered trademark) communication function of the terminal device 10 or the communication networks of each communication company to determine the position of the terminal device 10. Specifically, the positioning unit 14 performs Wi-Fi communication or the like to determine the distance to a nearby base station or access point, thereby determining the position of the terminal device 10.

(Beacon positioning)
The positioning unit 14 may also measure the position by using a Bluetooth (registered trademark) function of the terminal device 10. For example, the positioning unit 14 measures the position of the terminal device 10 by connecting to a beacon transmitter connected via the Bluetooth (registered trademark) function.

(geomagnetic positioning)
The positioning unit 14 also measures the position of the terminal device 10 based on the geomagnetic pattern of the structure measured in advance and a geomagnetic sensor included in the terminal device 10 .

(RFID positioning)
Furthermore, for example, if the terminal device 10 has the functionality of an RFID (Radio Frequency Identification) tag equivalent to a contactless IC card used at station ticket gates, in stores, etc., or has the functionality to read an RFID tag, the location of use is recorded together with information on payments, etc., made by the terminal device 10. The positioning unit 14 may obtain such information to determine the location of the terminal device 10. Alternatively, the location may be determined by an optical sensor, an infrared sensor, or the like provided in the terminal device 10.

If necessary, the positioning unit 14 may use one or a combination of the positioning methods described above to determine the position of the terminal device 10.

(Sensor unit 20)
The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10. The connection may be wired or wireless. For example, the sensors may be detection devices other than the terminal device 10, such as wearable devices or wireless devices. In the example shown in FIG. 4 , the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, a barometric pressure sensor 23, a temperature sensor 24, a sound sensor 25, a light sensor 26, a magnetic sensor 27, and an image sensor (camera) 28.

Note that the sensors 21-28 described above are merely examples and are not intended to be limiting. That is, the sensor unit 20 may be configured to include some of the sensors 21-28, or may include other sensors, such as a humidity sensor, in addition to or instead of the sensors 21-28.

The acceleration sensor 21 is, for example, a three-axis acceleration sensor that detects the physical movement of the terminal device 10, such as the direction of movement, speed, and acceleration of the terminal device 10. The gyro sensor 22 detects the physical movement of the terminal device 10, such as tilt in three axial directions, based on the angular velocity of the terminal device 10. The air pressure sensor 23 detects, for example, the air pressure around the terminal device 10.

The terminal device 10 is equipped with the acceleration sensor 21, gyro sensor 22, barometric pressure sensor 23, etc. described above, and is therefore able to determine the position of the terminal device 10 using techniques such as pedestrian dead-reckoning (PDR) that utilize these sensors 21-23. This makes it possible to obtain indoor position information that is difficult to obtain using positioning systems such as GPS.

For example, a pedometer using the acceleration sensor 21 can calculate the number of steps, walking speed, and distance walked. Furthermore, the gyro sensor 22 can be used to determine the user U's direction of travel, line of sight, and body tilt. Furthermore, the atmospheric pressure detected by the atmospheric pressure sensor 23 can be used to determine the altitude and floor on which the user U's terminal device 10 is located.

The temperature sensor 24 detects, for example, the temperature around the terminal device 10. The sound sensor 25 detects, for example, sound around the terminal device 10. The light sensor 26 detects the illuminance around the terminal device 10. The magnetic sensor 27 detects, for example, the geomagnetism around the terminal device 10. The image sensor 28 captures images of the area around the terminal device 10.

The above-mentioned air pressure sensor 23, temperature sensor 24, sound sensor 25, light sensor 26, and image sensor 28 can detect the environment and situation surrounding the terminal device 10 by detecting air pressure, temperature, sound, and illuminance, and capturing images of the surroundings. Furthermore, the accuracy of the location information of the terminal device 10 can be improved based on the environment and situation surrounding the terminal device 10.

(Control unit 30)
The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM, input/output ports, and various other circuits. The control unit 30 may also be configured with hardware such as an integrated circuit, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 30 includes a transmitting unit 31, a receiving unit 32, and a processing unit 33.

(Transmitter 31)
The transmission unit 31 can transmit, for example, various information input by the user U using the input unit 13, various information detected by each sensor 21 to 28 mounted on or connected to the terminal device 10, and location information of the terminal device 10 measured by the positioning unit 14 to the server device 100 via the communication unit 11.

(Receiving unit 32)
The receiving unit 32 can receive various information provided by the server device 100 and requests for various information from the server device 100 via the communication unit 11 .

(Processing unit 33)
The processing unit 33 controls the entire terminal device 10, including the display unit 12. For example, the processing unit 33 can output various information transmitted by the transmission unit 31 and various information received from the server device 100 by the reception unit 32 to the display unit 12 for display.

(Storage unit 40)
The storage unit 40 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. The storage unit 40 stores various programs, various data, etc.

4. Example of Server Device Configuration
Next, the configuration of the server device 100 according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the configuration of the server device 100 according to the embodiment. As shown in Fig. 5, the server device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

(Communication unit 110)
The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is connected to a network N by wire or wirelessly.

(Storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as an HDD, an SSD, an optical disk, etc. The storage unit 120 may store attribute information and history information (log data) of the user U together with identification information (such as a user ID) indicating the user U.

(Control unit 130)
The control unit 130 is a controller, and is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like, by executing various programs (corresponding to examples of information processing programs) stored in a storage device inside the server device 100 using a storage area such as a RAM as a working area. In the example shown in FIG. 5 , the control unit 130 includes an acquisition unit 131, a determination unit 132, an extraction unit 133, a grouping unit 134, a formation unit 135, an estimation unit 136, a selection unit 137, a generation unit 138, and a provision unit 139.

(Acquisition unit 131)
The acquisition unit 131 acquires a search query input by a user U. For example, when the user U inputs a search query into a search engine or the like to perform a keyword search, the acquisition unit 131 acquires the search query via the communication unit 110. That is, the acquisition unit 131 acquires the keywords input by the user U into the search box of a search engine, website, or app via the communication unit 110.

The acquisition unit 131 also acquires user information about user U via the communication unit 110. For example, the acquisition unit 131 acquires identification information (such as a user ID) indicating user U, location information about user U, attribute information about user U, etc. from user U's terminal device 10. The acquisition unit 131 may also acquire identification information indicating user U, attribute information about user U, etc. when user U is registered. The acquisition unit 131 then stores the user information in the storage unit 120.

The acquisition unit 131 also acquires various types of history information (log data) indicating the behavior of user U via the communication unit 110. For example, the acquisition unit 131 acquires various types of history information indicating the behavior of user U from user U's terminal device 10 or from various servers based on the user ID, etc. Then, the acquisition unit 131 stores the various types of history information in the storage unit 120.

Furthermore, the acquisition unit 131 acquires a screenshot from the user U via the communication unit 110. The acquisition unit 131 also acquires, as a selected comment, a comment selected by the user U from among the candidate comments via the communication unit 110.

Alternatively, the acquisition unit 131 acquires a screenshot and a question from the user U via the communication unit 110. The acquisition unit 131 also acquires an answer to the question from an expert.

(Determination unit 132)
The determination unit 132 determines whether the screenshots acquired from multiple users U are of the same subject.

The determination unit 132 also references past history information to determine whether the acquired screenshot and question text are the same as screenshots and question text acquired in the past.

The determination unit 132 also determines whether the portion of the screenshot taken by the user U matches the content of the question text, and if so, allows the user to make an inquiry to an expert. The determination unit 132 also determines whether the number of users U whose screenshots have the same content exceeds a threshold, and if so, allows the user to make an inquiry to an expert. The determination unit 132 also determines whether there are multiple experts whose area of expertise is the portion of the screenshot taken by the user U, and if there are multiple experts, allows the user to make an inquiry to an expert.

(Extraction unit 133)
If the acquired screenshot and question text are the same as a screenshot and question text acquired in the past, the extraction unit 133 extracts an answer to the acquired screenshot and question text from past history information.

The extraction unit 133 also extracts comments related to the subject of the screenshot as candidate comments from among the comments posted about the subject of the screenshot.

(Grouping unit 134)
The grouping unit 134 groups users U who have taken screenshots of the same object.

(Forming part 135)
The forming unit 135 forms a closed community in which users in the same group can participate.

(Estimation unit 136)
The estimation unit 136 estimates a portion of the screenshot that is likely to be of particular interest to the user U. The estimation unit 136 also estimates the intention of the question from the screenshot and the question text.

(Selection Department 137)
The selection unit 137 selects an expert to answer the question based on the screenshot and the intent of the question posed by the user U. Note that if the portion of the screenshot taken by the user U relates to multiple specialized fields, the selection unit 137 selects an expert for each specialized field.

(Generation unit 138)
The generation unit 138 generates one piece of content by merging a selected comment selected by the user U from among candidate comments related to the subject of the screenshot with the screenshot. At this time, the generation unit 138 generates content in which the selected comment is arranged at a relevant position in the screenshot. In addition, the generation unit 138 generates content in which a portion of the screenshot that is thought to be of particular interest to the user U is emphasized.

The generation unit 138 also generates article-like content by merging screenshots and responses.

(Providing Department 139)
The providing unit 139 provides the screenshot and the question text to the expert to make an inquiry. The providing unit 139 also provides the original text or a link to the subject of the screenshot together with the screenshot and the question text to the expert to make an inquiry.

The providing unit 139 also provides answers from experts to the user U. The providing unit 139 also provides the generated content to other users. The providing unit 139 also provides candidate comments to the user U.

The provider 139 also provides information sharing services within the formed closed community.

5. Processing Procedure
Next, a first processing procedure performed by the server device 100 according to the embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the first processing procedure according to the embodiment. Note that the processing procedure shown below is repeatedly executed by the control unit 130 of the server device 100.

For example, as shown in FIG. 6, the acquisition unit 131 of the server device 100 acquires a screenshot of a news article or the like taken by user U from the terminal device 10 of user U via the communication unit 110 (step S101).

Next, the determination unit 132 of the server device 100 determines whether the screenshots obtained from multiple users U are of the same subject (step S102).

Next, if the screenshots obtained from multiple users U have the same subject (step S102; Yes), the grouping unit 134 of the server device 100 groups users U who have taken screenshots of the same subject (step S103).

Next, the formation unit 135 of the server device 100 forms a closed community that users in the same group can participate in (step S104).

Next, the provider 139 of the server device 100 provides an information sharing service within the formed closed community (step S105).

Note that if the screenshots acquired from multiple users U do not relate to the same subject (step S102; No), the grouping unit 134, creation unit 135, and provision unit 139 of the server device 100 do not perform the processes of steps S103 to S105 described above.

The extraction unit 133 of the server device 100 then extracts comments related to the screenshot target from among the comments posted about the screenshot target as candidate comments (step S106). At this time, the provision unit 139 provides the candidate comments to the terminal device 10 of user U via the communication unit 110. The acquisition unit 131 acquires, from the terminal device 10 of user U via the communication unit 110, the comment selected by user U from the candidate comments as the selected comment.

Next, the generation unit 138 of the server device 100 merges the selected comment selected by the user U from among the candidate comments related to the subject of the screenshot with the screenshot to generate one piece of content (step S107).

Then, the providing unit 139 of the server device 100 provides the generated content to the terminal devices 10 of other users via the communication unit 110 (step S108). In other words, the generated content is shared among users.

Next, a second processing procedure performed by the server device 100 according to the embodiment will be described using FIG. 7. FIG. 7 is a flowchart showing the second processing procedure according to the embodiment. Note that the processing procedure shown below is repeatedly executed by the control unit 130 of the server device 100.

For example, as shown in FIG. 7, the acquisition unit 131 of the server device 100 acquires, via the communication unit 110, a screenshot of a news article or the like taken by user U from the terminal device 10 of user U, and a question regarding the content of the screenshot (step S201).

Next, the determination unit 132 of the server device 100 references past history information and determines whether the acquired screenshot and question text are the same as screenshots and question text acquired in the past (step S202).

Next, if the acquired screenshot and question text are the same as those acquired in the past (step S202; Yes), the extraction unit 133 of the server device 100 extracts an answer to the acquired screenshot and question text from past history information (step S203).

Next, if the acquired screenshot and question text are not the same as those acquired previously (step S202; No), the estimation unit 136 of the server device 100 estimates the intent of user U's question from the screenshot and question text (step S204).

Next, the selection unit 137 of the server device 100 selects an expert to answer the question based on the screenshot and the intent of the user U's question (step S205).

The providing unit 139 of the server device 100 then provides the screenshot and the question text to the expert via the communication unit 110 to make an inquiry (step S206). At this time, the providing unit 139 may also provide the original text or a link to the subject of the screenshot, along with the screenshot and the question text, to the expert via the communication unit 110 to make an inquiry. In addition, the acquiring unit 131 acquires an answer to the question text from the expert via the communication unit 110.

Next, the providing unit 139 of the server device 100 provides the answer from the expert to the user U (step S207).

Next, the generation unit 138 of the server device 100 merges the screenshot and the response to generate article-like content (step S208).

Then, the providing unit 139 of the server device 100 provides the generated content to the terminal devices 10 of other users via the communication unit 110 (step S209). In other words, the generated content is shared among users.

6. Modifications
The terminal device 10 and the server device 100 described above may be implemented in various different forms other than the above-described embodiment. Therefore, modifications of the embodiment will be described below.

In the above embodiment, some or all of the processing performed by the server device 100 may actually be performed by the terminal device 10 (or an app running on the terminal device 10). For example, processing may be completed standalone (by the terminal device 10 alone). In this case, the terminal device 10 is assumed to have the functions of the server device 100 in the above embodiment. Furthermore, in the above embodiment, the terminal device 10 is linked to the server device 100, so from the perspective of the user U, it appears that the processing of the server device 100 is also being performed by the terminal device 10. In other words, from another perspective, the terminal device 10 can also be said to be equipped with the server device 100.

In addition, in the above embodiment, the server device 100 may use image recognition and natural language processing (NLP) to vectorize the text of the screenshot and the text of the comment (candidate comment or selected comment), and determine whether they match based on the cosine similarity of the vectors. The server device 100 may then adopt the comment if they match, or discard the comment if they do not match.

In addition, in the above embodiment, the server device 100 may use image recognition and natural language processing to vectorize the text in the screenshot and the text in the expert's answer, and determine whether they match based on the cosine similarity of the vectors. The server device 100 may then adopt the answer if they match, and discard the answer if they do not match.

In addition, in the above embodiment, the server device 100 may use image recognition and natural language processing to vectorize the question text and the expert's answer text, and determine whether they match based on the cosine similarity of the vectors. The server device 100 may then adopt the answer if they match, and discard the answer if they do not match.

7. Effects
As described above, the information processing device (terminal device 10 and server device 100) according to the first invention of the present application comprises an acquisition unit 131 that acquires a screenshot from a user U, a generation unit 138 that merges a selected comment selected by the user U from among candidate comments related to the subject of the screenshot into the screenshot to generate one piece of content, and a provision unit 139 that provides the generated content to other users.

The information processing device according to the first aspect of the present application further includes an extraction unit 133 that extracts comments related to the target of the screenshot as candidate comments from among comments posted about the target of the screenshot. The provision unit 139 provides the candidate comments to the user U. The acquisition unit 131 acquires a comment selected by the user U from the candidate comments as a selected comment.

The information processing device according to the first aspect of the present application further includes a determination unit 132 that determines whether the subject of screenshots acquired from multiple users U is the same, a grouping unit 134 that groups users U who have taken screenshots of the same subject, and a formation unit 135 that forms a closed community in which users within the same group can participate. The provision unit 139 provides an information sharing service within the closed community.

The generation unit 138 also generates content in which the selected comment is placed at a relevant position on the screenshot.

The information processing device according to the first aspect of the present application further includes an estimation unit 136 that estimates the portion of the screenshot that is likely to be of particular interest to the user U. The generation unit 138 generates content that highlights the portion of the screenshot that is likely to be of particular interest to the user U.

From another perspective, the information processing device (terminal device 10 and server device 100) according to the second invention of the present application includes an acquisition unit 131 that acquires a screenshot and a question sentence from a user U, an estimation unit 136 that estimates the intent of the user U's question from the screenshot and the question sentence, a selection unit 137 that selects an expert to answer the question from the screenshot and the intent of the user U's question, and a provision unit 139 that provides the screenshot and the question sentence to the expert to make an inquiry. The acquisition unit 131 acquires an answer from the expert. The provision unit 139 provides the answer from the expert to the user U.

In addition, the information processing device according to the second aspect of the present invention further includes a generation unit 138 that generates article-like content by merging screenshots and responses.

In addition, if the portion of the screenshot taken by user U relates to multiple specialized fields, the selection unit 137 selects an expert for each specialized field.

The providing unit 139 also provides the original text or a link to the subject of the screenshot, along with the screenshot and question text, to the expert to make an inquiry.

The information processing device according to the second aspect of the present invention further includes a determination unit (132) that references past history information and determines whether the acquired screenshot and question text are the same as previously acquired screenshots and question text, and an extraction unit (133) that, if they are the same, extracts an answer to the acquired screenshot and question text from the past history information.

The information processing device according to the second aspect of the present invention further includes a determination unit 132 that determines whether the portion of the screenshot taken by user U matches the content of the question text, and if so, allows the user to make an inquiry to an expert.

In addition, the information processing device according to the second aspect of the present invention further includes a determination unit 132 that determines whether the number of users U whose screenshots have the same content exceeds a threshold, and if the threshold is exceeded, allows the user to inquire with an expert.

In addition, the information processing device according to the second aspect of the present invention further includes a determination unit 132 that determines whether there are multiple experts whose area of expertise is the portion screenshotted by user U, and, if there are multiple experts, allows the user U to inquire about the experts.

By using any one or a combination of the above processes, the information processing device according to the present application can acquire screenshots on the server side and perform actions triggered by the screenshot.

8. Hardware Configuration
The terminal device 10 and the server device 100 according to the above-described embodiments are realized by a computer 1000 having a configuration as shown in Fig. 8, for example. The following description will be given taking the server device 100 as an example. Fig. 8 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I/F (Interface) 1060, an input I/F 1070, and a network I/F 1080 are connected via a bus 1090.

The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic unit 1030 is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), etc.

The primary storage device 1040 is a memory device, such as RAM (Random Access Memory), that primarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The secondary storage device 1050 may be internal storage or external storage. The secondary storage device 1050 may also be a removable storage medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) memory card. The secondary storage device 1050 may also be cloud storage (online storage), NAS (Network Attached Storage), a file server, etc.

The output I/F 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a display, projector, printer, etc., and is realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (High Definition Multimedia Interface, registered trademark). The input I/F 1070 is an interface for receiving information from various input devices 1020, such as a mouse, keyboard, keypad, buttons, scanner, etc., and is realized by USB, for example.

Furthermore, the output I/F 1060 and the input I/F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. In other words, the output device 1010 and the input device 1020 may be wireless devices.

Furthermore, the output device 1010 and the input device 1020 may be integrated, such as a touch panel. In this case, the output I/F 1060 and the input I/F 1070 may also be integrated as an input/output I/F.

The input device 1020 may be a device that reads information from, for example, optical recording media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memories.

The network I/F 1080 receives data from other devices via the network N and sends it to the computing device 1030, and also transmits data generated by the computing device 1030 to other devices via the network N.

The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output I/F 1060 and the input I/F 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the server device 100, the arithmetic unit 1030 of the computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto the primary storage device 1040. The arithmetic unit 1030 of the computer 1000 may also load a program acquired from another device via the network I/F 1080 onto the primary storage device 1040 and execute the loaded program. The arithmetic unit 1030 of the computer 1000 may also cooperate with other devices via the network I/F 1080 and call and use program functions, data, etc. from other programs on the other devices.

[9. Other]
Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the scope of so-called equivalents. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown.

Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

For example, the above-mentioned server device 100 may be implemented using multiple server computers, and depending on the function, the configuration can be flexibly changed, such as by calling an external platform using an API (Application Programming Interface) or network computing.

Furthermore, the above-described embodiments and variations can be combined as appropriate to the extent that the processing content is not contradictory.

Furthermore, the "parts" (sections, modules, units) mentioned above can be read as "means" or "circuits." For example, an acquisition unit can be read as an acquisition means or an acquisition circuit.

REFERENCE SIGNS LIST 1 Information processing system 10 Terminal device 100 Server device 110 Communication unit 120 Storage unit 130 Control unit 131 Acquisition unit 132 Determination unit 133 Extraction unit 134 Grouping unit 135 Formation unit 136 Estimation unit 137 Selection unit 138 Generation unit 139 Provision unit

Claims (10)

an acquisition unit that acquires a screenshot and a question sentence from a user;
an estimation unit that estimates the intent of the user's question from the screenshot and the question text;
a selection unit that selects an expert to answer the question based on the screenshot and the intent of the user's question;
a providing unit that provides the screenshot and the question text to an expert and makes an inquiry;
Equipped with
the acquisition unit acquires an answer from the expert,
The information processing device, wherein the providing unit provides the answer from the expert to the user. a generation unit that generates article-like content by merging the screenshots and the answers;
The information processing apparatus according to claim 1 , further comprising: The information processing apparatus according to claim 1 , wherein the selection unit selects an expert for each of the specialized fields when the portion screenshotted by the user relates to a plurality of specialized fields. The information processing device according to claim 1 , wherein the providing unit simultaneously provides the screenshot, the question text, and an original text or a link to the subject of the screenshot to the expert to make an inquiry. a determination unit that refers to past history information and determines whether the acquired screenshot and question text are the same as those acquired in the past;
an extracting unit that extracts an answer to the acquired screenshot and question sentence from past history information if they are the same;
The information processing apparatus according to claim 1 , further comprising: a determination unit that determines whether the portion of the screenshot taken by the user matches the content of the question text, and if they match, allows the user to make an inquiry to an expert;
The information processing apparatus according to claim 1 , further comprising: a determination unit that determines whether the number of users whose screenshots are identical exceeds a threshold, and if the number exceeds the threshold, allows the user to inquire with an expert;
The information processing apparatus according to claim 1 , further comprising: a determination unit that determines whether there are multiple experts whose area of expertise is the part that the user has taken a screenshot of, and if there are multiple experts, allows the user to inquire with the experts;
The information processing apparatus according to claim 1 , further comprising: An information processing method executed by an information processing device,
an acquisition step of acquiring a screenshot and a question sentence from a user;
an estimation step of estimating the intention of the user's question from the screenshot and the question text;
a selection step of selecting an expert to answer the question based on the screenshot and the intent of the user's question;
a providing step of providing the screenshot and the question text to an expert and making an inquiry;
Including,
In the obtaining step, an answer is obtained from the expert,
In the providing step, the answer from the expert is provided to the user. an acquisition procedure for acquiring a screenshot and a question text from a user;
an estimation step of estimating the intention of the user's question from the screenshot and the question text;
a selection procedure for selecting an expert to answer the question based on the screenshot and the intent of the user's question;
a providing step of providing the screenshot and the question text to an expert and making an inquiry;
and causing a computer to execute the above.
The obtaining step obtains an answer from the expert;
The information processing program, wherein the providing step provides the answer from the expert to the user.