KR20240077148A - Information search method based on search action and gaze of user - Google Patents

Abstract

Disclosed are a method and device for searching for information using a user's navigation actions and gaze. The disclosed information search method comprises the steps of: providing, in response to a user's request to navigate to targeted information, information navigated from a navigation space into a current navigation window; collecting the user's navigation actions for the navigated information displayed in the current navigation window; updating, in response to the navigation actions, an information gain for information included in the navigation space; and providing, in response to the navigation action, information selected from information included in the navigation space based on the updated information gain into a next navigation window.

Description

Information search method and device based on user's gaze and search action {INFORMATION SEARCH METHOD BASED ON SEARCH ACTION AND GAZE OF USER}

The present invention relates to an information search method and device, and more specifically, to an information search method and device using user search actions and gaze.

A lot of research and development is being done on how users can accurately search for the data they are looking for.

The BIG (Bayesian Information Gain) framework is an information search method based on Bayesian Theory. Based on Bayesian theory, the BIG framework provides the next view in which information gain is maximized by updating the probability that information becomes a target and calculating information gain for all combinations of information.

Recently, the BIG framework has been applied to navigation and file search, and various applications such as BIGnav and BIGfile have been developed.

Related prior documents include the patent documents, Republic of Korea Patent Nos. 10-1683224, 10-0850776, and 10-2443797, and the non-patent documents “Liu, W., d'Oliveira, R. L., Beaudouin-Lafon, M. , &Rioul, O. (2017, May): Bayesian information gain for guiding multiscale navigation. , O., Mcgrenere, J., Mackay, W. E., &Beaudouin-Lafon, M. (2018, April): Bayesian information gain for fast file retrieval In　Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems　(pp. 1-13)".

The present invention is intended to provide an information search method and device that can provide search results with a high probability of matching the user's search target using the user's search action.

Additionally, the present invention is intended to provide an information search method and device that can provide search results with a high probability of matching the user's search target using the user's gaze information.

According to an embodiment of the present invention for achieving the above object, the steps include: providing information searched in a search space to a current search window according to a user's search request for target information; collecting the user's search action for the searched information displayed in the current search window; updating information gains for information included in the search space according to the search action; and providing information selected according to the updated information gain from among the information included in the search space to a next search window.

In addition, according to another embodiment of the present invention for achieving the above object, the steps include: providing information searched in the search space to the current search window according to the user's search request for target information; collecting the user's gaze information on the searched information displayed in the current search window; updating information gains for information included in the search space according to the gaze information; and providing information selected according to the updated information gain from among the information included in the search space to a next search window.

In addition, according to another embodiment of the present invention for achieving the above object, a memory; and at least one processor electrically connected to the memory, wherein the processor provides information searched in the search space to a current search window according to a user's search request for target information, and provides the information searched in the search space to the current search window, An information search device is provided that collects the user's search action and gaze information for searched information, and updates information gains for information included in the search space according to the search action and gaze information.

According to one embodiment of the present invention, the information gain for reference information included in the search space is updated using the user's search action or gaze information, and the searched information is provided according to the updated information gain, so that the user Search results with a high probability of matching the user's search target can be obtained.

1 is a diagram for explaining an information search system according to an embodiment of the present invention.
Figure 2 shows an example of search information displayed in a search window.
Figure 3 is a diagram for explaining an information search method according to an embodiment of the present invention.
Figure 4 is a diagram for explaining an information search method according to another embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention. While describing each drawing, similar reference numerals are used for similar components.

When a user performs a search action for target information, the BIG framework obtains a combination of all information included in the search space and calculates the information gain for the target information for each combination. Information gain is an indicator related to uncertainty in the information search process and is calculated from the target matching probability value for all information included in the search space. Among the information included in the search space, information with high information gain can be considered to have a high possibility of corresponding to the user's target information, that is, search target information.

Since the user's search action is an action to search for target information desired by the user, the more the information gain is updated according to the user's search action, the more likely it is that the user's target information will be provided as a search result. In light of this, the present invention proposes an information search method based on the user's search action. Search actions may include actions such as selecting or deselecting information displayed in the search window, scrolling up, or scrolling down.

Additionally, users not only use input devices such as a mouse or keyboard to search for information, but also constantly look at the search window and search for information corresponding to target information. Accordingly, the present invention proposes an information search method that considers not only the search action but also the user's gaze.

Depending on the embodiment, the search action and the user's gaze information may be used together or selectively in the information search process.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a diagram for explaining an information search system according to an embodiment of the present invention, and Figure 2 shows an example of search information displayed in a search window.

Referring to FIG. 1, the information search system according to an embodiment of the present invention includes a client terminal 110, an eye tracking device 120, an information search device 130, and a database 140.

The user uses the client terminal 110 to request the information search device 130 to search for target information. The information search device 130 searches for information in the search space using information gains for information included in the search space, and then provides the searched information to the client terminal 110. The searched information is displayed in the search window of the client terminal 110, as shown in FIG. 2. Figure 2 shows an embodiment in which searched information is displayed in a search window when the user's target information is font information or GUI image.

The user gazes at the searched information displayed in the search window and performs various search actions to obtain target information. Users can select information displayed in the navigation window, take navigation actions such as deselecting information that has already been selected, or scroll up or down the current navigation window to check new information displayed in the next navigation window. there is. Information selected by the user can be saved in a separate folder, and the user can later check the information stored in the folder in detail.

The database 140 stores reference information included in the search space. As shown in FIG. 2, information included in the search space may be information in the form of text or images, such as fonts, as an example, and such reference information may be expressed and stored as vectors. Information in the form of text or images may be converted into vectors using a preset vector conversion algorithm. For example, information in the form of text or images may be converted into vectors using a non-negative matrix factorization (NMF) algorithm. By converting the reference information into vectors, the similarity required for later information gain update can be easily calculated.

The gaze tracking device 120 tracks the gaze of a user looking at information displayed in the search window of the client terminal 110 and generates gaze information about the user's gaze movement. The generated gaze information is transmitted to the information search device 130.

The information search device 130 may provide information searched in the search space to the client terminal 110 according to the information gain, and collect at least one of the user's search action and gaze information. The information search device 130 may collect the user's search action performed through the input device of the client terminal 110 and collect gaze information from the eye tracking device 120. The information search device 130 updates the information gain using at least one of the collected search action and gaze information, and provides newly searched information to the client terminal 110 according to the updated information gain.

The information search device 130 includes a memory and at least one processor electrically connected to the memory, and the processor performs a series of processes for searching information and updating information gain. In one embodiment, the processor provides information searched in the search space to the current search window according to the user's search request for target information. Additionally, the user's search action and gaze information regarding the searched information currently displayed in the search window is collected. Additionally, the information gain for information included in the search space is updated according to the search action and gaze information.

The information search device 130 may continuously update the information gain according to periodically collected search action and gaze information.

According to one embodiment of the present invention, the information gain for reference information included in the search space is updated using the user's search action or gaze information, and the searched information is provided according to the updated information gain, so that the user Search results with a high probability of matching the user's search target can be obtained.

Figure 3 is a diagram for explaining an information search method according to an embodiment of the present invention.

The information search method according to an embodiment of the present invention may be performed on a computing device including a processor and memory, and the information search device described above may be an example of a computing device.

Referring to FIG. 3, the computing device according to an embodiment of the present invention provides information searched in the search space to the current search window in response to the user's search request for target information (S310). The computing device may provide information corresponding to the greatest information gain among the information included in the search space to the current search window. And, for example, if the user's target information is font information, the computing device can search for various reference font information in the search space and provide it in the current search window.

The computing device collects the user's search action for the searched information currently displayed in the search window (S320). As described above, the search action may include a selection action or deselection action for the searched information, or a scroll up action, a scroll down action, etc.

The computing device updates the information gain for the information included in the search space according to the collected search action (S330). Since the user's search action is performed in a direction corresponding to the information the user seeks, the computing device uses the user's search action to update the information gain so that information matching the user's search intention can be provided to the user. .

As an example, in step S330, the computing device calculates a probability value that at least one of the searched information, information corresponding to a search action, corresponds to target information. And the information gain can be updated using the calculated probability value. The information gain is updated for all information included in the search space.

When the user's search action is a selection action or a deselection action, the computing device calculates a probability value that the information selected or deselected by the user corresponds to target information, and updates the information gain using the calculated probability value. You can.

As an example, the computing device may calculate vector similarity between information corresponding to the search action and information in the search space excluding the information corresponding to the search action, and calculate a probability value according to the vector similarity. As an example, vector similarity may correspond to the Euclidean distance between vectors, and the smaller the Euclidean distance, the higher the similarity.

As an example, the computing device uses [Equation 1] to calculate the probability that the selected information corresponds to the target information ( ) can be calculated. The computing device can calculate the probability that information corresponding to the selection action corresponds to target information by applying vector similarity to the softmax function, as shown in [Equation 1].

And, as an example, the computing device calculates a probability value ( ) can be calculated. As in [Equation 2], the computing device can calculate the probability that information corresponding to the deselection action corresponds to target information by subtracting the value obtained by applying the vector similarity to the softmax function from 1.

In [Equation 1] and [Equation 2], indicates selected or unselected information, represents the remaining information excluding the selected or deselected information among the information in the search space. i represents the index of the remaining information in the search space represents the vector similarity between the information corresponding to the search action and the remaining information in the search space. and represents information corresponding to the user's navigation action at time t, represents target information.

Meanwhile, when the user's search action is a scroll up action and a scroll down action, the computing device may determine a preset constant value as a probability value depending on whether the searched information is included in the target information. If information selected by the user exists among the searched information currently displayed in the search window, the computing device may determine that the searched information is included in the target information.

As an example, the computing device uses [Equation 3] and [Equation 4] to calculate the probability value ( , ) can be determined.

When the user takes a scroll up action, the computing device sets 0.55 if the searched information is included in the target information, 0.45 if the searched information is not included in the target information, and the searched information corresponds to the target information. It can be determined by the probability value. And when the user takes a scroll down action, the computing device sets 0.45 if the searched information is included in the target information, 0.55 if the searched information is not included in the target information, and 0.55 if the searched information is not included in the target information. It can be determined by the corresponding probability value. The probability values 0.45 and 0.55 may vary depending on the embodiment.

In step S330, the computing device calculates the information gain ( ) can be updated. [Equation 5] is a general equation for calculating information gain.

here, When the information (X) displayed in the search window is x, the target information ( ) represents the entropy for . and represents the entropy before the user takes the search action y. Entropy is calculated according to Shannon's entropy function as in [Equation 6].

[Equation 6] Corresponds to the probability values of [Equation 1] to [Equation 4], and therefore, the information gain for the reference information of the search space can be updated through the probability values of [Equation 1] to [Equation 4]. .

The computing device provides information selected according to the information gain updated in step S330 from among the information included in the search space to the next search window (S340). For example, when the user takes a scroll up or scroll down action, the computing device provides new information other than the information currently displayed in the navigation window to the next navigation window, and at this time, the information provided to the next navigation window is the updated information benefit. It is decided according to. In one embodiment, information selected in order of highest information gain may be provided to the next search window.

Figure 4 is a diagram for explaining an information search method according to another embodiment of the present invention.

Unlike the above-described embodiment, the computing device according to one embodiment of the present invention updates the information gain using the user's gaze information.

Referring to FIG. 4, the computing device according to an embodiment of the present invention provides information searched in the search space to the current search window in response to the user's search request for target information (S410). Then, the computing device collects the user's gaze information on the searched information currently displayed in the search window (S420).

And the computing device updates the information gain for the information included in the search space according to the collected gaze information (S430). Since the user's gaze moves along the information the user is looking for, the computing device uses the user's gaze information to update the information gain so that information matching the user's search intention can be provided to the user.

In step S430, the computing device uses gaze information to detect a gaze event of interest, and calculates a probability value that at least one of the searched information, information corresponding to the gaze event of interest, corresponds to target information. Then, the information gain is updated using the calculated probability value.

The computing device can detect an attentive gaze event by comparing various analysis indicators with a preset threshold. The attention gaze event corresponds to an event in which the information the user is paying attention to occurs among the information currently displayed in the navigation window, and includes the viewing time, fixation duration, and gaze fixation point. At least one of the number of (fixation count) and the number of revisits (revisit count) can be used as an analysis indicator. Here, the number of gaze fixation points corresponds to the gaze fixation time divided by a preset constant, and the number of revisits is the number of times increased when the user gazes at the information already gazed again after gazing at other information. And since the gaze pattern is different for each user, the threshold may be set differently depending on the user.

The computing device may calculate vector similarity between information corresponding to the attention gaze event and the remaining information in the search space, and calculate a probability value according to the vector similarity. That is, the computing device may calculate a probability value according to the vector similarity between the information in the search space excluding the information that the user focused on and the information that the user focused on.

The computing device can apply the vector similarity to the softmax function to calculate the probability that the information corresponding to the attention gaze event corresponds to the target information. As an example, the probability value ( ) can be calculated.

here, represents information corresponding to the attention gaze event, represents the remaining information in the search space.

[Equation 6] Corresponds to the probability value of [Equation 7], and therefore, the information gain for the reference information of the search space can be updated through the probability value of [Equation 7].

Then, the computing device provides information selected according to the updated information gain among the information included in the search space to the next search window (S440).

The technical contents described above may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiments or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. A hardware device may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , those skilled in the art can make various modifications and variations from this description. Accordingly, the spirit of the present invention should not be limited to the described embodiments, and the scope of the patent claims described later as well as all things that are equivalent or equivalent to the scope of this patent claim shall fall within the scope of the spirit of the present invention. .

Claims (14)

providing information searched in the search space to the current search window according to the user's search request for target information;
collecting the user's search action for the searched information displayed in the current search window;
updating information gains for information included in the search space according to the search action; and
Providing information selected according to the updated information gain from among the information included in the search space to the next search window.
Information seeking method including.
According to clause 1,
The step of updating the information gain is
calculating a probability value that at least one of the searched information, information corresponding to the search action, corresponds to the target information; and
Updating the information gain using the probability value
Information seeking method including.
According to clause 2,
The information included in the search space is,
This is information expressed as a vector,
The step of calculating the probability value corresponding to the target information is
calculating vector similarity between information corresponding to the search action and remaining information in the search space; and
Calculating the probability value according to the vector similarity
Information seeking method including.
According to clause 3,
The search action is
A selection action or deselection action for the searched information
How to seek information.
According to clause 4,
The step of calculating the probability value is
Applying the vector similarity to the softmax function to calculate a probability value that the information corresponding to the selection action corresponds to the target information
How to seek information.
According to clause 4,
The step of calculating the probability value is
Calculating a probability value that information corresponding to the deselection action corresponds to the target information by subtracting the value obtained by applying the vector similarity to the softmax function from 1.
How to seek information.
According to clause 2,
The search action is
Includes scroll up action and scroll down action,
The step of calculating the probability value corresponding to the target information is
Depending on whether the searched information is included in the target information, a preset constant value is determined as the probability value.
How to seek information.
providing information searched in the search space to the current search window according to the user's search request for target information;
collecting the user's gaze information on the searched information displayed in the current search window;
updating information gains for information included in the search space according to the gaze information; and
Providing information selected according to the updated information gain from among the information included in the search space to the next search window.
Information seeking method including.
According to clause 8,
The step of updating the information gain is
Using the gaze information, detecting a gaze event of interest;
calculating a probability value that at least one of the searched information, information corresponding to the attention gaze event, corresponds to the target information; and
Updating the information gain using the probability value
Information seeking method including.
According to clause 9,
The step of detecting the attention gaze event is
Detecting the attention gaze event by comparing at least one of the user's gaze fixation time, the number of gaze fixation points, and the number of gaze revisits with a preset threshold.
The step of calculating the probability value is
According to clause 9,
The information included in the search space is,
This is information expressed as a vector,
The step of calculating the probability value corresponding to the target information is
calculating vector similarity between information corresponding to the attention gaze event and remaining information in the search space; and
Calculating the probability value according to the vector similarity
Information seeking method including.
According to clause 11,
The step of calculating the probability value is
Applying the vector similarity to the softmax function to calculate a probability value that information corresponding to the attention gaze event corresponds to the target information
How to seek information.
Memory; and
Includes at least one processor electrically connected to the memory,
The processor is
According to the user's search request for target information, the information searched in the search space is provided to the current search window,
Collecting the user's search action and gaze information about the searched information displayed in the current search window,
According to the search action and gaze information, information gain for information included in the search space is updated.
Information seeking device.
According to clause 13,
The information included in the search space is
Information in the form of text or images
Information seeking device.