JP2008210024A - Document set analysis apparatus, document set analysis method, program implementing the method, and recording medium storing the program - Google Patents

Abstract

In a document set analyzing apparatus, documents issued at different dates and times can be judged to have high relevance if the correlation between contents is high.
A document set specifying unit that specifies a document set according to a specific condition, a content similarity evaluation unit that evaluates a content similarity between documents included in the document set, and each of the documents included in the document set A time stamp similarity evaluation unit 30 that evaluates time similarity between documents, a relationship extraction unit 40 that extracts a relationship between documents based on the content similarity and the time similarity, and a relationship between the documents Based on the centrality determination 50 for calculating the centrality of the document, the relationship between the obtained documents, and the centrality of each document, topic words included in the document set from the entire document set , An information analysis 100 that identifies a document set related to a topic word, a role of the document in the document set, and an information output unit 110 that visualizes and outputs the specified document set.
[Selection] Figure 1

Description

  The present invention relates to analysis of a collection of a large number of documents existing in a computer or accessible via a computer network, in particular, a document with a time stamp, and a document set analysis apparatus, a document set analysis method, and a program implementing the method And a recording medium storing the program.

  Currently, techniques for performing search and data mining on a document set (also referred to as a document data set) including text with a date stamp, such as a web page, a blog article, a news article, and the like, are widely known.

  In this technology, when a user handles a large amount of documents, a document that the user wants to know the main topics existing in the document collection or wants to access a group of information related to a specific topic in the document collection. Often has a request to get information related to.

  The following methods are known as implementation methods for realizing these requirements.

  One is a method using a clustering algorithm (see, for example, Non-Patent Document 1). In this method, documents related to similar topics (or topic words) are expressed by expressing each document as a word vector and using similarities between vectors (such as cosine similarity) to integrate similar vectors. Identify as a cluster. The above-described request is realized by regarding each individual cluster as a set of information related to a specific topic.

  On the other hand, when there are a large number of documents with time stamps, the user has a requirement that “information is organized in a form that is conscious of the time stamps and that it is easy to access”.

  As a method for realizing this, there is a method (see, for example, Non-Patent Document 2) in which a topic of a document set is specified while being aware of time series. This method introduces the concept of forgetting, builds a model where old articles are less similar to any new document, and new articles are more similar to each other, and identifies the topics in the latest article. ing.

  Non-Patent Document 3 proposes a clustering method that introduces time information when events are extracted from news articles, thereby improving the accuracy of news article clustering and, consequently, classifying events. Has been reported to improve.

  Patent Document 1 discloses a search using time information.

As a related technique, a method of calculating the centrality of each node (document) by regarding the document as a node (for example, PageRank (for example, see Non-Patent Document 4)) is known. In addition, a search engine (for example, see Non-Patent Document 5) existing on a web for specifying a set of documents is also widely known. A technique for expressing a document as a word vector (for example, see Non-Patent Document 6) is also widely known.
JP 2001-117941 A (Patent 3425906) D. Cutting, D.C. Karger, J .; Pedersen, and J.M. Tukry, “Scatter / Gather: a cluster-based approach to browsing large document collections”, Proc. of SIGIR 1992, ACM, June 1992, pp 318-329. Yoshiharu Ishikawa, Hiroyuki Kitagawa, “Improved Clustering Method Based on the Concept of Forgetting”, The Database Society of Japan Letters Vol. 2, no. 3,2003 Y. Yang, T .; Pierce, and J.M. Carbonell, “A Study on Retrospective and On-Line Event Detection”, Proc. of SIGIR 1998 S. Brin, and L.L. Page, “The anatomy of a large-scale hypertextual Web Search Engine”, Proc. of WWW7, Elsevier science, April 1998, pp 107-117. NTT Resonant Co., Ltd., “Portal Site goo”, [online], 2007, NTT Resonant Co., Ltd., [Search February 22, 2007], Internet <URL: http: // / Www. goo. ne. jp /> Kita Kenji, Tsuda Kazuhiko, Isogo Miki, “Information Retrieval Algorithm”, Kyoritsu Shuppan, January 2002.

  However, the above prior art has the following problems.

  The method of Non-Patent Document 1 is basically a method based only on the contents of a document, and even when a document having a time stamp exists, it is not possible to organize the document in consideration of the document. For example, since the “Wangan War” and the “Iraq War” are composed of very close topics considering only the document contents, it is very difficult to distinguish two documents related to two topics.

  In addition, the methods of Non-Patent Documents 2 and 3 realize information organization based on closeness of information by being aware of time stamps. However, these methods are based on clustering on the assumption that all documents belong to any cluster. On the other hand, in actual data, there are documents belonging to so-called “others” that are not related to other documents, so that appropriate clustering cannot always be performed, and as a result, there is a lot of noise in response to the above requirements, etc. Problem arises.

  In Patent Document 1, time information is used for filtering. In other words, the time information is used such that the time difference is within the threshold value and is related, and when the time difference exceeds the threshold value, there is no relationship. For this reason, documents that have a time difference that exceeds the threshold have the disadvantage that they are uniformly judged to be `` unrelated '', and even when documents issued at different times in time are highly correlated with each other, There was a problem that it could not be determined that the relevance was high.

  In view of the above-mentioned points, the present invention considers the closeness of time stamps between documents as a criterion for similarity between documents, so that there is a document that includes time-conscious document organization and others. Even in the situation, not only is it affected, but only the document set that has strong connection exists is identified, and not only the document set is classified into a cluster, but also what role each document in the cluster has in the cluster, A set of documents whose purpose is to be able to clearly analyze whether it is positioned, and even if documents issued at different times in time are highly correlated with each other, the correlation is high. It is an object of the present invention to provide an analysis apparatus, a document set analysis method, a program that implements the method, and a recording medium that stores the program.

  In order to solve the above-mentioned problem, in the present invention, based on a document set specified based on a request from a user, a content similarity between documents and a temporal similarity based on temporal proximity are evaluated, The relationship between documents is specified based on these two similarities. Based on this relationship, the centrality of each document is evaluated. By using both the relationship between documents and the centrality of individual documents, it is possible to detect specific topics in a document set, cluster documents belonging to a specific topic, and clarify the positioning of each document in the cluster. Realize.

  In other words, the document set analysis apparatus according to claim 1 is a document set analysis apparatus that specifies a role of a document based on a relationship between documents with time stamps in the document set managed by the document data management unit. The document set specifying means for specifying the document set based on the document set specifying condition input from the input means, and the content similarity regarding the topic word between each document included in the specified document set is evaluated. Content similarity evaluation means, time stamp similarity evaluation means for evaluating time similarity regarding topic words between the documents included in the specified document set, and content similarity evaluated by the content similarity evaluation means And a relationship extracting means for extracting a relationship between documents based on the time similarity evaluated by the time stamp similarity evaluating means, and the relationship extracting means Therefore, based on the relationship between the extracted documents, the centrality determination unit that calculates the centrality of the document as an index indicating the high degree of relationship between the document and the document other than the document, and the relationship extraction unit Based on the relationship between the obtained documents and the centrality of each document obtained by the centrality determination means, topic words included in the document set from the entire document set, a document set related to the topic word, Information analysis means for specifying the role of a document in the document set, topic words included in the document set from the entire specified document set, a document set related to the topic word, and a role of the document in the document set And an information output means for visualizing and outputting the document set based on the information.

  The document set analysis method according to claim 5 is a document set analysis method for specifying a role of a document based on a relationship between documents with time stamps in the document set managed by the document data management means. The document set specifying unit includes a document set specifying step for specifying the document set based on the document set specifying condition input from the input unit, and a content similarity evaluation unit includes each of the contents included in the specified document set. A content similarity evaluation step for evaluating the content similarity regarding the topic word between the documents, and a time when the time stamp similarity evaluation means evaluates the time similarity regarding the topic word between the documents included in the specified document set The stamp similarity evaluation step, and the relationship extraction means includes the content similarity evaluated by the content similarity evaluation step and the time stamp similarity evaluation. A relationship extraction step for extracting a relationship between documents based on the temporal similarity evaluated by the step, and a centrality determination means, wherein the centrality determination means is based on the relationship between documents extracted by the relationship extraction step. As an index indicating the degree of relevance between documents other than documents, a centrality determination step of calculating the centrality of the document, and an information analysis means, the relationship between documents obtained by the relationship extraction step, Based on the centrality of each document obtained in the centrality determination step, the topic word included in the document set, the document set related to the topic word, and the role of the document in the document set are specified from the entire document set. And an information output means comprising: a topic word included in the document set from the whole specified document set, a document set related to the topic word, The document set based on the role of the book is characterized by comprising an information output step of outputting visualized, the.

  According to the above configuration, the relationship between documents is extracted based on the content similarity and time similarity, and the role of the document is specified. When is high, it can be determined that the relevance is high.

  Further, since the document group is analyzed in consideration of the document creation date and time in addition to the similarity of the document contents, the analysis accuracy is improved. For example, in the case of a news article, a group of documents that are frequently related but have a time stamp difference, such as an article at the time of the incident and an article at the time of the trial. For example, newspaper articles) can be classified and searched with high accuracy.

  In the present invention, based on the relationship between documents and the centrality value of each document, the document set is regarded as a three-dimensional graph structure, and by specifying a vertex or a group of nodes in the shape, Detection of a specific topic in a document set, clustering of documents belonging to a specific topic, and clarification of the position of each document in the cluster are realized.

  That is, the document set analysis apparatus according to claim 2 is the document set analysis apparatus according to claim 1, wherein the information analysis unit is configured to determine whether the information is based on the relationship between the documents and the centrality of each document. A graph structure construction means for expressing the relationship between documents in two-dimensional coordinates, expressing the centrality in the third dimension of the two-dimensional coordinates, and expressing the document set in a three-dimensional graph structure; From the obtained graph structure, a vertex node extracting means for extracting, as a vertex, a node having a higher centrality than nodes other than itself connected by an edge, and in a direction in which the centrality decreases from the obtained vertex node. A mountain-shaped node group identifying means for tracing a graph structure and identifying a mountain composed of nodes, and a labeling hand for assigning a label indicating a role to the node based on the identified vertex node and mountain-shaped node It is characterized in that it comprises, when.

The document set analysis method according to claim 6 is the document set analysis method according to claim 5, wherein the information analysis step is based on the relationship between the documents and the centrality of each document. A graph structure construction step of expressing the relationship between documents in two-dimensional coordinates, expressing the centrality in the third dimension of the two-dimensional coordinates, and expressing the document set in a three-dimensional graph structure; From the obtained graph structure, a vertex node extraction step for extracting, as a vertex, a node having a higher centrality than nodes other than itself connected by an edge, and a graph in a direction in which the centrality decreases from the obtained vertex node. A mountain-shaped node group identification step for tracing a structure and identifying a mountain composed of nodes;
And a labeling step of assigning a label indicating a role to the node based on the specified vertex node and mountain-shaped node.

  According to the above configuration, a graph structure based on the relationship between documents can be acquired.

  Further, the document set analysis apparatus according to claim 3 is the document set analysis apparatus according to claim 1 or 2, wherein the time stamp similarity evaluation unit is configured to respond to a time stamp difference between the documents. It is characterized by evaluating temporal similarity using a continuously changing forgetting function.

  The document set analysis method according to claim 7 is the document set analysis method according to claim 5 or 6, wherein the time stamp similarity evaluation step is performed according to a time stamp difference between the documents. It is characterized by evaluating temporal similarity using a continuously changing forgetting function.

  According to the above configuration, temporal proximity can be used as a continuous value.

  Further, the document set analysis apparatus according to claim 4 is the document set analysis apparatus according to any one of claims 1 to 3, wherein the relation extraction unit is configured to compare the content similarity and the time stamp similarity. It is characterized in that the relationship between documents is extracted based on the time similarity obtained by performing predetermined weighting on the time similarity evaluated by the sex evaluation means.

  The document set analysis method according to claim 8 is the document set analysis method according to any one of claims 5 to 7, wherein the relation extraction step includes the content similarity and the time stamp similarity. It is characterized in that the relationship between documents is extracted based on the temporal similarity obtained by performing predetermined weighting on the temporal similarity evaluated in the sex evaluation step.

  According to the above configuration, since the temporal similarity is weighted, the similarity does not become excessively dependent only on temporal proximity.

  A document set analysis program according to claim 9 is characterized in that the document set analysis method according to any one of claims 5 to 8 is described as a computer program executable by a computer.

  According to the above configuration, the document set analysis method according to any one of claims 5 to 8 can be described as a computer program.

  A recording medium according to a tenth aspect is characterized in that the document set analysis method according to any one of the fifth to eighth aspects is described as a computer program executable by a computer, and the computer program is recorded. It is said.

  According to the above configuration, a computer program that implements the document set analysis method according to any one of claims 5 to 8 can be recorded on a recording medium.

(1) According to the invention described in claims 1 to 10, the role of the document can be clearly analyzed based on the relevance of the document obtained by the combination of the content similarity and the time similarity.

  In particular, it is possible to determine that relevance is high even if the documents are issued at different dates and times with high time correlation.

Further, since the document group is analyzed in consideration of the document creation date and time in addition to the similarity of the document contents, the analysis accuracy is improved. For example, in the case of a news article, a group of documents that are frequently related but have a time stamp difference, such as an article at the time of the incident and an article at the time of the trial. For example, newspaper articles) can be classified and searched with high accuracy.
(2) According to the inventions of claims 2 and 6, it is possible to specify only a document set having a strong connection.
(3) According to the third and seventh aspects of the invention, the temporal proximity can be used as a continuous value.
(4) According to the inventions described in claims 4 and 8, since the time similarity is weighted, the similarity does not become excessively dependent only on temporal proximity.
(5) According to the invention described in claim 9, it is possible to provide a computer program that implements the document set analysis method described in any one of claims 5 to 8.
(6) According to the invention described in claim 10, it is possible to provide a recording medium recording a computer program in which the document set analysis method described in any one of claims 5 to 8 is implemented.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The document set analysis apparatus according to the present embodiment identifies topics (that is, topic words) existing in the searched news articles, clusters the documents related to the identified topics, and further converts the documents in the clusters into It is a device that performs document analysis to clarify the position of each document.

  The configuration of the document set analysis apparatus in this embodiment will be described with reference to FIG. The document set analysis apparatus includes a document DB (Database) 1 as document data management means, a document set specification section 10 as document set specification means, a content similarity evaluation section 20 as content similarity evaluation means, and a time stamp similarity evaluation. Time stamp similarity evaluation unit 30 as means, relationship extraction unit 40 as relationship extraction means, centrality determination unit 50 as centrality determination means, information analysis unit 100 as information analysis means, information output as information output means The unit 110 is configured. Further, the information analysis unit 100 includes a graph structure construction unit 60 as a graph structure construction unit, a vertex node extraction unit 70 as a vertex node extraction unit, a mountain node group identification unit 80 as a mountain node group identification unit, and a labeling unit. As shown in FIG.

  The document set specifying unit 10 accesses the document DB 1 based on a specification or request including a document set specifying condition for specifying a document set (for example, a specification or request from a user) or a predetermined document set specifying condition. A document set composed of a plurality of documents is specified. The document set specifying condition may be input by an input means (for example, a keyboard device) provided in advance.

  The content similarity evaluation unit 20 evaluates the content similarity between the documents in the document set with respect to the topic (or topic word). For example, the similarity between documents can be determined by evaluating each document with a word vector and using the cosine similarity (for example, see Non-Patent Document 1), or constructing a language model based on one document and the other document. It is conceivable to perform an evaluation based on the language model of how much probability is generated from the language model.

The time stamp similarity evaluation unit 30 evaluates the temporal proximity between the documents in the document set. The temporal proximity between documents is evaluated based on the difference between time stamps. As the proximity evaluation value, it is conceivable to adopt a method in which the difference between the time stamps is used as a distance, or an index that emphasizes the difference between the latest time stamps and does not increase much even if the distance is more than a certain distance. .
As an example, an index based on the assumption that “similarity decreases at a certain rate whenever the time stamp between documents deviates by a certain time” can be considered. This assumption can be expressed by the following equation (1).

  Here, λ is a constant indicating the degree of decrease in similarity due to forgetting, t is a difference between time stamps of two documents, and TimeWeight (t) is a function indicating the degree of forgetting, and the time of two documents The time similarity is shown when the stamp difference is t.

  Solving this ordinary differential equation, the time similarity TimeWeight (t) when the time stamp difference is t can be expressed by the following equation (2).

Here, T0 is a weight when the time stamp difference is 0, and the time similarity gradually decreases as the time stamp difference t increases, and finally approaches zero as much as possible.
However, in the equation as it is, the value indicating the degree of decrease in similarity due to forgetting is not intuitive, so the equation (3) is modified as follows and the parameter indicating the degree of decrease is set to t1 / 2.

In the above equation, t _1/2 represents the difference in time stamps (half-life) at which the time similarity is 50% due to forgetting. FIG. 3 shows the forgetting function when T0 = 1 and t _1/2 = 10.

  The relationship extraction unit 40 evaluates the similarity between documents based on the content similarity and temporal proximity evaluated by the two similarity evaluation functions (the content similarity evaluation unit 20 and the time stamp similarity evaluation unit 30). Calculate and identify the relationship of whether there is a relationship between documents.

  As an example of the inter-document similarity sim (i, j) using two similarities, a technique such as the following equation (4) is conceivable.

  Here, sim ′ (i, j) indicates the similarity based on the document contents of the documents i and j, and α is a parameter for adjusting the weight of the temporal similarity.

  This method is basically a method based on the similarity of contents, and is a method that does not become a similarity that depends only on temporal proximity. FIG. 4 shows how the weight based on the time similarity is applied according to the equation (4).

  In FIG. 4, (a) shows a case where the time stamp difference t (i, j) changes, and (b) shows a case where the content similarity (sim '(i, j)) is different.

  Next, the extraction of the relationship between documents when this similarity between documents is used will be described. When the relationship between documents is expressed as a matrix A, it is conceivable to define it as the following formula (5).

  Here, TopSimp (i) indicates a set of documents included in p documents having a high similarity to the document i. In general, when all the similarities are used, since a low similarity tends to be noise, a link is set only between documents having a high similarity. Sim (i, j) indicates the cosine similarity between the document i and the document j when the document is expressed as a word vector with log tf-idf weight (for example, Non-Patent Document 6). The log tf-idf weight is a weight of each element when each document is expressed by a vector.

  Furthermore, when all the similarities are used as described above, there is a link having a clearly smaller weight than other links. Therefore, it is conceivable to remove a link that transitions with a very low probability among outlinks. This operation is expressed by the following equation (6).

  Here, l i, q represents the total value of the transition probabilities obtained by arranging the outlinks from the node i in descending order of the transition probabilities and adding them until the threshold q is exceeded. TopLinkq (i) indicates a set of link destination nodes of the links to be added at that time.

  The centrality determination unit 50 considers the relationship between documents obtained by the relationship extraction unit 40 as a node, and uses the relationship between documents as an edge with a weight (ie, similarity between documents). And the centrality of each node (document) is calculated. Note that the above-described centrality may use a simple method for calculating the number of links, PageRank (see Non-Patent Document 4), or the like.

  The graph structure construction unit 60 converts the graph structure indicating the relationship between documents based on the relationship between the documents obtained by the centrality determination unit 50 and the centrality score of each document into a two-dimensional plane (for example, xy). A three-dimensional graph structure is constructed in which the centrality score of each document is assigned to the third dimension (eg, z-axis). FIG. 5 is a conceptual diagram of this three-dimensional graph structure. Note that FIG. 5 will be described later.

  The vertex node extraction unit 70 is higher than the nodes other than itself (one-to-one correspondence with the document) connected from the graph structure constructed by the graph structure construction unit 60 by the edges (ie, edges) connecting the nodes. A node with centrality is extracted as a vertex.

  The mountain-shaped node group identification unit 80 traces the graph structure from the vertex node extracted by the vertex node extraction unit 70 in the direction in which the centrality is lowered, and identifies a mountain composed of the nodes. That is, the documents are clustered by the processing up to the mountain node group specifying unit 80.

  The labeling unit 90 labels the vertex nodes extracted by the vertex node extracting unit 70, the mountain nodes centered on the vertices identified by the mountain node group identifying unit 80, and the relationship between them. I do.

  Here, the graph structure and centrality will be described. According to the definition of centrality score, nodes in areas where there are many edges have a high score. Based on the graph structure, when a model in which a person walks across nodes (a model in which a user browses nodes along a graph) is considered, in such an area with high centrality, there are many transitions within the area, and between nodes Is also highly relevant. That is, the area is composed of nodes related to the same topic. Therefore, each mountain in FIG. 5 is considered to correspond to a different topic.

  Further, it is considered that each document has a characteristic depending on the position of the node included in the mountain in FIG. In the following, the characteristics of a document corresponding to each node will be described. Further, a method for identifying a role in a document set will be described for each node having each characteristic.

  The node in the first stage in FIG. 5 is a node at the top of the mountain (for example, corresponding to a node indicated by reference signs a1 and b1), and only one node exists in one mountain. Since these nodes are nodes having the highest state transition from the surrounding nodes and are the nodes most closely related to the surrounding nodes, it can be said that these nodes are documents that best express the topic. That is, the document indicated by the vertex node specifies the topic of the area and when the topic is in time. Hereinafter, the label of the document (node) that identifies the topic in this area is set as the core document (or core node).

  Nodes in the second stage are nodes close to the vertex (for example, nodes indicated by symbols a2, a3, a4 and b2, b3 in FIG. 5). These nodes are nodes that can be reached from a core node directly or indirectly by following only a bidirectional link. Bidirectional links are linked to each other and show high relevance. These nodes have many state transitions with the core node, and the content of the document is also highly related to the core document. Hereinafter, a label of a document (node) having a high relationship with the core document is referred to as a supplemental document (or supplemental node).

  The node at the third stage is a node linked to a core node or a supplemental node, such as nodes indicated by reference signs a5, a6, a7, and b4 in FIG. It is a node that has a higher probability of transition to a core node or supplemental node of a specific topic than state transition to an external node or self-transition. These nodes are not necessarily the center of the topic, but contain information related to the topic, and are often nodes that contain highly novel information such as information around the topic. Hereinafter, a label of a document (node) that often includes highly novel information such as information around the topic is referred to as a subtopic document (or subtopic node).

  The node at the final stage is a node that does not have a strong relationship with any topic node. For example, it is a node indicated by reference sign c1 in FIG. This node has few similar nodes and high self-transition probability. Hereinafter, a label of a document (node) having few similar nodes and a high self-transition probability is referred to as an outlier document (outlier node). By allowing the outlier document to exist, it is possible to prevent other documents from belonging to any one of the clusters and causing noise.

  Based on the above method, each node is labeled as follows.

  First, for each node, information indicating what topic each node is related to and the role of the document when expressing the topic is labeled (ie, core node). As given.

  Next, a label (that is, a supplemental node) as a cluster of documents related to the topic expressed by the vertex node is assigned to the mountain-shaped node group.

  Then, the combinations of mountain nodes are labeled (ie, subtopic nodes or outlier nodes) based on their connected state, with high relevance of topics expressed by the two mountains.

  The information output unit 110 displays (visualizes) the contents of the document set to the user by using the relationship between nodes, the centrality of each node, and the role in the document set obtained by the information analysis unit 100. ) Visualization is performed with a display device, for example.

  The document DB 1 (document data management means) is a document data storage device (for example, a hard disk device or a memory) having a search function that can specify a document set according to conditions such as a search keyword specified by the user and the last update date of the document. Device). The document DB 1 may be constructed by collecting information in advance from a web or the like. It is also conceivable to use a search engine (see Non-Patent Document 5) existing on the web as it is as the document DB 1.

  Next, a document set analysis method according to this embodiment will be described with reference to FIG.

  First, a document set specifying condition designated by the user or predetermined is read from the input means (S1). The input means can be assumed to be a keyboard device, for example.

  Next, the document set specifying unit 10 specifies a set of documents that meet the document set specifying condition (S2).

  Next, the content similarity evaluation unit 20 calculates the content similarity between each document pair in the document group specified by the document set specifying unit 10 (S3).

  Next, the time stamp similarity evaluation unit 30 calculates temporal closeness (time similarity) between each document pair in the document group specified by the document set specifying unit 10 (S4).

  Next, the relationship extraction unit 40 extracts a pair having strong relevance based on the calculated content similarity and time similarity, and identifies the relevance with weight (S5).

  Next, the centrality determination unit 50 creates an index (for example, PageRank) based on the information specified by the similarity evaluation units 20 and 30 and the relationship extraction unit 40, and specifies the centrality of each node. (S6).

  Next, based on the information obtained from the similarity evaluation units 20 and 30, the relationship extraction unit 40, and the centrality determination unit 50, the graph structure construction unit 60 corresponds to nodes (one-to-one correspondence with documents) in a three-dimensional space. ) Is constructed (S7).

  Next, the vertex node extraction unit 70 extracts a vertex node based on the graph structure obtained by the graph structure construction unit 60 (S8).

  Next, the mountain node group specifying unit 80 extracts a mountain node group from the graph structure obtained by the graph structure construction unit 60 and the vertex obtained by the vertex node extraction unit 70 (9).

  Next, the labeling unit 90 determines the relationship between the node, the mountain-shaped node group, and the node group based on the information obtained by the graph structure building unit 60, the vertex node extracting unit 70, and the mountain-shaped node group specifying unit 80. Then, labeling is performed (S10).

  Then, the labeled nodes, mountain-shaped node groups, and node group relationships are visualized as a list or 3D map (S11).

  As described above, according to the present embodiment, the similarity between the documents based on the content and the time stamp is specified from the document set given by the document set specifying condition, and the documents are strongly connected based on the similarity. The weight of each part is specified with weight, and the centrality of each document is calculated based on this information, assuming the connection between the documents as a graph structure. The relationship between the documents and the centrality value of each document obtained above are calculated. By identifying the document group as a three-dimensional graph structure and identifying the position of each document from its positional relationship, "identification of major topics" and " It is possible to acquire “specification of documents related to topic”, “role of each document among documents related to each topic”, “relation between topics”, and the like.

  More specifically, in this embodiment, the mutual similarity between documents is evaluated based on a document set specified based on a request from a user, and the relationship between documents is specified based on the similarity. Based on this relationship, the centrality of each document is evaluated. By using both the relationship between documents and the centrality of individual documents, it is possible to detect specific topics in a document set, cluster documents belonging to a specific topic, and clarify the positioning of each document in the cluster. Realize.

  Also, based on the relationship between documents and the centrality value of each document, the document set is regarded as a three-dimensional graph structure, and by specifying the vertex and mountain-shaped node group, the document set It is possible to detect a specific topic, cluster a document belonging to a specific topic, and clarify the position of each document in the cluster.

  Further, according to the present embodiment, the accuracy of “specification of main topics” and “specification of documents related to topics” is higher than that in the case where time is not conscious (not considered).

That is, FIG. 6 shows the result of analyzing the newspaper article by applying the method of FIGS. 1 to 5, and the strength of time similarity (time weight) as a parameter for changing the time similarity. ) For determining), t _1/2 indicating the half-life of time similarity, and the result of investigating the relationship between clustering accuracy. In FIG. 6, a broken line indicates a baseline when time information is not considered.

  The current survey conditions are evaluated using the search results obtained by searching the newspaper article corpus. Since newspaper articles are used and the issue date is every day, the unit of the value indicating the time difference is days.

According to FIG. 6, the time information (alpha, t _1/2) as compared with the case without consideration of, it is the higher the clustering accuracy when considering time information, in particular the half-life t _1/2 30 to 90 It can be seen that the clustering accuracy is significantly higher than the baseline (broken line) when the time similarity is set to about 0.5 and the time similarity weight is set to around 0.5.

  Furthermore, according to the present embodiment, when acquiring a core document in “the role of each document among documents related to each topic”, the content is not only a representative but actually a topic. It is possible to extract a document at a stage, that is, when a related document is continuously issued. As a result, it is possible to acquire information such as when a specific topic has been raised.

  Note that the present invention can be realized by configuring some or all of the functions of each means in the document set analysis apparatus of the present embodiment with a computer program and executing the program using the computer. It is needless to say that the procedure in the document set analysis method of the above can be configured by a computer program and the program can be executed by the computer. For example, FD (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, CD (Compact Disk), DVD (Digital Versatile D) sk), and recorded in a removable disk, or stored, it is possible or distribute. It is also possible to provide the above program through a network such as the Internet or electronic mail.

  Further, a computer program describing a method related to the document set analysis apparatus described above may be implemented to access a memory or an external storage device that stores input / output data required for the method related to the document set analysis apparatus. .

  Although the embodiments of the present invention have been described above, the present invention is not limited to the described embodiments, and various modifications can be made within the scope described in each claim.

  For example, the information analysis unit in the present embodiment is configured by means such as a graph structure construction unit to a labeling unit. However, the information analysis unit is not limited to these units, and a relation between a document group and a document is regarded as a graph structure. Other processing means are also conceivable. More specifically, the labeling unit may specify the role by further finely labeling (for example, five or more levels).

The block diagram of the document set analysis apparatus in this embodiment. 6 is a flowchart illustrating a document set analysis method according to the present embodiment. The graph which shows the forgetting function in this embodiment. FIG. 5 is a characteristic diagram of similarity between documents using temporal similarity in the present embodiment. The three-dimensional structure conceptual diagram in this embodiment. The characteristic view which shows the relationship between the influence intensity parameter of the time information in this embodiment, and the precision of clustering.

Explanation of symbols

1 ... Document DB
DESCRIPTION OF SYMBOLS 10 ... Document set specific | specification part 20 ... Content similarity evaluation part 30 ... Time stamp similarity evaluation part 40 ... Relation extraction part 50 ... Centrality determination part 60 ... Graph structure construction part 70 ... Vertex node extraction part 80 ... Mountain node group Specific part 90 ... Labeling part 100 ... Information analysis part

110 ... Information output part a1, a2, a3, a4, a5, a6, a7, a8, b1, b2, b3, b4, c1 ... node

Claims (10)

A document set analysis device for identifying a role of a document based on a relationship between documents with time stamps in a document set managed by a document data management means,
Document set specifying means for specifying the document set based on the document set specifying condition input from the input means;
Content similarity evaluation means for evaluating the content similarity regarding the topic word between the documents included in the specified document set;
Time stamp similarity evaluation means for evaluating a time similarity regarding a topic word between each document included in the specified document set;
A relationship extraction unit that extracts a relationship between documents based on the content similarity evaluated by the content similarity evaluation unit and the time similarity evaluated by the time stamp similarity evaluation unit;
Based on the relationship between the documents extracted by the relationship extraction unit, the centrality determination unit that calculates the centrality of the document as an index indicating the high degree of relationship between the document and a document other than the document;
Based on the relationship between the documents obtained by the relationship extracting means and the centrality of each document obtained by the centrality determining means, the topic words and the topic words included in the document set are converted from the entire document set. An information analysis means for identifying a related document set and a role of the document in the document set;
Information output means for visualizing and outputting the document set based on the topic word included in the document set from the entire specified document set, the document set related to the topic word, and the role of the document in the document set;
A document set analyzing apparatus comprising: The document set analysis apparatus according to claim 1,
The information analysis means includes
Based on the relationship between the documents and the centrality of each document, the relationship between the documents is expressed in two-dimensional coordinates, and the centrality is expressed in the third-dimensional coordinates with respect to the two-dimensional coordinates. A graph structure constructing means for representing a three-dimensional graph structure;
From the obtained graph structure, vertex node extraction means for extracting, as a vertex, a node having a higher centrality than nodes other than itself connected by edges;
From the obtained vertex node, follow a graph structure in the direction of decreasing centrality, and a mountain-shaped node group specifying means for specifying a mountain composed of nodes;
Labeling means for assigning a label indicating a role to the node based on the identified vertex node and mountain node;
A document set analyzing apparatus comprising: The document set analysis device according to claim 1 or 2,
The time stamp similarity evaluation unit evaluates time similarity using a forgetting function that continuously changes in accordance with a time stamp difference between the documents. The document set analysis device according to any one of claims 1 to 3,
The relationship extraction unit extracts a relationship between documents based on the content similarity and a time similarity obtained by performing predetermined weighting on the time similarity evaluated by the time stamp similarity evaluation unit. A document set analyzing apparatus characterized by the above. A document set analysis method for identifying a role of a document based on a relationship between documents with time stamps in a document set managed by a document data management means,
A document set specifying step in which the document set specifying means specifies the document set based on the document set specifying condition input from the input means;
A content similarity evaluation means for evaluating a content similarity regarding a topic word between the documents included in the specified document set;
A time-stamp similarity evaluation unit that evaluates a time-similarity related to a topic word between documents included in the specified document set; and
A relationship extraction step for extracting a relationship between documents based on the content similarity evaluated by the content similarity evaluation step and the time similarity evaluated by the time stamp similarity evaluation step;
A center for calculating the centrality of the document as an index indicating the degree of relevance between the document and a document other than the document based on the relationship between the documents extracted by the relationship extraction step. Sex determination step;
Topics included in the document set from the entire document set based on the relationship between documents obtained by the relationship extracting step and the centrality of individual documents obtained by the centrality determining step. An information analysis step for identifying a word, a document set related to a topic word, and a role of the document in the document set;
Information that the information output means visualizes and outputs the document set based on the topic word included in the document set from the entire specified document set, the document set related to the topic word, and the role of the document in the document set An output step;
A document set analysis method comprising: The document set analysis method according to claim 5,
The information analysis step includes
Based on the relationship between the documents and the centrality of each document, the relationship between the documents is expressed in two-dimensional coordinates, and the centrality is expressed in the third-dimensional coordinates with respect to the two-dimensional coordinates. A graph structure construction step that expresses a three-dimensional graph structure,
From the obtained graph structure, a vertex node extraction step of extracting, as a vertex, a node having a higher centrality than nodes other than itself connected by an edge;
From the obtained vertex node, follow the graph structure in the direction of decreasing centrality, and specify a mountain-shaped node group specifying step for specifying a mountain composed of nodes;
A labeling step of assigning a label indicating a role to the node based on the identified vertex node and the mountain node;
A document set analysis method characterized by comprising: The document set analysis method according to claim 5 or 6,
In the time stamp similarity evaluation step, the time similarity is evaluated using a forgetting function that continuously changes in accordance with a time stamp difference between the documents. The document set analysis method according to any one of claims 5 to 7,
The relationship extraction step extracts a relationship between documents based on the content similarity and a time similarity obtained by performing predetermined weighting on the time similarity evaluated by the time stamp similarity evaluation step. Document collection analysis method characterized by the above.   9. A document set analysis program, wherein the document set analysis method according to claim 5 is described as a computer program executable by a computer.   9. A recording medium in which the document set analysis method according to claim 5 is described as a computer program executable by a computer and the computer program is recorded.

Images