JPH0612494A - Image matching device and image search device - Google Patents

Abstract

(57) [Summary] [Purpose] An image matching device capable of matching images using the image features of the constituent elements even if the constituent element names of the images differ, and the image desired by the searcher by using this. An image retrieval device for retrieving an image similar to. [Structure] The constituent element name comparison unit 13 compares the constituent element names, and for the elements having the same constituent element names, the corresponding constituent element coincidence degree calculation unit 14 calculates the degree of coincidence of the constituent elements using the image feature. To do. For the elements whose component names do not match, the similarity degree extraction unit 16 sets the similar word dictionary storage unit 1
The degree of similarity between the constituent elements stored in 5 is extracted, and the degree of coincidence between the elements is calculated by the similar degree-to-similarity element matching degree calculation unit 17. The degree of coincidence between them is calculated by the inter-image coincidence degree calculation unit 1
The images are combined at 8 to obtain the degree of coincidence between images. By searching for an image based on the obtained degree of coincidence between the images, the search result intended by the searcher can be obtained.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image retrieving apparatus for retrieving an image similar to an image desired by a user from a created image database and an image collating apparatus used in that case.

[0002]

2. Description of the Related Art As a conventional image retrieval method, for example,
"6th Fuzzy System Symposium Proceedings" p
Some are shown on pages 341-344.

This is to create a semantic network from words indicating the contents of an image and use this as an image index or search key. Now, considering an image in which there are women, men, and dogs as shown in FIG. 12, the content of the image is expressed in words: "There is a woman. A woman is on the left side of a man. A dog is almost behind the woman. There is. " This can be represented by a semantic network as shown in FIG. 13 (a). This is composed of nodes that represent the constituent elements of the image and arrows that represent the relationship between the nodes, and this is used as data that represents the index or search key of the image. Arrow e
The data structure of is the label L (e) of the arrow, the label L of the starting point v
(v), the label L (w) of the end point w, and the membership value μ (e) consisting of four groups e: (μ (e), L (e), L (v), L (w)). It Using this expression, FIG. 13 (a) can be described as shown in FIG. 13 (b).

Words used to indicate the content of an image, which describe the constituent elements of the image and the relationship between the elements, may be slightly different between the index and the search key. For example,
The "left side" may be expressed as "next to". This is dealt with using a fuzzy thesaurus. An example of a fuzzy thesaurus is shown in FIG. This fuzzy thesaurus contains all the words used to create indexes and search keys, and gives a fuzzy relationship between those words.

The similarity between the index and the search key is calculated by describing the images of the index and the search key as shown in FIG.
These are matched and obtained. Here, the set of arrows in the graph of the index is defined by (Equation 1) and the search key is defined by (Equation 2). Further, the elements of the index and the search key are defined as in (Equation 3). Then, if a fuzzy thesaurus of registered words is used as a search term (Equation 4), the similarity between the index and the search key is obtained by (Equation 5).

[0006]

[Equation 1]

[0007]

[Equation 2]

[0008]

[Equation 3]

[0009]

[Equation 4]

[0010]

[Equation 5]

As described above, the similarity between images is calculated between the search key and the index of the image in the image database. The image with the highest calculated similarity is output as a search result, or is output in the descending order of similarity.

[0012]

As described above, in the conventional image retrieval apparatus as described above, regarding the image in the image database, the constituent elements in the image and their positional relationship are represented by a semantic network in advance. As
The image recalled by the searcher is also represented by a semantic network as a search key, and the degree of similarity is obtained by graph matching between the image index and the search key.

However, there are many descriptions of the positional relationship of the constituent elements in the image, and it is considered that the same image varies depending on the user. For example, the image of FIG. 12 is represented by the semantic network of FIG. 13 (a), but can also be represented by the semantic network of FIG. 14 (a). Therefore, the description of the image is as shown in FIG. 15B, which is different from that in FIG. 13B. That is, if the description of the positional relationship between the image index and the search key is different, the search result desired by the searcher cannot be obtained.

Since the description of the positional relationship of the constituent elements in the image is considered to be different depending on the user even in the same image, the semantic network of the index of the image describes all the positional relationships of the constituent elements in the image. There is a need. However, if all the positional relationships of the constituent elements in the image are to be represented by a semantic network, it is difficult to create and a large memory capacity is required to store the index of one image. Therefore, in an actual system, if the description of the positional relationship between the image index and the components of the search key is different, the result desired by the searcher may not be obtained.

Further, in the above-mentioned image retrieval apparatus, the contents of the image indicated by the database creator or the retriever in words are only represented by the semantic network, and the image feature amount in the actual image is used. Absent. Therefore,
A search that specifies image characteristics such as the size and color of a component cannot be performed.

In view of the above point, the present invention is an image collating apparatus capable of collating two images using the image features of the constituent elements even if the constituent element names in different languages are used, and
An object of the present invention is to provide an image collation device capable of collating images in consideration of relationships between constituent elements, and an image retrieval device capable of retrieving an image similar to an image desired by a searcher from an image database.

[0017]

According to a first aspect of the present invention, there is provided a collation image input section for inputting an image to be collated, a collated image input section for inputting an image to be collated, and a collation image input by the collation image input section. And the matching factor between the corresponding component that calculates the matching factor of the image features for each matching component name and the matching component name of the matching image input in the matching image input unit The calculation unit, the similar word dictionary storage unit that stores knowledge of the degree of similarity between component names, and the degree of similarity between constituent items whose component names do not match using the knowledge stored in the similar word dictionary storage unit. A similarity degree extraction unit to extract, a similarity degree calculation unit between similar components that calculates the degree of matching of image features between components whose component names do not match using the similarity degree extracted by the similarity degree extraction unit, between corresponding elements The degree of coincidence calculated by the coincidence degree calculation unit An image feature matching system consisting of a degree of coincidence match degree calculating section between similar components is calculated inter-image match degree calculating section for obtaining the degree of similarity between images.

A second aspect of the present invention is a collation image input section for inputting an image to be collated, a collation image input section for inputting an image to be collated, a collation image input by the collation image input section and a collation image input section. The component name comparison unit that compares the component names of the images to be collated that was input in step 1, and the corresponding inter-component coincidence calculation unit that calculates the image feature matching score for each component whose component name matches, and the element name The element name tree structure storage unit that stores the tree structure that describes the hierarchical relationship of the element and the element that obtains the distance between the component name that does not match the component name using the tree structure stored in the element name tree structure storage unit An inter-name distance extraction unit, a similarity degree calculation unit that calculates the degree of similarity between element names using the distance between element names obtained by the element name distance extraction unit, and a degree of similarity obtained by the similarity degree calculation unit are used. Similarity degree extracted by the similarity degree extraction unit Using the matching degree calculation unit between similar components for calculating the matching degree of the image features between the constituent elements whose constituent element names do not match, the matching degree calculated by the matching degree calculation unit between corresponding elements and the matching degree calculation unit between similar constituent elements The image matching apparatus includes an inter-image matching degree calculation unit that obtains the matching degree between images from the matching degree calculated by.

A third aspect of the invention is a collation image input section for inputting an image to be collated, a collation image input section for inputting an image to be collated, a collation image input by the collation image input section and a collation image input section. The corresponding constituent element extraction unit that extracts constituent elements whose constituent element names of the collated image input in step 1 are extracted, the image feature matching degree calculation unit that obtains the degree of coincidence of the constituent elements using the image characteristics of the constituent elements, and the constituent elements Of the relative relationship matching degree calculation unit that obtains the matching degree of each other by using the positional relationship with other components, and the matching degree calculated by the image feature matching degree calculation unit and the matching degree calculated by the relative relationship matching degree calculation unit. An image including an inter-element coincidence degree calculating unit that calculates the degree of coincidence between the constituent elements and an inter-image coincidence degree calculating unit that obtains the degree of coincidence between images from the degree of coincidence calculated by the corresponding inter-component element degree of coincidence calculating unit It is a matching device.

A fourth aspect of the invention is a search request image input section for inputting a search request image, an image storage section for storing images, an image input to the search image input section, and constituent elements of the image stored in the image storage section. A component designating section that designates a component, an image feature extracting unit that extracts the image feature of the component designated by the component designating unit, a search request image input to the search request image input unit, and a storage unit stored in the image storage unit. The image retrieval apparatus comprises an image collation apparatus including the means of the first, second, or third invention for collating existing images, and an image output unit that extracts from the image storage unit from the collation result of the image collation apparatus.

[0021]

According to the present invention, even if the constituent element names of two images do not match, the present invention provides a degree of matching between elements that can be identified as one another and elements having the same image, so that the images The degree of coincidence can be calculated.

Further, it is possible to automatically extract the positional relationship between the constituent elements and obtain the degree of coincidence between images in consideration of the positional relationship between the constituent elements.

Further, by using these image collating devices in the image searching device, it is possible to retrieve an image similar to the image desired by the searcher from the image database.

[0024]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the arrangement of an image collating apparatus according to the first embodiment of the present invention. In FIG.
Reference numeral 11 is a collation image data input unit for inputting data of an image to be collated, 12 is a collation image data input unit for inputting data of an image to be collated, 13 is a component name comparison unit for comparing component names, and 14 is Corresponding component matching score calculation unit for calculating the matching score between corresponding components, 15 is a similar word dictionary storage unit for storing knowledge about the similarity between the element names,
Reference numeral 16 is a similarity degree extraction unit that extracts the degree of similarity between element names using the dictionary stored in the similar word dictionary storage unit 17,
Is an inter-image coincidence degree calculation unit that calculates the degree of coincidence between similar components, and 18 is an inter-image coincidence degree calculation unit that calculates the degree of coincidence between images.

The operation of the image collating apparatus of the first embodiment constructed as above will be described. First, the image data to be collated is input to the collation image data input unit 11, and the image data to be collated is input to the collation image data input unit 12. Image data is given a component name and the image feature of the component. As the image feature amount, the position, area, main color, etc. of the component can be considered. Here, FIG. 2 (a)
As the constituent elements shown in Fig. 2, an image including the month, person, and house is input as the collation image to the collation image data input unit 11, and the image as shown in Fig. 2 (b) is the collation image as the collation image data. It is assumed that the data is input to the input unit 12. Assuming that the characteristic amounts of the constituent elements are, for example, the positions and areas of the constituent elements, the collation image data input unit 11 and the collation image data input unit 12 are performed.
From, the month, person, house, and the position and area information of each are input as the component name.

The component name comparison unit 13 compares the data of the collation image input from the collation image data input unit 11 with the component name of the collation image data input from the collation image data input unit 12, and constructs it. If the element names match,
Divide into those that do not match.

The corresponding component-to-component matching score calculation unit 14 calculates the matching score for each component that the component name comparison unit 13 has determined to match the element name. The degree of coincidence is calculated, for example, by the following method. With respect to the input image feature amount of the constituent element, the degree of coincidence for each characteristic amount is obtained from the membership function of the degree of coincidence determined based on the distance between the image characteristic amounts of the two constituent elements. An example of the membership function of the degree of coincidence is shown in FIG. The degree of coincidence of the constituents is obtained by combining the degrees of coincidence of the respective feature amounts obtained from the membership function of FIG. For example, the Dempster-Shafer associative rule is used to combine the degrees of coincidence of the feature amounts. Dempster-Sha
According to fer theory, p ₁ and p ₂ are the basic probabilities obtained based on independent evidence, and B _1q and B _2r (q, r = 0,1,2, ...) are the respective focus elements. , The basic probabilities can be combined by (Equation 6).

[0029]

[Equation 6]

As the focus element, the two components are similar
A means that there is something similar and Ac means that there is no similarity.
Image feature x of component_iThe similarities in
p⁽ⁱ⁾(A), the dissimilarity is p⁽ⁱ⁾(A^C)
It In addition, p⁽ⁱ⁾(A, A^C). This time the image
X as the image feature_iConfidence p when
⁽ⁱ⁾(A), p ⁽ⁱ⁾(A, A^C) Is explained.

Consider the positions and areas of the constituents as the image feature quantities. It is assumed that the position difference between the corresponding components between the two images is x ₁ and the area difference is x ₂ . The degrees of coincidence s ₁ and s ₂ are obtained from x ₁ and x ₂ using a membership function. For s ₁ and s ₂ obtained from the membership function, these values are divided by the number of feature quantities (2 in this case), and p ⁽¹⁾ (A),
p ⁽²⁾ (A). Furthermore, the ignorance amount p ⁽¹⁾ (A, A ^C ), p
⁽²⁾ Find (A, A ^C ) from (Equation 7). Sought p
^{From (i)} (A) and p ⁽ⁱ⁾ (A, A ^C ), the degree of coincidence p _j (A) of the constituent elements is obtained using the associative rule of (Equation 6).

[0032]

[Equation 7]

As described above, the degree of coincidence between the corresponding constituent elements can be obtained. In addition, here, the degree of coincidence is calculated for each feature amount for a plurality of feature amounts of the constituent elements,
Demp is used to combine them and obtain the degree of agreement between components.
Although the ster-Shafer associative rule was used, the method of combining using rules and the membership of the degree of coincidence after obtaining the distances in the multidimensional feature space for multiple features of the constituent elements The degree of coincidence between constituent elements may be obtained by a function.

The similar word dictionary storage unit 15 stores a fuzzy thesaurus that defines the degree of similarity between constituent element names. As an example, FIG. 4 shows a fuzzy thesaurus showing the degree of similarity between the three constituent elements of the moon, the sun, and the star. From Fig. 4, it can be seen that the sun and moon are similar components with a degree of 0.8, the stars and moon with a degree of 0.6, and the sun and stars with a degree of 0.5.

The similarity degree extraction unit 16 uses the fuzzy thesaurus stored in the similar word dictionary storage unit 15 for the components whose element names are determined not to match by the component name comparison unit 13 to have similar configurations. Extract the elements and find the similarity s. Here, consider a case where image data as shown in FIG. 2A is given as a collation image and image data as shown in FIG. 2C is given as a collation image. Although the component names of the image in FIG. 2 (a) are moon, person, and house, the component names of the image in FIG. 2 (c) are sun, person, house, and person, house match. , Moon, sun do not match. Therefore, if the constituent element names do not match, the degree of similarity is checked using a fuzzy thesaurus stored in the similar word dictionary storage unit 15. Here, the component month of the matching image is
It can be seen that it is similar to the sun, which is a component of the collated image, with a degree of 0.8.

The matching degree calculation unit 17 between similar components calculates the matching degree according to the degree of similarity for each of the similar components extracted by the degree of similarity extraction unit 16. First, the degree of coincidence q _k (A) is _obtained from the image features of the constituent elements by the same method as that of the correspondence degree between corresponding constituents calculation unit 14. Next, using the degree of similarity s obtained by the degree of similarity extraction unit 16, the degree of coincidence p _sk (A) between similar components is obtained from the degree of coincidence q _k (A) and the degree of similarity s.
For example, the degree of coincidence between similar components is obtained using a function such as (Equation 8).

[0037]

[Equation 8]

The inter-image matching score calculating unit 18 calculates the matching scores between the matching components obtained by the matching matching score calculating unit 14 and the matching target images, and the matching scores between similar components. The degree of coincidence between the matching image and the collated image is calculated by combining the degrees of coincidence between the constituent elements that are similar but do not match the constituent element names obtained by the coincidence degree calculating unit 17. For example, the Dempster-Shafer associative rule is used to combine the coincidences. The details will be described below.

Now, assuming that there are N constituent elements of the collation image, from the K coincidences p ₁ (A) to p _K (A) obtained by the inter-corresponding-element coincidence calculation unit 14, (Equation 9) More p ₁ '(A) ~ p
_{Find K} '(A). Also, from (Equation 10), the ignorance amount p ₁ '(A,
A ^C ) -p _K '(A, A ^C ) is calculated. Similarly, the (N−K) number of coincidences p _{s (1+} ) obtained by the similarity calculation unit 17 between similar components
_{From K)} (A) to p _sN (A), from ( _Equation 11), p _{s (1 + K)} '
From (A) to _{p sN} '(A), the ignorance amount p _{s (1 + K)} ' from ( _Equation 12)
(A, A ^C ) to _{p sN} '(A, A ^C ). Sought p _j '
Using (A), p _j ′ (A, A ^C ), p _sj ′ (A), and p _sj ′ (A, A ^C ), the inter-image matching degree M is obtained from (Equation 6).

[0040]

[Equation 9]

[0041]

[Equation 10]

[0042]

[Equation 11]

[0043]

[Equation 12]

The degree of coincidence M between images can be obtained as described above. In FIG. 5, as a collation image, FIG.
2 schematically shows how to obtain the degree of coincidence between images when FIG. 2 (c) is input as the collated image. At 51,
The degree of coincidence between “person” and “house”, which are the corresponding constituent elements, is calculated in the correspondence degree between-corresponding-element degree-of-coincidence calculation unit 14 using the image characteristics of the constituent elements. In 52, the degree of coincidence between “month” and “sun” in which the element names do not coincide is obtained by the degree-of-similarity element coincidence calculation unit. These "person" coincidence, "house" coincidence, "moon"-"sun" coincidence,
The inter-image coincidence degree calculation unit 18 combines them to obtain the inter-image coincidence degree.

As described above, in the present embodiment, the degree of coincidence is obtained by using the image feature of the constituent element for each constituent element having the same constituent element name between the two images. At this time, even if there are constituent elements that do not have the same constituent name, even if the expressions are different or the images are the same, the similarity degree is calculated using the similar word dictionary, and the degree is calculated. The degree of coincidence can be calculated. The degree of coincidence between two images is obtained by combining the degree of coincidence between these constituent elements. Therefore, even if the component names in words are different, it is possible to obtain the degree of coincidence between images using the image features of the components.

Next, an image collating apparatus according to the second embodiment of the present invention will be described. FIG. 6 is a block diagram showing the arrangement of the image collating apparatus according to the second embodiment. The same parts as those in FIG. 1 are designated by the same reference numerals and the description thereof will be omitted. In FIG. 6, 61 is an element name tree structure storage unit that stores a tree structure that describes the hierarchical relationship of element names, and 62 is an element name tree structure storage unit 61.
The inter-element-name distance extraction unit that obtains the distance between the element names using the tree structure stored by, 63 is a similarity degree calculation unit that calculates the degree of similarity between element names, and 64 is the degree of coincidence between similar components. Is a matching degree calculation unit between similar components.

The operation of the image collating apparatus of the second embodiment constructed as above will be described. In the constituent element name comparison unit 13, the constituent element names are matched and the constituent element names are not matched.
The process up to the process of calculating the matching score for each component in the corresponding matching score calculation unit 14 is the same as in the first embodiment of the present invention, and the description thereof is omitted.

The element name tree structure storage unit 61 stores a tree structure describing the hierarchical relationship of the component element names. FIG. 7 shows an example of the hierarchical relationship of component element names. From FIG. 7, it can be seen that the generic concept of crescent moon, half moon, and full moon is “month”, and the generic concept of moon is “astronomical object”.

The element name distance extraction unit 62 uses the element name tree structure stored in the element name tree structure storage unit 61,
Find the distance between element names. For example, the distance between element names is
It is the distance from the first constituent element name to the second constituent element name through the tree structure, and is considered to be 1 when the upper concept is moved once, and 1 when the lower concept is similarly moved. Therefore, from FIG. 6, the distance between the element names “crescent moon” and “full moon” is 2.

The similarity calculating section 63 calculates the similarity between the element names from the distance between the element names obtained by the element name distance extracting section 62. Here, a function for calculating the degree of similarity from the distance between elements is prepared in advance, and the degree of similarity is obtained according to this function. It is considered that the larger the distance between elements is, the smaller the degree of similarity between element names is. Therefore, when the degree of similarity is y and the distance between elements is x, for example, (Equation 1
The degree of similarity is obtained using a function such as 3). If the distance between the element names is found to be 2 by the element name distance extraction unit 62, the degree of similarity is 0.5.

[0051]

[Equation 13]

The matching degree calculation unit 64 between similar constituent elements uses the degree of similarity between the element names whose element names do not match and the degree of matching obtained from the image features, which is obtained by the similarity degree calculation unit 63.
The degree of coincidence between similar components is calculated by the same method as 17 in FIG. 1 of the first embodiment.

The operation of the inter-image matching score calculation unit 18 is the same as that of the first embodiment, and calculates the matching score between the collation image and the collation image.

The degree of coincidence between images can be obtained as described above. As described above, in the present embodiment, the degree of coincidence is obtained for each constituent element whose constituent element name matches between the two images, by using the image feature of the constituent element. At this time, even if there are constituent elements whose constituent element names do not match, the degree of similarity of the constituent elements is obtained from the distance between the element names obtained from the hierarchical relationship of the element names, and the degree of coincidence according to the degree is obtained. be able to. Therefore, even if the component names in words are different, it is possible to obtain the degree of coincidence between images using the image features of the components.

Next, an image collating apparatus according to the third embodiment of the present invention will be described. FIG. 8 is a block diagram showing the arrangement of the image collating apparatus according to the third embodiment. The same parts as those in FIG. 1 are designated by the same reference numerals and the description thereof will be omitted. In FIG. 8, reference numeral 81 is a corresponding component extraction unit that compares the component names and extracts the components having the same element name, 82 is an image feature matching score calculation unit that obtains the matching score of the components using the image features,
Reference numeral 83 is a relative relationship matching degree calculation unit that obtains the matching degree of the constituent elements by using the relative positional relationship with other constituent elements, and 84 is a corresponding constituent element matching degree calculation unit that calculates the matching degree between corresponding constituent elements, Reference numeral 85 denotes an inter-image coincidence degree calculation unit that calculates the degree of coincidence between images.

The operation of the image collating apparatus of the third embodiment constructed as above will be described. The process of inputting the image data to be collated into the collation image data input unit 11 and the image data to be collated into the collation target image data input unit 12 is as follows.
This is similar to the first embodiment of the present invention. Here, for the sake of explanation, the images including the component element names E1, E2, E3 as shown in FIG. 9A are used as the matching images, and the component element names F1, F2 as shown in FIG. 9B are used. It is assumed that an image including the F3 element is input as the collation image.

The corresponding component extraction unit 81 compares the data of the collation image input from the collation image data input unit 11 with the component name of the data of the collation image input from the collation image data input unit 12, Extract the component with the same component name. Here, the matching image component names E1, E
2, E3 and the component names F1, F2, F3 of the collated image are compared, and E1
And the component names of F1, E2 and F2, E3 and F3 match.
FIG. 10 shows how to obtain the degree of coincidence between images in such a case.

For the sake of explanation, first, the corresponding constituent element E1
Consider finding the degree of agreement between −F1. The degree of coincidence between the corresponding constituent elements is calculated by combining the degree of coincidence obtained from the image feature between the corresponding constituent elements and the degree of coincidence obtained from the relative positional relationship with other constituent elements.

First, in the image feature matching score calculation unit 82, the constituent elements E1-F1 using the image characteristics of the constituent elements E1, F1.
The degree of coincidence between the corresponding constituent elements is obtained by the same method as that used by the degree-of-correspondence calculation section 14 between corresponding constituent elements of the first embodiment. This is shown at 101.

Next, in the relative relationship coincidence calculation unit 83, the coincidence between the constituent elements E1 to F1 is calculated using the relative positional relationship between the constituent element for which the matching degree is obtained and all other elements. The combination of the element E1 and other elements in the matching image is shown in 91 (E1, E2) and 92 (E1, E2).
3). Corresponding images in the collated image are (F1, F2) shown in 93 and (F1, F3) shown in 94. The relative positional relationship of each combination is represented by the distance to the other element and its direction as seen from the element for which the degree of coincidence is obtained. Here, the positional relationship 91 is the distance D _E and the direction R from E1 to E2.
_{In E} , the positional relationship 93 is the distance D _F from F1 to F2.
And the direction R _F. In 103, the positional relationship 91
And the degree of coincidence of the positional relationship 93 is obtained. First, the positional relationship 91
And the positional relationship 93 (D _E −D _F ) and the direction difference (R
_The degree of coincidence between the distance and the direction is obtained from the membership function of the degree of coincidence similar to that shown in FIG. 3 determined based on _E− R _F ).
Then, when they are combined by the same method as the corresponding component matching degree calculation unit 14 of the first embodiment, the matching degree of the positional relationship 91 and the positional relationship 93 is obtained. Similarly, the degree of coincidence between the positional relationship 92 and the positional relationship 94 is obtained as 104. Next, as shown at 102, 103, 104
The degree of coincidence of the relative positional relationship obtained in step S1 is combined in the same manner as the corresponding degree of coincidence between constituent elements calculating unit 14 of the first embodiment to obtain the degree of coincidence using the relative relationship between the elements E1 and F1. .

Next, in the matching degree calculation unit 84 between corresponding constituent elements,
The matching degree in the image feature between the elements E1 and F1 obtained by the image feature matching degree calculating section 82 and the relative relationship matching degree calculating section 83
The degree of coincidence in the relative relationship obtained in (1) is combined by the same method as that of the inter-corresponding-element coincidence calculation unit 14 of the first embodiment, and the degree of coincidence between corresponding constituents E1 to F1 is obtained. 10
5 shows the process in which the degree of coincidence between the elements E1 and F1 is obtained.

Similarly, between the elements E2 and F2 and between the elements E3 and F3, the image feature matching degree calculating unit 82 obtains the image feature matching degree, and the relative relationship matching degree calculating unit 83 obtains the relative relationship matching degree. These are combined in the corresponding component matching degree calculation unit 84, and the matching degrees between the elements E2-F2 and between the elements E3-F3 are obtained.

In the inter-image matching score calculating unit 85, the matching score between the corresponding constituent elements obtained by the corresponding inter-component matching score calculating unit 84 is the same as that in the inter-image matching score calculating unit 18 of the first embodiment. And the degree of coincidence between the images is obtained. Figure 10
As shown in, between elements E1-F1, between elements E2-F2, and elements E3-
The degree of coincidence between F3s is combined to obtain the degree of coincidence between images.

As described above, in this embodiment, even if the user does not describe the relationship between the constituent elements, the positional relationship between the constituent elements is automatically extracted, and the image characteristics of the constituent elements and the matching between the images are compared. Therefore, it is possible to perform matching between images in consideration of the positional relationship between the constituent elements as well as the image characteristics.

Next, an image search apparatus according to the fourth embodiment of the present invention will be described. FIG. 11 is a block diagram showing the arrangement of the image search apparatus of the fourth embodiment. In FIG. 11, 1
Reference numeral 10 is an image input unit for capturing an image to be stored as an image database, 111 is a component designating unit for designating the components of the input image, and 112 is an image feature amount from the region of the component designated by the component designating unit 111. An image feature extraction unit for extracting the image features, 113 an image feature storage unit for storing the image features extracted by the image feature extraction unit 112, 114 an image storage unit for storing the accumulated image input from the image input unit 110, and 115 a user Is a search request image input unit for inputting an image requesting a search, 116 is a component specifying unit for specifying a component of the input search request image, 117 is a component specifying unit specified by the component specifying unit 116. An image feature extraction unit for extracting an image feature amount from the area, and 118 an image feature and an image feature storage unit of the search request image input from the search request image input unit 115. 13 compares the image feature of the storage image stored an image matching system for determining the degree of matching of images, using the first, the image matching system of the second or third embodiment. An image output unit 119 obtains a candidate image from the image matching degree obtained by the image matching apparatus 118, extracts it from the image storage unit 114, and outputs it.

The operation of the image retrieval apparatus of the fourth embodiment constructed as above will be described.

When storing images, the image input unit 110 takes in the images to be stored as the image database, and the image storage unit 114 stores the images. Component designation section 111
At, the database creator or the like specifies the area of the constituent element of the input image and inputs the constituent element name. The image feature extraction unit 112 extracts an image feature from the region of the component designated by the component designation unit 111, and the extracted image feature is stored in the image feature storage unit 113.

At the time of image retrieval, retrieval request image input section 115
Captures the image requested by the user for the search. The search request image is given as a handwritten image of the image to be searched by the user, a similar image, or an image showing the region of the constituent element. Component designation section 116
At, the user or the like specifies the area of the constituent element of the input image, inputs the constituent element name, and the image feature extraction unit 117 extracts image characteristics from the specified constituent element area. Similar to the first, second, or third embodiment, the image matching device 118 compares the image features of the accumulated image extracted at the time of image accumulation and stored in the image feature storage unit 113 with the image features of the search request image. Is performed, and the degree of coincidence of the images is obtained. Then, the image output unit 119 outputs, as candidate images, the one having the highest degree of coincidence among the obtained degrees of coincidence or a plurality of those having the highest degree of coincidence.

As described above, since the image features are extracted from the regions of the constituent elements in the designated image in this embodiment and the image characteristics of the constituent elements are used, the image characteristics such as the size and color of the constituent elements in the image are determined. The specified image search can be performed. Further, if the image collating apparatus of the first and second embodiments is used, even if the component element names do not match, the degree of matching is obtained for elements with different word expressions or elements with the same image. Images similar to the user's search request can be searched. Further, when the image collating apparatus according to the third embodiment is used, even if the user does not describe the positional relationship between the constituent elements, the positional relationship between the constituent elements is automatically extracted from the search request image, and the image characteristic Since it is used for the search together with the search result, a search result close to the search request image of the user can be obtained.

[0070]

As described above, in the image collating apparatus according to the present invention, when the degree of coincidence is obtained by using the image feature of the constituent element for each constituent element whose constituent element name matches between the two images, the constituent element Even if the names do not match, the degree of matching is calculated for elements with different word expressions or elements with the same image,
The degree of coincidence between two images can be obtained.

Further, the image collating apparatus of the present invention automatically extracts the positional relationship between the constituents from the search request image without the user having to describe the positional relationship between the constituents, and the image feature and the image are also extracted. Since it is used for matching between the elements, it is possible to obtain the degree of coincidence in consideration of the positional relationship of the constituent elements.

Further, since the image retrieving apparatus of the present invention extracts the image feature from the region of the component in the specified image and uses the image feature of the component, the image such as the size and color of the component in the image is extracted. Image search with specified features can be performed.
At this time, even if the constituent element names do not match, the degree of matching is obtained for elements having different word expressions or elements having the same image, so that an image close to the user's search request can be searched. Further, even if the user does not describe the positional relationship between the constituent elements, the positional relationship between the constituent elements is automatically extracted from the search request image and is used for the search together with the image feature, so that it is close to the user's search request image. You can get the search results.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an image matching device according to a first embodiment of the present invention.

FIG. 2 is a diagram showing an example of an input image in the same embodiment.

FIG. 3 is a diagram showing an example of a membership function showing the degree of coincidence in the embodiment.

FIG. 4 is a diagram showing an example of a dictionary defining a degree of similarity between constituent elements in the same embodiment.

FIG. 5 is a diagram showing how to determine the degree of coincidence between images by the image matching apparatus in the embodiment.

FIG. 6 is a block diagram showing a configuration of an image matching device according to a second embodiment of the present invention.

FIG. 7 is a diagram showing an example of a hierarchical relationship of component name in the same embodiment.

FIG. 8 is a block diagram showing a configuration of an image matching device according to a third embodiment of the present invention.

FIG. 9 is a diagram showing an example of an input image in the embodiment.

FIG. 10 is a diagram showing how to determine the degree of coincidence between images by the image matching apparatus in the embodiment.

FIG. 11 is a block diagram showing a configuration of an image search device according to a fourth embodiment of the present invention.

FIG. 12 is a diagram showing an image for explaining a conventional image search method.

FIG. 13 is a diagram showing a semantic network that expresses the content of an image used in a conventional image search method.

FIG. 14 is a diagram showing an example of a fuzzy thesaurus used in a conventional image retrieval device.

FIG. 15 is a diagram of a semantic network for explaining the problems of the conventional image search method.

[Explanation of symbols]

 11 collation image data input unit 12 collation image data input unit 13 component name comparison unit 14 corresponding component inter-matching degree calculation unit 15 similar word dictionary storage unit 16 similarity degree extraction unit 17 similar component inter-matching degree calculation unit 18, 85 inter-image matching score calculation unit 61 element name tree structure storage unit 62 element name distance extraction unit 63 similarity degree calculation unit 64 similar component matching score calculation unit 81 corresponding component extraction unit 82 image feature matching score calculation unit 83 relative Relationship matching degree calculation unit 84 Corresponding component matching degree calculation unit 110 Image input unit 111, 116 Component designation unit 112, 117 Image feature extraction unit 113 Image feature storage unit 114 Image storage unit 118 Image matching device 119 Image output unit

Claims (4)

[Claims] 1. A collation image input unit for inputting an image to be collated, a collation image input unit for inputting an image to be collated, a collation image input by the collation image input unit, and an input by the collation image input unit. The component name comparison unit that compares the component name of the collated image, the corresponding inter-component coincidence calculation unit that calculates the image feature matching score for each component whose component name matches, and between the component names A similar word dictionary storage unit that stores knowledge of the similarity degree of, and a similarity degree extraction unit that uses the knowledge stored in the similar word dictionary storage unit to extract the similarity degree between constituent elements whose constituent element names do not match, The similar component-to-component matching score calculation unit that calculates the matching score of the image characteristics between the components whose component name does not match using the similarity score extracted by the similarity score extraction unit, and the corresponding-element matching score calculation unit, The calculated coincidence and the similar structure Image feature matching system is characterized in that an inter-image match degree calculating section for determining the degree of coincidence between the images from the degree of matching between elements match degree calculating section is calculated. 2. A collation image input unit for inputting an image to be collated, a collation image input unit for inputting an image to be collated, a collation image input by the collation image input unit, and an input by the collation image input unit. A component name comparison unit that compares the component names of the compared images, a corresponding inter-component matching score calculation unit that calculates the matching score of the image features for each matching component name, and a hierarchy of element names An element name tree structure storage unit that stores a tree structure that describes a relationship, and an element name that obtains a distance between component element names that do not match the component name using the tree structure stored in the element name tree structure storage unit An inter-distance extraction unit, a similarity degree calculation unit that calculates the degree of similarity between element names using the distance between element names obtained by the element name distance extraction unit, and a similarity degree obtained by the similarity degree calculation unit. Is extracted by the similarity degree extraction unit that is similar using Similarity factor between similar components that calculates the degree of matching of image features between components whose component names do not match using the degree of similarity, the matching factor calculated by the corresponding factor matching factor calculation unit, and the similar component An image matching device, comprising: an inter-image matching degree calculating unit that obtains a matching degree between images from the matching degree calculated by the inter-matching degree calculating unit. 3. A collation image input unit for inputting an image to be collated, a collation image input unit for inputting an image to be collated, a collation image input by the collation image input unit, and an input by the collation image input unit. Corresponding component extraction unit that extracts a component whose component name of the collated image matches, an image feature matching score calculation unit that obtains the matching score of the component using the image feature of the component, and the matching of the component And the degree of coincidence calculated by the image feature coincidence degree calculation unit and the degree of coincidence calculated by the relative relation coincidence degree calculation unit. An inter-element coincidence degree calculation unit that calculates the degree of coincidence between constituent elements and an inter-image coincidence degree calculation unit that obtains the degree of coincidence between images from the degree of coincidence calculated by the corresponding inter-component element coincidence degree calculation unit Image matching characterized by Location. 4. A search request image input unit for inputting a search request image, an image storage unit for storing images, an image input to the search image input unit and constituent elements of the image stored in the image storage unit. A component designating unit, an image feature extracting unit that extracts image features of the component designated by the component designating unit, a search request image input to the search request image input unit, and the image storage unit. The image collating apparatus according to any one of claims 1 to 3 for collating images, and an image output section for taking out from the image storage section from a collation result of the image collating apparatus. Search device.