JP2015158848A - Image retrieval method, server, and image retrieval system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress excess of operational costs when creating learning data after installing a camera.SOLUTION: A computer including a processor and a memory detects a first object and a second object from an inputted image, extracts a first image feature amount of the first object and a second image feature amount of the second object, determines that the first object and the second object are different objects, generates a transformation matrix such that the variance between the first image feature amount and the second image feature amount, namely different objects, becomes large, and converts the image feature amount by the transformation matrix and then stores the image feature amount.

Description

  The present invention relates to an image search system and method, and relates to information search in a computer.

  In recent years, with the increase in violent crimes and security awareness, many surveillance cameras are being installed in places where people gather, such as stores, airports, and roads. Images taken by these surveillance cameras are stored in a storage device such as a surveillance recorder and viewed as necessary. However, with the widespread use of IP cameras (network connection cameras), a large number of cameras can be connected via a network, and the capacity of storage devices has been increased. Accordingly, it is very difficult to visually confirm all video data as in the prior art.

  Therefore, various similar search techniques have been proposed in order to search and present a scene where a specific person or object is shown from a large amount of video data stored in the storage device. Here, the similar search technique refers to a technique in which data similar to a search query designated by a user is searched from target data and the result is presented. In particular, the similar image search technique is a technique for searching for data having a high degree of similarity between feature quantities using feature quantities such as hue, shape, composition, etc. extracted from the image itself. For example, when searching for a person, vector data such as an edge pattern of a face image and a clothes color histogram can be used as a feature amount. Also, the similarity increases as the distance between feature quantity vectors decreases.

  However, in general, such a feature quantity is a high-dimensional vector such as several hundreds to thousands of dimensions. Therefore, when calculating the distance between feature quantity vectors, a large amount of computation is a problem.

  Therefore, it is necessary to reduce the number of distance calculations by compressing a high-dimensional feature quantity vector to a low dimension. A method using discriminant analysis has been proposed as a method of compressing a high-dimensional vector to a low dimension.

  Patent Document 1 discloses a technique for converting a feature quantity vector using discriminant analysis in order to obtain a feature quantity vector effective for discrimination of a character image or a face image from an input feature quantity vector.

  Patent Document 2 discloses a technique for improving accuracy by using both high-quality image data and low-quality image data when performing dimensional compression using discriminant analysis on a character image. Yes.

JP 2009-140513 A JP 2004-310639 A

  Discriminant analysis is to obtain a feature vector transformation matrix that increases the variance between classes and reduces the variance within a class when learning data in the form of pairs of classes and feature vectors is given. This is a supervised dimension reduction method. Hereinafter, this transformation matrix is referred to as a discrimination matrix.

  When performing dimension compression by discriminant analysis on a feature vector extracted from a face image, a set of face images of the same person is treated as the same class. Therefore, a discriminant matrix is obtained in which the distance between the vectors of the principals is small and the distance between the vectors of the others is large. That is, even when the face direction, facial expression, and lighting conditions are different, the degree of similarity between the persons is increased, and the degree of similarity between the other persons is reduced even when shooting in the same environment.

  When applied to similar image search, one discriminant matrix is created from the entire learning data, and projection using this discriminant matrix is performed on all feature quantity vectors extracted from the face image. Then, the vector distance between the projected feature quantity vectors is calculated, and similar face images are searched by sorting in order from the smallest distance value. Therefore, it is presumed that the accuracy of finding the person himself / herself is improved when a similar image search is performed using a feature quantity vector subjected to dimension reduction by discriminant analysis.

  Hereinafter, a dimension compression method using discriminant analysis will be described. Here, a method for generating a discriminant matrix Φ for converting a d-dimensional feature quantity vector x extracted from a face image into a d′-dimensional feature quantity vector by discriminant analysis will be described. The d dimension is the number of dimensions of the number of images extracted from the face image. The d ′ dimension is the number of dimensions after compression, and is the number of dimensions set according to the required accuracy, computer performance, and the like.

  First, as shown in the following equation, intra-class variance matrix W is calculated using data belonging to each other, that is, data belonging to the same class, and inter-class variance matrix B is calculated using data belonging to other people, that is, different classes.

Here, the number of classes is c ≧ 2, the total number of data is n, the data set is X = {x}, and the average value of the entire data is _ave x. Further, the data set of classes i x _i, the number of data of the data set x _i n _i, the average of the data of the data number n _i and _ave x _i. T represents a transposed matrix.

  Using the intraclass variance matrix W and the interclass variance matrix B, an eigenvector matrix Ψ and an eigenvalue matrix Λ satisfying the following equation (3) are obtained.

  BΨ = WΨΛ (3)

Here, ψ is a matrix having eigenvectors ψi (i = 0,..., D) as column vectors, and Λ is a matrix having eigenvalues λ _i) (λ ₁ ≧ λ ₂ ≧ ... ≧ λ _d ) as diagonal elements. It is. A matrix Φ = {φ1, φ2,..., Φd} in which d ′ eigenvectors obtained in this way are arranged in descending order of eigenvalues is a discrimination matrix. A space projected using this discriminant matrix Φ is called a discriminant space.

  Using the d-dimensional feature vector X before compression and the discriminant matrix Φ, the d′-dimensional feature vector Y after compression is expressed as the following equation (4).

Y = Φ ^T X (4)

  Note that the dimension number d ′ after compression and the class number c of the learning data have a relationship as shown in the following equation (5).

  m ≦ (c-1) (5)

  Also, the discriminant matrix is created by obtaining the eigenvector matrix Ψ ′ and the eigenvalue matrix Λ ′ satisfying the following equation (6) using only the interclass variance matrix B without using the intraclass variance matrix W. It is also possible to do.

  BΨ ′ = Ψ′Λ ′ (6)

  When dimensional compression is performed using discriminant analysis in this way, it is necessary to prepare in advance learning data showing a person's face image and classify each face image for each person. In addition, since there is a limit to the amount of the feature that can be compressed while retaining the features of a person, it is generally considered that when a feature amount of several thousand dimensions is compressed, the feature amount is several hundred dimensions.

  Therefore, as shown in the above equation (5), it is necessary to collect as a learning data an image in which several hundred or more different people are shown. Furthermore, in order to calculate intra-class variance, a lot of learning data of the same person is required. For these reasons, it takes a great deal of time to create learning data manually.

  On the other hand, if the lighting conditions of the environment where the person is photographed and the orientation and size of the face image are controlled and do not change, as in the face authentication device, the discriminant space created once can be used in another location. Conceivable. Therefore, when dimensional compression is performed for a controlled environment, it takes a lot of time to create initial learning data, but it is possible to reuse the same learning data.

  However, when the shooting parameters of the camera are different, or when the shooting environment such as the surrounding lighting conditions and the angle and size in which a person is photographed is different, the appropriate discrimination space is likely to be different. For example, when a discriminant space is learned using a face photograph that faces the camera, such as an ID photograph, an appropriate projection cannot be performed in an environment in which the face image facing obliquely or the illumination is dark.

  Therefore, when a similar face image search is performed for an image taken in an uncontrolled situation where the surrounding environment and human behavior cannot be predicted, such as a surveillance camera, the face image taken by the subject surveillance camera is used. It is desirable to create learning data.

  From the above, high-precision dimensional compression cannot be performed even if a discriminant space created in advance in different environments is used. Therefore, using the face images captured by a number of cameras installed in actual locations, learning the discriminant space and creating one discriminant matrix, and projecting the feature vector using this discriminant matrix Is required. In this case, since learning data cannot be created in advance, learning data is created after the camera is installed, and the operation cost is very high.

  An image search method for searching for an image using a computer having a processor and a memory, wherein the computer detects a first object and a second object from an input image, and the computer , A second step of extracting a first image feature quantity of the first object and a second image feature quantity of the second object, and the computer comprising the first object and the second object. A third step of determining that the object is different from the object, and the computer having a large variance between the first image feature quantity and the second image feature quantity that are different objects A fourth step of generating a simple transformation matrix, and a fifth step of storing the image feature quantity after the computer has transformed the image feature quantity using the transformation matrix. No.

  According to the present invention, in order to increase the variance B between feature quantities of different objects, it is possible to generate a better transformation matrix and improve search accuracy by determining that the objects in the same image are different objects. To do. Since learning data for creating a transformation matrix can be automatically collected, the process for creating learning data can be reduced and the operating cost of the system can be suppressed.

1 is a block diagram illustrating a configuration of an image search system according to a first embodiment of this invention. It is explanatory drawing which shows the 1st Example of this invention and shows feature-value management information. FIG. 2 shows a first embodiment of the present invention, showing generation of another person information, and an image of a camera. FIG. Fig. 3 shows a first embodiment of the present invention, showing generation of different person information, and an image of another camera. It is a block diagram which shows the 1st Example of this invention and shows a discrimination matrix production | generation process. It is a flowchart which shows the 1st Example of this invention and shows the feature-value vector registration process. It is a flowchart which shows the 1st Example of this invention and shows a search process. It is a schematic diagram which shows 2nd Example of this invention and shows the production | generation of another person information and the same person information. It is a flowchart which shows the 2nd Example of this invention and shows discriminant matrix production | generation processing.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

  Hereinafter, an image search system according to a first embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing the configuration of the image search system of the first embodiment.

  The image search system of the first embodiment includes a server computer 110, a client computer 130, discriminant matrix information 140, a search database 150, and a camera 160. Each device is connected to each other by a communication infrastructure 120.

  The server computer 110 includes an external interface 111, a central processing unit (CPU) 112, a memory 113, and a large-capacity external storage device (HD) 114.

  The external interface 111 is an interface (I / F) for connecting the server computer 110 to the communication infrastructure 120. The CPU 112 is a processor that executes processing of the server computer 110. The memory 113 is a work area for processing executed by the CPU 112, and stores various data and programs loaded from the HD 114. The HD 114 is a mass storage device such as a hard disk, and stores programs executed by the CPU 112, data (discriminant matrix information 140, search database 150), and the like. The HD 114 may be an external storage device connected to the server computer 110.

  The client computer 130 is a computer connected to the communication infrastructure 120. Although FIG. 1 shows one client computer 130, any number of client computers 130 may be provided. If the server computer 110 has a function equivalent to that of the client computer 130, all processing may be performed by the server computer 110.

  The client computer 130 may be a computer having any configuration. FIG. 1 shows a configuration of a typical client computer 130. That is, the client computer 130 of FIG. 1 includes a CPU 131, a memory 132, an I / F 133, an input device 134, and an output device 135.

  The CPU 131 is a processor that executes a program stored in the memory 132. The memory 132 is a storage device that stores programs executed by the CPU 131. The I / F 133 is an interface connected to the communication infrastructure 120 and used for communication between the client computer 130 and the server computer 110. The input device 134 is a device that receives input from the user of the client computer 130. The input device 134 is, for example, a keyboard or a mouse. The output device 135 is a device that displays information to the user of the client computer 130. For example, an image display device such as a CRT or a liquid crystal display. As the input device 134 and the output device 135, a display provided with a touch sensor may be used as an input / output device.

  The image search system according to the present embodiment has a configuration in which a server computer 110 and a client computer 130 connected via a communication infrastructure 120 (network) provide a service. The configuration may be such that a service is provided by an application.

  The discriminant matrix information 140 stores a discriminant matrix (or transformation matrix) 300 for performing dimensional compression of the feature vector. Note that a matrix obtained by transposing the discriminant matrix 300 may be stored.

  The search database 150 is a database for storing image feature amounts (feature amount vectors) extracted from images to be searched, and stores, for example, feature amount management information 200 (see FIG. 2).

  Cameras 160a to 160n are cameras installed in the monitoring target area. Hereinafter, the generic name of the cameras 160a to 160n is referred to as the camera 160. Note that when the video or image to be processed is captured in advance and all the video or image is transmitted from the client computer 130 to the server computer 110, the camera 160 may not be provided. Alternatively, image data (video or image) to be processed may be stored in the HD 114 in advance. Alternatively, the image data received from the camera 160 may be stored in the HD 114.

  The CPU 112 operates as a functional unit that provides a predetermined function by executing processing of each program. For example, the CPU 112 functions as a discriminant matrix generation unit by performing processing according to the discriminant matrix generation program 400. Here, as shown in FIG. 4, the discriminant matrix generation unit includes an image acquisition unit 401, a face detection processing unit 402, a person information generation unit 403, a feature amount extraction unit 404, an interclass variance calculation unit 405, and a discrimination matrix generation unit. 406 and a functional unit of the discriminant matrix storage unit 407.

  The CPU 112 functions as a search feature value conversion unit by performing processing according to the search feature value conversion program 500. Here, as shown in FIG. 5, the search feature quantity conversion unit includes functional units of an image acquisition unit 501, a face detection processing unit 502, a feature quantity extraction unit 503, a feature quantity conversion unit 504, and a feature quantity storage unit 505. including.

Further, the CPU 112 functions as a search unit by performing processing according to the search program 600. Here, as illustrated in FIG. 6, the search unit includes functions of an image input unit 601, a face detection processing unit 602, a feature amount extraction unit 603, a feature amount conversion unit 604, a similarity search unit 605, and a search result output unit 606. Part.
As described above, the CPU 112 also operates as a functional unit that provides the functions of a plurality of processes executed by each program. A computer and a computer system are an apparatus and a system including these functional units.

  Information such as a program and a table for realizing each function of the server computer 110 is stored in a storage device such as an HD 114, a nonvolatile semiconductor memory, a hard disk drive, or an SSD (Solid State Drive), or a computer such as an IC card, an SD card, or a DVD. It can be stored in a readable non-transitory data storage medium.

  FIG. 2 is an explanatory diagram illustrating the feature amount management information 200 according to the first embodiment.

  The feature amount management information 200 includes a search data ID 201 and a search target image feature amount 202. The search data ID 201 is an identifier for identifying a feature amount, and is used for associating image data and the like. The search target image feature amount 202 is a feature amount vector that is extracted from the image and then converted using the discrimination matrix 300.

  The discriminant matrix information 140 and the search database 150 may be stored in the HD 114 included in the server computer 110, or may be stored in another hard disk different from the HD 114. When the image data is stored in the HD 114, the search data ID 201 is assigned to the image data corresponding to the search target image feature amount 202. Alternatively, when the image data is stored in another device, an instruction to assign the search data ID 201 to the image data corresponding to the search target image feature amount 202 may be transmitted.

  3A and 3B show generation of the different person information in the first embodiment, and are images of the cameras 160a and 160b. FIG. 4 is a block diagram illustrating an example of a discriminant matrix generation process performed by the discriminant matrix generation program 400 of the first embodiment.

  Hereinafter, the discriminant matrix generation process will be described with reference to FIGS. 3A, 3B, and 4. FIG.

  In this embodiment, the server computer 110 functions as a discriminant matrix generation unit by executing the discriminant matrix generation program 400. The discriminant matrix generation unit includes an image acquisition unit 401, a face detection processing unit 402, a person information generation unit 403, a feature amount extraction unit 404, an interclass variance calculation unit 405, a discriminant matrix generation unit 406, and a discrimination matrix storage unit 407. The discriminant matrix 300 is generated.

  The CPU 112 shown in FIG. 1 loads various programs stored in the HD 114 into the memory 113, reads the various programs loaded in the memory 113, and executes the read various programs, thereby obtaining an image acquisition unit. 401, the face detection processing unit 402, the person information generation unit 403, the feature amount extraction unit 404, the interclass variance calculation unit 405, the discrimination matrix generation unit 406, and the discrimination matrix storage unit 407 are realized as described above.

  First, in the image acquisition unit 401, the server computer 110 acquires an image from the camera 160 via the communication infrastructure 120. The image acquisition unit 401 acquires an image as learning data. The server computer 110 may acquire the image for each frame by decoding the video after acquiring the video from the camera 160. In addition, an image or video captured by the camera 160 is temporarily stored in the client computer 130, and the image or video is transmitted from the client computer 130 to the server computer 110 via the communication infrastructure 120 and received by the image acquisition unit 401. Also good. Alternatively, an image captured in advance as learning data may be stored in the HD 114, and an image may be acquired (or input) from the HD 114.

  Next, the face detection processing unit 402 performs face detection processing on the acquired image, and acquires the face area of the person shown in the image. As the face detection process, a known or publicly known technique may be applied, and thus will not be described in detail here.

  Next, in the person information generation unit (object information generation unit) 403, when a plurality of persons are captured in an image acquired from one camera 160 for the face area detected by the face detection processing unit 402, The person information is generated assuming that the person in the picture is a different person.

  For example, with reference to FIGS. 3A and 3B, when the persons 320A, 320B, and 320C appear in the image 310 acquired from the camera 160a, it is estimated that the persons 320A, 320B, and 320C are different persons. . Therefore, the person information generation unit 403 generates information that the persons 320A, 320B, and 302C are different persons.

  Further, when the persons 340A, 340B, 340C, and 340D are shown in the image 330 acquired from the camera 160b, the persons 340A, 340B, 340C, and 340D are also estimated to be different persons. Therefore, the person information generation unit 403 generates information that 340A, 340B, 340C, and 340D are different persons.

  For example, the person information generation unit 403 may assign an identifier to each person in the face area as information about the same person or information about another person, and a different identifier to another person.

  Next, the feature quantity extraction unit 404 extracts a d-dimensional feature quantity vector as a face image feature quantity from the face area detected by the face detection processing unit 402. The face image feature amount is, for example, a multidimensional vector created based on an edge pattern, a color histogram, or the like. Note that the calculation of the feature vector is not described in detail here because a known or publicly known technique such as the edge pattern or the color histogram may be used.

  Note that the processing of the person information generation unit 403 and the feature amount extraction unit 404 may be performed in parallel, or one of them may be performed first.

  Next, the interclass variance calculation unit 405 calculates the interclass variance B using the feature quantity vector extracted from the face area according to the following equation (7).

Here, if the total number of frames of learning data is n _f , the class number c _j ≧ 2 is the number of face images (face regions) detected from the j-th frame image, and x _ij is the j-th frame image. The feature amount vector extracted from the i-th face area of the image, and the average of the feature amount vector data is _ave x _j .

Next, the discriminant matrix generation unit (conversion matrix generation unit) 406 obtains an eigenvector matrix Ψ _B and an eigenvalue matrix Λ _B that satisfy the following equation (8).

BΨ _B = Ψ _B Λ _B (8)

Here, Ψ _B is a matrix having eigenvectors ψ _Bi (i = 0,..., D) as column vectors, and Λ _B is a diagonal element with eigenvalues λ _Bi (λ _B1 ≧ λ _B2 ≧... ≧ λ _Bd ). Is a matrix. A matrix Φ _B = [Φ _B1 , Φ _B2 ,..., Φ _Bd ] obtained by arranging d ′ eigenvectors ψ _Bi in the descending order of the eigenvalue λ _Bi is a discriminant matrix 300 of d columns × d ′ rows. The variance B between classes is increased by performing the conversion described later using this discriminant matrix Φ _B.

  Finally, the discriminant matrix storage unit 407 stores the discriminant matrix (transformation matrix) 300 in the discriminant matrix information (transformation matrix information) 140.

  Through the above processing, the server computer 110 extracts a face area from the input image, and extracts person information and a feature vector from the face area. Then, the server computer 110 calculates a discriminant matrix 300 such that the inter-class variance B becomes large from the extracted person information and feature quantity vector, and stores the discriminant matrix 300 in the discriminant matrix information 140.

  FIG. 5 is a block diagram illustrating a feature vector registration process performed by the search feature conversion program 500 according to the first embodiment.

  In this embodiment, the server computer 110 functions as a search feature value conversion unit by executing the search feature value conversion program 500. The search feature quantity conversion unit registers the feature quantity vector in the feature quantity management information 200 using the image acquisition unit 501, face detection processing unit 502, feature quantity extraction unit 503, feature quantity conversion unit 504, and feature quantity storage unit 505. Execute the process. Note that the image acquisition unit 501, face detection processing unit 502, and feature amount extraction unit 503 may be the same as or different from the image acquisition unit 401, face detection processing unit 402, and feature amount extraction unit 404 shown in FIG. Also good.

  First, the image acquisition unit 501 acquires an image to be searched for similar images from the camera 160 via the communication infrastructure 120. Note that the image may be acquired by decoding the video after acquiring the video. Alternatively, the image acquisition unit 501 may transmit an image or video to be searched for similar images from the client computer 130 via the communication infrastructure 120. Alternatively, an image captured in advance may be stored in the HD 114, and an image may be acquired (or input) from the HD 114.

  Next, the face detection processing unit 502 performs face detection processing on the acquired image, and acquires the face area of the person shown in the image. The face detection process is the same as that of the face detection processing unit 402 in FIG. 4, and a known or publicly known technique may be applied.

  Next, the feature quantity extraction unit 503 extracts a d-dimensional feature quantity vector as the feature quantity of the face image from the face area detected by the face detection processing unit 502. The face image feature amount is, for example, a multidimensional vector created based on an edge pattern or a color histogram. When the face detection processing unit 502 detects a plurality of face areas, d-dimensional feature quantity vectors are extracted from all the face areas. Note that the feature quantity vector is the same as that of the feature quantity extraction unit 404 of FIG. 4, and a known or publicly known technique may be used.

  Next, the feature quantity conversion unit 504 calculates a product of the d-dimensional feature quantity vector extracted by the feature quantity extraction unit 503 and the discriminant matrix 300 acquired from the discriminant matrix information 140, and obtains a d′-dimensional feature quantity vector. Convert to Note that the number of dimensions is d ′ <d, and the feature vector is compressed by the discriminant matrix 300.

  Finally, the feature quantity storage unit 505 stores the d′-dimensional feature quantity vector obtained by the feature quantity conversion unit 504 in the feature quantity management information 200 of the search database 150.

  Here, the feature amount storage unit 505 stores the feature amount vector in the search target image feature amount 202 of the feature amount management information 200, and assigns a search data ID 201 corresponding to the feature amount vector. In the feature quantity management information 200, clustering or hashing may be generated and index information may be stored together in order to perform high-speed search during search processing. The feature amount management information 200 may include an image identifier or location (file path or the like) corresponding to the search target image feature amount 202.

  Through the above processing, the server computer 110 calculates a d-dimensional feature vector from the input image (or video), converts it into a d′-dimensional feature vector using the discriminant matrix 300, and performs dimensional compression. The feature quantity vector is stored in the feature quantity management information 200.

  FIG. 6 is a block diagram showing search processing performed by the search program 600 of the first embodiment.

  In this embodiment, the server computer 110 functions as a search unit by executing the search program 600. The search unit executes a search process using the image input unit 601, the face detection processing unit 602, the feature amount extraction unit 603, the feature amount conversion unit 604, the similarity search unit 605, and the search result output unit 606. The face detection processing unit 602, the feature amount extraction unit 603, and the feature amount conversion unit 604 may be the same as the face detection processing unit 402, the feature amount extraction unit 404, and the feature amount conversion unit 504 illustrated in FIG. It may be different.

  First, in the image input unit 601, an image (search target image) showing a person as a search key (search target) for similar image search is input from the client computer 130 via the communication infrastructure 120. Accept.

  Next, the face detection processing unit 602 executes face detection processing on the input image (search target image), and acquires the face area of the person shown in the image. The face detection processing is the same as the face detection processing unit 402 in FIG.

  Next, the feature amount extraction unit 603 extracts a d-dimensional feature amount vector as a face image feature amount from the face area detected by the face detection processing unit 502. The face image feature amount is, for example, a multidimensional vector created based on an edge pattern or a color histogram. When the face detection processing unit 502 detects a plurality of face areas, d-dimensional feature quantity vectors are extracted from all the face areas. When a plurality of face areas are detected by the face detection processing unit 602, a face area as a search key may be specified from the client computer 130, or feature quantity vectors may be extracted from all the plurality of face areas. It may be used for future processing. The feature quantity vector is the same as that of the feature quantity extraction unit 404 in FIG.

  Next, the feature quantity conversion unit 604 calculates the product of the d-dimensional feature quantity vector extracted by the feature quantity extraction unit 603 and the discriminant matrix 300 acquired from the discriminant matrix information 140 to obtain a d′-dimensional feature quantity vector. Get. When a plurality of search keys are used, all feature vectors are converted using the discrimination matrix 300. Note that the feature value conversion is the same as the feature value conversion unit 504 in FIG.

  Next, the similarity search unit 605 calculates a distance between vectors of the feature quantity vector as a search key and the search target image feature quantity 202 stored in the search database 150. Then, the search data IDs 201 are arranged in ascending order from the smallest vector distance.

  Finally, the search result output unit 606 outputs the search result to the client computer 130 based on the sorted search data ID 201. For example, when image data is associated with the search data ID 201, an image data string is output.

  With the above processing, the server computer 110 calculates the d′-dimensional feature vector for the search target image input from the client computer 130 and calculates the inter-vector distance of the search target image feature 202 in the search database 150. . Then, the server computer 110 transmits the search data ID 201 or the image as a search result to the client computer 130 in ascending order of the vector distance. The number of search data IDs 201 or the number of images that the server computer 110 transmits to the client computer 130 as a search result may be limited to a predetermined value.

  In the first embodiment, the facial feature amount extracted from the detected face area has been described as an object. However, any feature feature can be used as long as it can be detected from an image. For example, a person feature amount extracted from a person region or a feature amount of an object other than a person may be used.

  Based on the above, the image search system according to the first embodiment detects the first object and the second object from the input image, the first image feature amount of the first object, A second image feature amount of a second object is extracted, the first object and the second object are determined to be different objects, and the first image feature amount to be different objects from each other; Generating a transformation matrix (discriminant matrix 300) such that a variance B between the second image feature amount and the second image feature amount is increased, and performing a search using the image feature amount after being transformed using the transformation matrix; And

  With this feature, it is possible to generate a transformation matrix in which the distance between vectors between individuals is small and the distance between vectors between others is large without any human intervention, and search accuracy is improved. Since learning data for creating a transformation matrix can be automatically collected, the process for creating learning data can be reduced and the operating cost of the system can be suppressed.

  The image search system according to the second embodiment of the present invention will be described below with reference to FIGS.

  The image search system of the second embodiment is realized by using the same computer system as the image search system of the first embodiment, a block diagram showing a configuration, an explanatory diagram showing feature quantity management information, The block diagram showing the feature vector registration process and the block diagram showing the search process are the same.

  FIG. 7 is a schematic diagram showing generation of different person information and identical person information in the second embodiment, and FIG. 8 is a block diagram showing discrimination matrix generation processing in the second embodiment.

  Hereinafter, the discriminant matrix generation process of the second embodiment will be described with reference to FIGS.

  In the second embodiment, the server computer 110 functions as a discriminant matrix generation unit by executing the discriminant matrix generation program 400. As shown in FIG. 8, the discriminant matrix generation unit includes an image acquisition unit 801, a face detection processing unit 802, a person tracking unit 803, a person information generation unit 804, a feature amount extraction unit 805, an interclass variance calculation unit 806, an intraclass The discriminant matrix 300 is generated by the variance calculation unit 807, the discriminant matrix generation unit 808, and the discriminant matrix storage unit 809.

  First, in the image acquisition unit 801, the server computer 110 acquires an image from the camera 160 via the communication infrastructure 120. The image acquisition unit 801 acquires an image as learning data, as in the first embodiment.

  The server computer 110 may acquire the image for each frame by decoding the video after acquiring the video from the camera 160. In addition, an image or video captured by the camera 160 or another camera is temporarily stored in the client computer 130, and the image or video is transmitted from the client computer 130 to the server computer 110 via the communication infrastructure 120 to obtain an image. It may be received by the unit 801. Alternatively, an image captured in advance as learning data may be stored in the HD 114, and an image may be acquired (or input) from the HD 114.

  Next, the face detection processing unit 802 executes face detection processing on the acquired image, and acquires the face area of the person shown in the image. The face detection 0 process is the same as the face detection processing unit 402 shown in FIG. 4 of the first embodiment, and a known or known technique may be applied.

  Next, the person tracking unit 803 tracks a person captured in successive frames (images). When the face detection processing unit 802 detects a plurality of face areas, each face area is tracked. The tracking of the face area of the person tracking unit 803 associates the face area of the same person between different frames, and a well-known or publicly known technique may be used, and will not be described in detail here.

  Next, in the person information generation unit 804, when a plurality of persons are shown in the image acquired from one camera 160 for the face area detected by the face detection processing unit 802, the person shown at the same time is a different person. As a result, personal information for another person is generated. As person information for another person, a person ID may be given to a face area in the image, or only information about another person may be held. Further, the person tracking unit 803 generates person information for the same person, assuming that the tracked face area is the same person. As person information for the same person, a person ID may be assigned and grouped. As described above, the person tracking unit 803 specifies the same object (the same person) if the first object or the second object is the same among the plurality of images input to the server computer 110.

  For example, referring to FIG. 7, when face regions (persons) 720A, 720B, and 720C are shown in an image (frame) 710 acquired from the camera 160a, the persons 720A, 720B, and 720C are different persons. It is estimated to be. Therefore, the person tracking unit 803 generates information that the persons 720A, 720B, and 702C are different persons. The person tracking unit 803 is the same for the persons 740A, 740B, and 740C shown in the image 730 and the persons 760A, 760B, and 760C shown in the image 750. In addition, the person tracking unit 803 detects 740A, 740B, and 740C in the image 730 as a result of tracking the persons 720A, 720B, and 720C of the image 710, and detects 760A, 760B, and 760C in the image 750. If detected, the persons 720A, 740A, and 760A are the same person, the persons 720B, 740B, and 760B are the same person, and the persons 720C, 740C, and 760C generate information that is the same person.

  For example, the person information generation unit 804 assigns an identifier for each person in the face area as information about the same person or information about another person, assigns the same identifier to the same person, and assigns a different identifier to another person. good.

  Next, the feature quantity extraction unit 805 extracts a d-dimensional feature quantity vector as a face image feature quantity from the face area detected by the face detection processing unit 802. The face image feature amount is, for example, a multidimensional vector created based on an edge pattern or a color histogram. Note that the processing of the person tracking unit 803 and the person information generation unit 804 and the processing of the feature amount extraction unit 805 may be performed in parallel, or one of them may be performed first. Note that the feature quantity vector is the same as that of the feature quantity extraction unit 404 of the first embodiment, and a known or publicly known technique may be used.

  Next, the inter-class variance calculation unit 806 calculates the inter-class variance B using the feature quantity vector extracted from the face area according to the equation (7) shown in the first embodiment.

  Next, the intra-class variance calculation unit 807 calculates the intra-class variance W using the feature quantity vector extracted from the face area according to the following equation (9).

Here, assuming that the number of persons tracked by the person tracking unit 803 is n _p , p _j ≧ 2 is the number of face images (face regions) detected from the j-th person, and x _ij is the j-th number. This is a feature vector extracted from the i-th face area of a person, and the average of the feature vector data is _ave x _j .

Next, the discriminant matrix generation unit 808 obtains an eigenvector matrix Ψ _BW and an eigenvalue matrix Λ _BW that satisfy the following equation (10).

BΨ _BW = WΨ _BW Λ _BW (10)

Here, Ψ _BW is a matrix having eigenvectors ψ _BWi (i = 0,..., D) as column vectors, and Λ _BW has eigenvalues λ _BWi (λ _BW1 ≧ λ _BW2 ≧... ≧ λ _BWd ) as diagonal elements. It is a matrix with. A matrix Φ _BW = {Φ _BW1 , Φ _BW2 ,..., Φ _{BWd ′} ] obtained by arranging d ′ eigenvectors _φBWi obtained in this order in descending order of the eigenvalues becomes a discriminant matrix 300 of d columns × d ′ rows. As a result, a discriminant matrix 300 having a large inter-class variance B and a small intra-class variance can be obtained.

  Finally, the discriminant matrix storage unit 809 stores the calculated discriminant matrix 300 in the discriminant matrix information 140.

  As described above, when a plurality of images are input, in addition to obtaining a transformation matrix (first transformation matrix) in which the variance B increases between different classes (face regions), the variance W is obtained in the same class (face region). A smaller transformation matrix (second transformation matrix) can be obtained. Thereby, in the second embodiment, in addition to the effects of the first embodiment, it is possible to improve the detection accuracy of the same person.

  In the first embodiment, the interclass variance calculation unit 405 calculates an interclass variance B using a feature vector extracted from a face image (face area) shown in one image. In the third embodiment, when calculating the inter-class variance B, the calculation is not performed using only the face image shown in one image, but as shown in the second embodiment, each face image (face As a result of tracking (region), calculation may be performed using a plurality of face images regarded as the same person.

  In the third embodiment, the inter-class variance B is calculated using the feature vector extracted from the face area according to the following equation (11).

Here, if the total number of frames of learning data is n _f , the class number c _j ≧ 2 is the number of face images (face regions) detected from the j-th frame image. Further, y _ij is an average value of feature quantity vectors extracted from the i-th face area of the j-th frame image and other face images regarded as the same person as a result of tracking, and _ave y _j is a feature This is the average value of the quantity vector y _ij .

  That is, when a plurality of images 710, 730, and 750 are input as learning data as shown in FIG. 7, the server computer 110, for example, the face area 720A of the image 710, the face area 740A of the image 730, and the image It is determined that the person is the same as the 750 face area 760A. Then, as described above, the server computer 110 calculates the inter-class variance B using the average value of the feature amount vectors of the three face regions 720A, 740A, and 760A.

  As described above, the discriminant matrix 300 that increases the interclass variance B by calculating the interclass variance B from the average value of the feature amount vectors of the face regions regarded as the same person in a plurality of frames (images). It is possible to improve the accuracy. The plurality of images may be continuous images or images every predetermined time.

<Modification>
In the interclass variance calculation unit 405 of the first embodiment, an example is shown in which the interclass variance B is calculated using the feature vector extracted from the face image (face area) shown in one image. The interclass variance B may be calculated using the feature vector of the face area of the image.

  For example, when the images 710, 730, and 750 are input as learning data as shown in FIG. 7, the face area (person) 720A of the image 710, the face area 740B of the image 730, and the image 750 are input from the second embodiment. , The server computer 110 recognizes each face area 760C as a different person. Then, the server computer 110 calculates the interclass variance B using the feature amount vectors of the three face regions 720A, 740B, and 760C.

  As described above, the accuracy of the discriminant matrix 300 that increases the inter-class variance B is improved by calculating the inter-class variance B from the feature vector of the face area regarded as a different person in a plurality of frames (images). It becomes possible to make it.

  The configuration of the computer, the processing unit, the processing unit, and the like described in the present invention may be partially or entirely realized by dedicated hardware.

  In addition, the various software exemplified in the present embodiment can be stored in various recording media (for example, non-transitory storage media) such as electromagnetic, electronic, and optical, and through a communication network such as the Internet. It can be downloaded to a computer.

  The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described.

110 server computer 111 external interface 112 CPU (central processing unit)
113 memory (main memory)
114 HD (large capacity external storage device)
120 communication infrastructure 130 client computer,
140 discriminant matrix information 150 search database 160a-160n camera 200 feature amount management information 401 image acquisition unit 402 face detection processing unit 403 person information generation unit 404 feature amount extraction unit 405 interclass variance calculation unit 406 discriminant matrix generation unit 407 discriminant matrix storage Unit 501 image acquisition unit 502 face detection processing unit 503 feature amount extraction unit 504 feature amount conversion unit 505 feature amount storage unit

Claims (15)

An image search method for searching for an image with a computer having a processor and a memory,
A first step in which the computer detects a first object and a second object from an input image;
A second step in which the computer extracts a first image feature quantity of the first object and a second image feature quantity of the second object;
A third step in which the computer determines that the first object and the second object are different objects;
A fourth step in which the computer generates a transformation matrix such that a variance between the first image feature quantity and the second image feature quantity that are different objects increases;
A fifth step in which the computer stores the image feature quantity after the image feature quantity is transformed using the transformation matrix;
An image search method comprising: The image search method according to claim 1,
The image search method further includes a sixth step in which the computer receives an image to be searched and searches for the received image using the image feature quantity converted by the conversion matrix. . The image search method according to claim 1,
The first step includes
Detecting a first object and a second object from each of a plurality of input images;
The third step includes
Determining that the first object and the second object are different objects in the same image;
Identifying the same object among the first object or the second object between the plurality of images;
An image search method comprising: The image search method according to claim 3,
The fourth step includes
Generating a first transformation matrix having a large variance between the first image feature quantity and the second image feature quantity that are different objects;
Generating a second transformation matrix in which a variance between images of the first image feature quantity or the second image feature quantity specified as the same object in the plurality of images is reduced;
An image search method comprising: The image search method according to claim 3,
The fourth step includes
An image search method, comprising: generating the transformation matrix from an average value of the first image feature quantity or the second image feature quantity specified as the same object in the plurality of images. The image search method according to claim 3,
The fourth step includes
When there are a first image and a second image including a first object and a second object among the plurality of images, an image feature amount of the first object of the first image, and a second image An image search method characterized by generating the transformation matrix from the image feature amount of the second object. A server for retrieving images with a processor and memory,
The server
A detection processing unit for detecting the first object and the second object from the input image;
A feature amount extraction unit that extracts a first image feature amount of the first object and a second image feature amount of the second object;
An object information generating unit that determines that the first object and the second object are different objects;
A transformation matrix generation unit that generates a transformation matrix such that a variance between the first image feature quantity and the second image feature quantity that are different objects increases;
The server characterized by having. The server according to claim 7,
A server further comprising: a search unit that receives an image to be searched and searches for the received image using the image feature amount converted by the conversion matrix. The server according to claim 7,
The detection processing unit
Detecting a first object and a second object from each of a plurality of input images;
The object information generation unit
It is determined that the first object and the second object are different objects in the same image, and the same object among the plurality of images is specified among the first object and the second object. A server characterized by that. The server according to claim 9, wherein
The transformation matrix generation unit
Generating a first transformation matrix having a large variance between the first image feature quantity and the second image feature quantity that are different objects, and specifying the same object in the plurality of images; A server that generates a second transformation matrix that reduces a variance between images of a first image feature amount or a second image feature amount. The server according to claim 9, wherein
The transformation matrix generation unit
The server that generates the transformation matrix from an average value of the first image feature quantity or the second image feature quantity specified as the same object in the plurality of images. The server according to claim 9, wherein
The transformation matrix generation unit
When there are a first image and a second image including a first object and a second object among the plurality of images, an image feature amount of the first object of the first image, and a second image A server that generates the transformation matrix from an image feature amount of the second object. A server with a processor and memory;
An image search system having an imaging device connected to the server,
The server
A detection processing unit that detects a first object and a second object from an image input from the imaging device;
A feature amount extraction unit that extracts a first image feature amount of the first object and a second image feature amount of the second object;
An object information generating unit that determines that the first object and the second object are different objects;
A transformation matrix generation unit that generates a transformation matrix such that a variance between the first image feature quantity and the second image feature quantity that are different objects increases;
An image search system comprising: The image search system according to claim 13,
A client computer connected to the server;
An image search system further comprising: a search unit that receives a search target image from the client computer and searches for the received image using the image feature quantity converted by the conversion matrix. The image search system according to claim 13,
The detection processing unit
Detecting a first object and a second object from each of a plurality of input images;
The object information generation unit
It is determined that the first object and the second object are different objects in the same image, and the same object among the plurality of images is specified among the first object and the second object. An image search system characterized by that.

Images