JP2007041964A - Image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of calculating a parameter quantifying the intimacy between persons who are subjects of a photograph and efficiently searching and viewing the photograph.
An image processing apparatus includes a photo input unit that inputs a photographed photo, a closeness calculation unit that calculates a closeness from the input photo, and a closeness that is output from the closeness calculation unit. As related information and linked to the photo input from the photo input unit 10 and stored in the data storage unit 13, and the database management unit 12 for searching for the necessary photo using the related information as a key, and the photo and its related information A data storage unit 13 to be stored, an extraction condition input unit 14 that receives an input of an extraction condition from a user, and accepts an extraction condition from the user, and a search result photo to the database management unit 12 with a search key based on the extraction condition Is output, and an output unit 16 that displays the extracted image data is provided.
[Selection] Figure 1

Description

  The present invention relates to an image processing apparatus for processing image data such as a still image or a moving image, and in particular, an image processing having a function of searching and classifying a favorite photograph from a group of image data stored in a database. Relates to the device.

  In recent years, personal photographing devices such as digital still cameras (hereinafter simply referred to as “digital cameras”) and mobile phones with cameras have been rapidly spreading. In such a photographing apparatus, the storage capacity has increased, and the user can take and record many photographs. However, increasing the amount of photos to be recorded makes it difficult to view image data in a limited time or to search for the desired photo from a large number of photos. It has been demanded.

  In many cases, photos taken with digital cameras are taken from people such as acquaintances and family members, and a new viewing method for selecting and viewing photos using the subject person as a search key, and viewing photos using a computer. As a method, there is a so-called electronic album tool that uses a keyword assigned to each photo to manage the photo and use it for viewing. For example, Adobe Photoshop Album, which is a product of Adobe Systems. In these tools, the user assigns in advance a person's name shown as image information to all pictures as a keyword, and narrows down the pictures showing the target person using the person name as a search key when viewing. A thumbnail display or a slide show display that is a sequential display is performed. However, this conventional method has a problem that it takes time and effort because the user needs to input and set keywords in advance for each of a large number of photographs taken.

  In addition, close friends and family members are assumed as the search targets when viewing photos, but close friends are often subject to photography in the first place, and frequently appear as subjects, In this case, it is not possible to narrow down the number of images suitable for comfortable viewing, and as a result, there arises a problem of viewing efficiency that the user has to view many photos.

Then, a method of estimating the personal relationship of the subject from the image information of the photograph and using it for searching for an image can be considered. For example, as a conventional technique for estimating the human relationship between subject persons, personal attributes such as age group and sex of the photographed person are estimated from the face of the person, and comprehensively based on the individual attributes estimated for all the subjects. An information providing apparatus that judges and estimates group attributes such as a lover, family, and friend is disclosed (see, for example, Patent Document 1). For example, this information providing apparatus performs grouping by estimating a group of “friends” or “lovers” for an image of a couple of teenage men and women.
JP 2004-227158 A

  However, in the above-mentioned patent document, based on the personal attributes of all the subjects, all the subjects are simply grouped by applying them to a group attribute that is assumed in advance, such as a lover, family, or friend. It is impossible to compare the relationship between persons as subjects in a photograph, for example, the intimacy of any two people as subjects in a single photograph.

  Also, with the widespread use of digital cameras and camera-equipped mobile phones, the amount of photos managed by individual users has increased, and as a method of searching for desired photos from a large number of photos, in addition to time-series sorting and album name search with album software Therefore, there is a demand for new entertainment methods that do not require user effort.

  The present invention has been made in view of these problems, and automatically calculates parameters related to intimacy between persons who are subjects from image data, and can efficiently search and view photos based on intimacy. An object of the present invention is to provide an image processing apparatus that can be used.

  In order to solve the above-described conventional problems, an image processing apparatus according to the present invention is an image processing apparatus that performs image processing using image data, a storage unit that stores the image data, and a subject from the image data A person detecting means for detecting a person, a distance measuring means for calculating a distance between each person when two or more persons are detected by the person detecting means, and a distance between the persons based on the distance. Parameter calculating means for calculating a parameter relating to the familiarity of the image data, and data extracting means for extracting image data stored in the storage means based on the parameter.

  With this configuration, the image processing apparatus according to the present invention calculates the distance between the persons shown in the photograph input by the distance measuring unit, and calculates the parameter related to the intimacy between the persons who are subjects in the parameter calculating unit. Then, by extracting the image data from the storage means by the data extraction means using the parameters, it is possible to search for, view, and the like photos based on the familiarity without requiring the user's trouble.

  Further, the image processing apparatus according to the present invention further includes a communication history holding unit that holds a communication history, and the parameter calculation unit uses the parameter related to familiarity based on the communication history held by the communication history holding unit. It is characterized by increasing or decreasing.

  Furthermore, the image processing apparatus according to the present invention further includes history holding means for holding at least one of a storage history, a reference history, and a correction history of the image data, and the parameter calculation means includes the history holding means. The parameter related to closeness is increased or decreased based on the history to be stored.

  With these configurations, parameter calculation is performed based on history information such as the storage history, reference history, and correction history of image data held in the history holding unit, and communication history with other users held in the communication history holding unit. The means increases or decreases the parameter related to the familiarity, and the data extracting means can extract the image data stored in the storage means based on the familiarity with higher accuracy.

  In order to achieve the above object, the present invention can be realized as an image processing method using characteristic constituent means of the image processing apparatus as steps, or as a program including all these steps. The program is not only stored in a ROM or the like included in the image processing apparatus, but can also be distributed via a recording medium such as a CD-ROM or a communication network.

  The image processing apparatus according to the present invention automatically calculates a parameter related to intimacy between persons who are photograph subjects from the distance between the persons who are subjects, and is useful for efficient search, viewing, etc. of a large number of photos. There is an effect of being able to.

  Embodiments of an image processing apparatus according to the present invention will be described below with reference to the drawings.

(Embodiment 1)
FIG. 1 is a functional block diagram showing the configuration of the image processing apparatus 100 according to the first embodiment.

  In general, the closer the distance between the two persons, the closer the two are considered to be intimate relations. Therefore, the image processing apparatus 100 according to the first embodiment uses the two persons shown in the photograph. It is characterized in that the intimacy, which is an index representing the strength of the relationship between human beings by the degree of intimacy, is quantified and calculated from the distance between them and used for searching and viewing photos.

  As shown in FIG. 1, an image processing apparatus 100 according to the present invention includes a photo input unit 10 that inputs a photographed photo, a closeness calculation unit 11 that calculates a closeness from the input photo, and a closeness calculation. A database management unit 12 that searches the necessary photos using the related information as a key, and stores the intimacy output by the unit 11 as related information in the data storage unit 13 linked to the photo input from the photo input unit 10; , A data storage unit 13 for storing a photograph and its related information, an extraction condition input unit 14 for receiving an extraction condition input from a user, and receiving an extraction condition from the user and managing a database with a search key based on the extraction condition The data extraction part 15 which outputs the photograph of a search result to the part 12 and the output part 16 which displays the extracted image data are provided.

  In addition, the closeness calculation unit 11 detects a region (hereinafter referred to as a face region) in which a person's face is captured from the input photograph, and outputs a person detection unit 21 and a person detection unit. A distance measurement unit 23 that calculates a distance between the two face regions from the face region detected in 21, and a parameter calculation unit 22 that calculates a parameter related to intimacy from the distance between the face regions calculated by the distance measurement unit 23. Prepare.

  As a specific method for managing the link between the image and the related information in the database management unit 12, a closeness table is stored in a part of the photo data file as represented by the Exif (Exchangeable image file format) format. It is also possible to use a common management method for the photo file name and the file name of the closeness table (for example, the photo file foo.pic and the closeness table file foo.dat). Alternatively, it may be managed by using pointer information for linking a photo file and a closeness table file.

  Further, the data storage unit 13 contains M photos Pic, and an identifier j (j = 1, 2,..., M) can be used as a subscript to specify an arbitrary photo as Pic (j). As a means for specifying a photo, a photo file naming rule or a pointer list indicating a photo storage location may be used.

  Next, an outline of the operation of the familiarity calculating unit 11 of the image processing apparatus 100 according to the first embodiment will be described. FIG. 2 is a flowchart showing the operation steps of the familiarity calculation unit 11.

  First, the person detection unit 21 of the familiarity calculation unit 11 detects a face area from the input photograph and counts the number of face areas (step S201).

  Then, the person detection unit 21 determines whether there are two or more detected face regions (step S202), and when two or more face regions are detected (Yes in step S202), the parameter calculation unit 22 The distance measuring unit 23 measures the distance between the detected faces (step S203), and calculates the familiarity from the measured distance (step S204). On the other hand, when two or more face areas are not detected (No in step S202), the process is terminated. Here, in step S201 for detecting a face area in the person detection unit 21, an edge image is extracted from the input photograph, and a rectangular image including the face image is extracted using a face learning dictionary prepared in advance. This method can be realized by “Patent Document: JP-A-2003-44853”.

  Although this conventional method is mentioned as a method for detecting a face area, it is needless to say that another method may be used as long as it is a method for detecting a face area, and the method is not limited to this. Also, it is assumed that an ellipse inscribed in the rectangular area including the extracted face image is a face area, or a skin color area in the rectangular area is used as a face area, thereby obtaining a face area closer to the actual face shape. Can do. Further, step S203, which is a distance calculation between faces, is further realized by an algorithm shown in FIG.

  FIG. 3 is a flowchart showing detailed operation steps of step S203 shown in FIG.

  First, assuming that the number of face areas detected in step S201 is N, the familiarity calculation unit 11 calculates a combination of two arbitrary face areas among the detected N face areas. Is determined (step S301).

  Next, a loop for g combinations in the familiarity calculation unit 11 is started (step S302), and the centroids of the two face areas are calculated (step S303), and the distance between the two face areas Is calculated (step S304), and the distance between the face regions is stored in the familiarity table T (j) (step S305). The process is repeated g times, and the loop for the g combinations ends (step S306). . Note that g is the number of combinations of all detected face areas obtained by (Equation 1) below. For example, when there are two subjects, g = 1, and when there are three subjects. g = 3.

(Equation 1)
g = _N C ₂ = N! / (2! (N-2)!)

  Further, in step S302, the familiarity calculating unit 11 sets parameters related to the familiarity calculated for each combination of the face areas of the person shown in the picture Pic (j) as shown in FIG. ) To create.

  FIG. 4 is a reference diagram illustrating an example of a data configuration of the closeness table T (j) of the photograph Pic (j).

  This closeness table T (j) is composed of a plurality of records, and each record uses an identifier (i = 1, 2,..., G) representing a combination of faces in one photograph, and each face combination. Is composed of a familiarity field for storing the familiarity F (i) corresponding to, and a combination field for storing a combination of face regions. The combination of the face areas of the closeness table T (j) is stored in the combination field, and the distance is temporarily stored in the closeness field. The distance temporarily stored in the familiarity field is converted into the familiarity in step S305, and stored in the familiarity field of the familiarity table T (j). Here, the familiarity table linked to the picture Pic (j) is represented as a familiarity table T (j).

  Next, a method for calculating the distance between face areas in the distance measuring unit 23 of the familiarity calculating unit 11 will be described more specifically.

  FIG. 5 is a reference diagram illustrating an example of a photograph 50 input to the photograph input unit 10 of the image processing apparatus 100 of the present invention. In the photograph 50 shown in FIG. 5, a person 51 and a person 52 are shown as subjects.

  FIG. 6 is a diagram illustrating the face areas of the person 51 and the person 52 in the photograph 50 shown in FIG. 5. The face area of the person 51 is represented by an ellipse 61 and the face area of the person 52 is represented by an ellipse 62. The center of gravity of the ellipse 61 is defined as the center of gravity 63, and the center of gravity of the ellipse 62 is defined as the center of gravity 64. The distance measuring unit 23 connects the two centroids 63 and 64 with a straight line, and measures the length of a line segment outside the area of the ellipse 61 and the ellipse 62 among the straight lines as the distance d between the face areas.

  From the distance d obtained by this method, the familiarity is calculated as (Equation 2) shown below. In (Expression 2), S is a closeness, α is a coefficient, and d is a distance between persons.

(Equation 2)
S = αd

  Here, in the image processing apparatus 100 according to the present invention, the output closeness is based on the distance between the faces, and the distance between the faces (between persons) is considered to be closer as the intimacy is increased. The smaller the value of the familiarity S obtained in Equation 2), the higher the actual intimacy between persons. Α is a number of 0 or more.

In addition, although the distance between persons was demonstrated as the distance between face areas, it is good also as a distance between the external shapes of a person including a body.
Further, the distance d between the face areas may be normalized by the size of the face area of the subject person.

  FIG. 7 is a reference diagram showing two photographs taken under the same conditions except for the zoom magnification.

  In FIG. 7, comparing the distances d1 and d2 between the face areas of the person 51 and the person 52 in a photograph 70 taken at a certain zoom magnification and a photograph 71 taken at a higher magnification zoom, the higher magnification can be obtained. D2 shot with zoom is larger than d1.

  Then, if the closeness of the photograph 70 and the photograph 71 is calculated as in (Expression 2), the value obtained in the (Expression 2) of the photograph 71 taken with the higher magnification zoom is larger than that of the photograph 70, and the intimacy is close. Becomes lower. However, since the same subject is taken under the same conditions, the photographs 70 and 71 should have the same closeness.

  Therefore, consider normalizing the distance d between the face regions by the width of the face region. By so doing, it is possible to correct an increase in the distance d due to the zoom magnification, and it is possible to calculate a desired familiarity. A formula for calculating the familiarity S for normalizing the distance d between the face areas using a and b, where a and b are the widths of the two face areas is shown in (Equation 3).

(Equation 3)
c = (a + b) / 2
α = 1 / c
S = αd

  When the distance between persons is the distance between the outlines of the persons including the body, the distance between the persons may be normalized by the width of the outline of the person.

  In (Equation 3), the familiarity S may be obtained by multiplying the distance d between the face regions by using the size of the photograph (for example, the number of pixels in the long side direction) as a coefficient.

  FIG. 8 is a reference diagram showing a photograph 80 and a photograph 81 obtained by trimming a part of the photograph 80 while paying attention to a person.

  As shown in FIG. 8, when a photograph that has already been taken is cropped by paying attention to a person, the person in the trimmed area is considered to be more important than other areas. It is desirable to increase the familiarity of the person in the photo. In this case, since the photograph is trimmed, the distance d1 between the persons of the subject does not change between the photograph 80 and the photograph 81, but the number of pixels y2 in the lateral direction of the photograph is smaller than y1. In the image processing apparatus 80 according to the present invention, it is possible to calculate a more desirable closeness S of the trimmed photograph based on the pixel numbers y1 and y2. A formula for calculating the familiarity S using the coefficient y depending on the number of pixels in the horizontal direction of the photograph is shown in the following (Formula 4).

(Equation 4)
α = y
S = αd

  By using this (Equation 4), it is possible to calculate the familiarity S using the size (number of pixels) of the photograph.

  In (Equation 4), the familiarity S is calculated by using y as the length in the horizontal direction of the photograph. However, the present invention is not limited to this. In FIG. 8, the horizontal direction is represented by y1 and y2, and the long side direction is used. However, the present invention is not limited to this, and the short side direction may be used.

  Further, the number of pixels y may be normalized by the width of the face area. A formula for calculating the familiarity in which the distance d between the face areas and the number of pixels y in the horizontal direction of the photograph are normalized by the width of the face area is shown in (Formula 5). Here, S is the familiarity, a is the width of the face of the person 1, b is the width of the face of the person 2, d is the distance between the persons, and y is a coefficient based on the number of pixels in the horizontal direction of the photograph.

(Equation 5)
c = (a + b) / 2
α = y / c ²
S = αd

  By using this (Equation 5), the familiarity calculating unit 11 can correct the increase in the distance d due to the zoom magnification and calculate a more desirable familiarity of the trimmed photograph.

  Next, a method for extracting image data in the image processing apparatus 100 using the familiarity calculated by the above configuration and method will be described.

  When a person name is assigned to each face area of the familiarity table T (j) for each picture Pic (j) stored in the data storage unit 13 of FIG. The data extraction unit 15 receives the person name, and the photo Pic (j) in which the face area of the target person exists among the photos stored in the data storage unit 13 can be extracted by the database management unit 12.

  However, if a large number of photographs showing the target person are stored, the user needs to further narrow down the extracted photographs, making it difficult to view efficiently. Therefore, in the present invention, by using the intimacy calculated by the intimacy calculating unit 11, the photos can be sorted in the order of intimacy, and the user can view the photos in descending order of intimacy. For this reason, the user can preferentially and efficiently view the extracted photo from the photo showing the friendship. Here, a person name is assigned to each face area, but even if no person name is assigned, any face can be specified by specifying the face area from the photograph with an input means such as a mouse. You can select an area.

  FIG. 9 is a flowchart showing operation steps of the database management unit 12, the data extraction unit 15, and the output unit 16 of the image processing apparatus 100 according to the first embodiment.

  First, the data extraction unit 15 receives the names of P people input to the extraction condition input unit 14 and instructs the database management unit 12 to search for a photo in which all of the P people are shown (step S1). S901).

  FIG. 10 is a reference diagram showing an example of the photo search input screen 103 that accepts the names of P people.

  In FIG. 10, a text input box 101 for inputting a person name and a button 102 for instructing execution of a search are displayed on the photo search input screen 103.

  FIG. 11 is a reference diagram showing an extracted data list L1 in which photographs extracted from the data storage unit 13 in the database management unit 12 are listed, and a sorted data list L2 in which the data in the extracted data list L1 is rearranged. .

  The extracted data list L1 shown in FIG. 11A includes a plurality of records, and each record includes a field that stores a photo ID that is an identifier of the extracted photo, and a field that stores a representative familiarity described later. Become.

  The sorted data list L2 shown in FIG. 11B includes a plurality of records. Each record has a field for storing a photo ID for managing the extracted photo, a field for storing a representative familiarity, and a record. It consists of a field for storing numbers.

  The database management unit 12 repeats the loop of the next processing (step S903 to step S905) for all M photos stored in the data storage unit 13 (step S902). That is, the jth photo Pic (j) and the closeness table T (j) linked to the photo Pic (j) are extracted from the data storage unit 13 (step S903). Then, the database management unit 12 examines the combination fields of all the records of the familiarity table T (j) and includes all the names of P persons (that is, all the P persons are shown in the picture Pic (j)). (Step S904).

  If all P people are included in the photo (Yes in step S904), a record of a combination of j as the photo ID and the representative familiarity of the photo Pic (j) is added to the extracted data list L1. (Step S905). On the other hand, if not all P persons are included (No in step S904), the process shifts to the processing loop for the next picture Pic (j + 1).

  Here, the representative familiarity of the picture Pic (j) is, for example, when two person names A and B are input in the text input box 161, the two persons A and B are shown (the person name is It is assumed that the familiarity F (i) corresponding to the combination of the face areas of A and B from the familiarity table T (j) linked to the photograph Pic (j) given). When the names of three persons A, B, and C are entered in the text input box 161, they are linked to a picture Pic (j) in which three persons are shown (person names are given). From the familiarity table T (j), the familiarity of A and B, the familiarity of A and C, and the familiarity of B and C are set as the representative familiarity, or the familiarity of A and B , A and C intimacy, and B and C average intimacy are set as representative intimacy. When all the M times loops are completed (step S906), the data extraction unit 15 sorts all the records in the extraction data list L1 in ascending order using the representative familiarity as a key, and the sorted result is sorted into the sorted data list L2. Store (step S907).

  In the description of the data extraction unit 15 described above, it has been described that the closeness is given to the photo in advance, but the closeness may be calculated at the time of data extraction.

  Next, the data extraction unit 15 sequentially reads the records in the sorted data list L2 from the top, and sequentially displays the photos corresponding to the photo IDs on the screen (step S908). In addition, as a display method other than sorting, ranking can be performed and pictures can be displayed. Here, the rank can be classified as A, B, C, etc. according to the value of intimacy. Specifically, rank A indicating that the familiarity is high if the representative familiarity is 0 or more and less than 50, rank B indicating that the familiarity is medium if the representative familiarity is less than 100 or less, and 100 or more. For example, rank C indicates that the intimacy is low.

  FIG. 12 is a reference diagram of a screen 120 in which photographs are arranged in order of closeness.

  In FIG. 12, the screen 120 displays nine photos from left to right and from top to bottom in ascending order of familiarity. At this time, by displaying the record number of the data list L2 at the upper left of each photograph and the rank of the intimacy at the lower part, an easy-to-understand display can be realized. It is also possible to display only photos of any rank.

  The user can select a photograph that he / she likes on the screen 120 using an input means such as a mouse, and can select or enlarge the photograph. In addition, in the image processing apparatus 100 according to the first embodiment, for example, when selecting a family photo to be used for a New Year's card, a plurality of photos showing the family can be viewed in the order of closeness, You can easily select the best photo.

  In addition to displaying an image as a result of extraction, as a result of data extraction, a list of pointer information to image data and related information such as a photo ID and a file name of image data is output as a data extraction result list. Also good.

  As described above, in the image processing apparatus 100 according to the first embodiment, the familiarity calculating unit 11 is based on the distances of the face area of the subject measured by the distance measuring unit 23, and 5), it is possible to calculate the closeness S between persons who are subjects, and the data extraction unit 15 includes the closeness S in a large number of photographs stored in the data storage unit 13. Therefore, it can be used for searching for a desired photo, and can be used for efficient search and viewing of a large number of photos.

  Also, by presenting the extracted photos to the user in descending order of closeness, the number of views until reaching the target image can be reduced, and an efficient photo search and viewing method can be proposed.

  Furthermore, photos are grouped according to intimacy and classified into groups such as “good friends” group, “somewhat good friends” group, and “not so good friends” groups, and presented to users. It is possible to provide a new sorting method for entertaining photos.

(Embodiment 2)
Next, a second embodiment of the image processing apparatus according to the present invention will be described with reference to the drawings. Note that the image processing apparatus according to the second embodiment is characterized in that, when calculating the familiarity, the number of subjects, the orientation of the face, the facial expression, and the like are determined, and parameters related to the familiarity are changed. .

  FIG. 13 is a functional block diagram of the image processing apparatus 1300 according to the second embodiment. The image processing apparatus 1300 according to the second embodiment includes a closeness calculation unit 1301 that uses the number of persons, face orientation, and facial expression of a subject in addition to the distance between subjects. In addition, the same code | symbol is attached | subjected about the structure similar to the image processing apparatus which concerns on Embodiment 1 mentioned above, and the detailed description is abbreviate | omitted.

  The familiarity calculation unit 1301 illustrated in FIG. 13 includes a person detection unit 21 and a parameter calculation unit 1302. The parameter calculation unit 1302 includes a number processing unit 1303, a face orientation determination unit 1304, in addition to the distance measurement unit 23. And a facial expression determination unit 1305.

  The parameter calculation unit 1302 calculates the familiarity using a value output from at least one of the number processing unit 1303, the face orientation determination unit 1304, or the facial expression determination unit 1305. With such a configuration, it is possible to calculate the familiarity more reflecting the situation at the time of shooting.

  Next, the processes of the number processing unit 1303, the face orientation determination unit 1304, and the facial expression determination unit 1305 of the parameter calculation unit 1302 will be described in order.

  The number processing unit 1303 determines how many people are in the input photo based on the number N of face areas detected by the person detection unit 21 and outputs a coefficient K. For example, in a group photo, since a large number of people are captured in a limited frame, the distance between each person becomes close. Since this is an intimacy, the distance is not close. Therefore, in a photo such as a group photo, it is desirable that the value of intimacy be large in order to reduce the intimacy.

  Therefore, the number of persons processing unit 1303 outputs K = N−1 as a coefficient that has a minimum intimacy when N = 2 and gradually increases as N increases. By calculating the familiarity using the coefficient K output from the number processing unit 1303, it is possible to calculate the familiarity in consideration of the number of people in the photograph. This is effective when calculating the intimacy of a group photo such as a group photo, and by calculating the intimacy using a coefficient K = N-1 as a group photo A more desirable intimacy of a photo can be calculated.

  The face orientation determination unit 1304 detects the orientation of each face area detected by the person detection unit 21 and calculates a coefficient E from the face area orientation.

  FIG. 14 shows a top view of the positional relationship between the camera and the heads 141 and 142 of the subject persons 51 and 52 when the photograph 50 shown in FIG. 5 is taken.

  The direction 144 and the direction 147 shown in this figure pass through the centers of the heads 141 and 142, respectively, and are parallel to the optical axis of the camera, and the directions 143-145 and 146-148 pass through the centers of the heads 141 and 142, respectively. Intersects the direction 144 and the direction 147 at right angles on the plane. The directions of the heads 141 and 142 are θ1 and θ2 clockwise from the directions 145 and 148 that are leftward with respect to the camera.

  It is determined that the faces of the head 141 and the head 142 are intimate rather than facing the direction 144 and the direction 147, respectively, and facing the direction 145 and the direction 146, respectively.

  Therefore, for example, the coefficient E1 to be applied to the familiarity when the face direction of the head 141 is the direction 145 is E1 = 0.8, E1 = 1.0 when the direction is 144, E1 = 1. 2. This E1 can be changed linearly as shown in FIG. 15 in accordance with the change of the angle θ1.

  FIG. 15 is a diagram illustrating changes in the coefficients E1 and E2 depending on the face orientations (angles θ1 and θ2) of the heads 141 and 142 shown in FIG.

  As shown in FIG. 15A, the person 141 sets the coefficient E1 to be applied to the intimacy when the face orientation is the direction 145 to E1 = 0.8, and when the face 144 is the direction 144 to E1 = 1.0 and the direction 143 In this case, E1 = 1.2 and can be changed linearly according to the change of the angle.

  Further, as shown in FIG. 15B, the person 142 sets the coefficient E2 to be applied to the intimacy when the face orientation is the direction 146 to E2 = 0.8, and when the face direction is the direction 147, E2 = 1.0. In the direction 148, E2 = 1.2, and can be changed linearly according to the change in angle.

  Thus, the face orientation determination unit 1304 calculates the coefficient E from the coefficients E1 and E2 obtained from the respective face orientations. The coefficient E is, for example, the product of E1 and E2. By using this coefficient E for calculating intimacy, it is possible to calculate intimacy in consideration of the face direction of the subject. The detection of the face orientation is described in detail in the above-mentioned “Patent Document: JP-A-2003-44853”. In the first embodiment, it is assumed that this conventional method is used as the face orientation detection method. However, any other method may be used as long as it can detect the face orientation. Needless to say, the method is not limited.

  The expression determination unit 1305 determines an expression from the face area detected by the person detection unit 21, and calculates a coefficient H based on the type of expression. When taking a picture, if the subject person wants to take pictures happily, the expression can also be fun. Therefore, the expression of joy is used as an expression that expresses fun compared to the expressionless face. For example, if the facial expression indicated by the face area is a joyful facial expression, H = 0.8. An expression other than pleasure may be used. For example, if the expression is angry, H = 1.2 can be set. By using this coefficient H, it is possible to calculate a more desirable closeness in consideration of the facial expression of the subject.

As a facial expression determination method, an image including a face is input, and the facial feature point movement is obtained by comparing the prepared expressionless face with the input full expression face. , “Angry”, “Sorrow”, “Fear”, and “Surprised” have been proposed. For details, see “Zudenek et al.“ Automatic Extraction Method of Facial Expression Features Using Motion Information ”. IEICE Transactions Vol. J81-D-II, No. 6, pp. 1150-1159, 1998”
It is stated in. When this expression determination method is applied to the second embodiment, it is necessary to register an expressionless face in advance. In the second embodiment, description will be made assuming that this conventional method is used as a facial expression determination method. However, another method may be used as long as it can determine a facial expression, and the facial expression determination method is not limited. Needless to say.

  By using the familiarity calculation unit 1301 having such a configuration, it is possible to calculate the familiarity that more reflects the situation at the time of shooting using the number of people, the orientation of the face, and the facial expression in addition to the distance between the persons. . A formula for calculating the intimacy S using the number of people, the direction of the face, and the expression in addition to the distance between the persons is shown in (Formula 6). Here, K is a coefficient based on the number of subjects, E is a coefficient based on the face orientation, and H is a coefficient based on facial expressions.

(Equation 6)
c = (a + b) / 2
α = (y / c ² ) × KEH
S = αd

  The normalization based on the width of the face area, the coefficient K based on the number of people, the coefficient E based on the face orientation, and the coefficient H based on the facial expression can be used in any combination. Further, when extracting data, a photograph stored in the data storage unit 13, a communication history, or the like may be used.

  Next, a method for performing data extraction using related information of photographs stored in the data storage unit 13 at the time of data extraction will be described.

  FIG. 16 shows operation steps of the database management unit 12, the data extraction unit 15, and the output unit 16 when changing the representative familiarity of the extracted photo using the related information of the photo stored in the data storage unit 13. It is a flowchart showing. A method using statistical processing will be described as one of the methods using the related information of photographs stored in the data storage unit 13.

  In this figure, the processing from step S1601 to step S1604 is the same as the processing from step S905 to step S908 in step S901 to step S904 and the processing from step S1606 to step S1609 in FIG. Therefore, the description is omitted.

  In this figure, the database management unit 12 examines the combination fields of all the records in the familiarity table T (j) and includes all the names of P persons (that is, all the P persons are included in the picture Pic (j)). After the process (step S1604) to check whether the photograph is listed in the extracted data list L1, all or a part of the photographs stored in the data storage unit 13 are statistically processed as a mother set. Is used to temporarily change the familiarity F (i) stored in the familiarity table T (j) linked to Pic (j) (step S1605). Then, using the familiarity F (i) temporarily changed in step S1605, the representative familiarity is calculated from the familiarity table T (j), and the extracted data list L1 is created (step S1606).

  With this method, in the image processing apparatus 1300 according to the second embodiment, for example, photographs extracted with a person name and listed in the extracted data list L1 are stored in the data storage unit 13. The intimacy can be changed according to what proportion it occupies. Specifically, for example, among the photos stored in the last month, if the number of photos of the target two people is the largest, in order to strengthen the intimacy of the two people, Is increased by 0.8 times in step S1605.

  As described above, the image processing apparatus 1300 according to the second embodiment includes the number-of-people processing unit 1303, the face orientation determining unit 1304, and the facial expression determining unit 1305 in the familiarity calculating unit 1301. It can be automatically changed according to the shooting situation such as the number of subjects, facial expressions, face orientation, etc., and more accurate intimacy can be calculated for more appropriate photo search, classification, etc. Is possible.

  In addition, the familiarity calculated at the time of shooting is updated at the time of viewing using the reference history, the storage history, the correction history, the number of times the subject is used, etc., thereby calculating the familiarity more suited to the user's preference. Thus, it is possible to provide the user with search and viewing in consideration of the human relationship during viewing.

(Embodiment 3)
Next, a third embodiment of the image processing apparatus according to the present invention will be described with reference to the drawings. Note that the image processing apparatus according to the third embodiment includes a communication history holding unit that records a communication history such as mail transmission / reception, and changes the closeness according to the communication history.

  FIG. 17 is a functional block diagram of an image processing apparatus 1700 to which a communication history holding unit 1701 according to the third embodiment is added. In FIG. 17, the same functional configuration as that of the image processing apparatus according to the first embodiment described above is denoted by the same reference numeral, and detailed description thereof is omitted.

  The communication history holding unit 1701 holds a history of communication such as mail transmission / reception with other terminals. In the present invention, the communication configuration itself is not important and is not particularly described. However, with regard to the communication history, the communication history is used in step S1605 in FIG. Can be used, and can be used as an intimacy representing the user's personal relationship more accurately.

  For example, when communication such as email exchange or photo exchange with a specific person is more than communication with another person, a communication history is recorded in the communication history holding unit 1701, and the person and the user Above, it can be judged that it is the most intimate. In this way, by increasing or decreasing the intimacy obtained from the photograph with the intimacy that can be estimated from the communication history, it is possible to use intimacy that more accurately represents the user's personal relationship.

  In the third embodiment, a photo is taken as an example of data extraction. However, the present invention is not limited to a photo, and a moving image or the like may be used. In addition, the use of intimacy is not only the extraction of data, but it also automatically distributes data to people who are intimate, and provides conditions that prohibit copying and redistribution to data that is distributed to people who are not very intimate It can also be used for attached distribution.

  As described above, according to the image processing apparatus 1700 according to the third embodiment, it is possible to change the intimacy between persons who are subjects of a photograph using the communication history recorded in the communication history holding unit 1701. Therefore, by using the communication history, image data that matches the situation at the time of extraction is extracted based on a more precise familiarity, and the user can search and view photos that take into account human relationships at the time of viewing. Can be provided.

(Modification)
Next, a modification of the image processing apparatus according to the present invention will be shown. The present modification is characterized in that the closeness is calculated by accurately grasping the actual distance at the time of shooting between subjects using a distance sensor such as GPS.

  FIG. 18 is a reference diagram in the case where three persons are photographed in the front-rear direction at the time of photographing, and FIG. 18A is a diagram of the three persons as subjects viewed from above. FIG. 18B is a reference diagram of a photograph taken with the arrangement of FIG.

  Therefore, in this modification, as shown in the functional block diagram of the image processing apparatus 1900 in FIG. 19, for example, a distance sensor 1901 such as a GPS or a distance measuring laser is provided to measure the distance between persons in real space. Then, based on this accurate distance, the parameter calculation unit 22 calculates the closeness between persons. In the image processing apparatus 1900 of this modification, not only the distance between face areas on a photograph but also the distance between persons in a real space can be measured and used for calculating a new density.

  Accordingly, in the image processing apparatus 1900 of this modification, it is possible to determine the intimacy by giving the distance between persons in the real space to the photograph, and it is possible to search for image data based on the more intimate intimacy. Viewing is possible.

  In the above description of the embodiment, the data extraction unit for extracting photo data from the data storage unit and the output unit for displaying the photo data on the screen are provided. However, these processing units are provided. For example, it is possible to consider a closeness calculation device that simply calculates closeness from the distance between persons without using photo data.

  An image processing apparatus according to the present invention includes, for example, a digital still camera that captures a still image or a moving image, a photographing device such as a camera-equipped mobile terminal or a movie camera, an image scanner that reads a still image or a moving image, or a data reading device such as a movie capture. The present invention can be applied to an apparatus, a PC, or the like that processes image information of a still image or a moving image read in the above.

Functional block diagram showing the configuration of the image processing apparatus according to the first embodiment A flowchart showing the operation steps of the intimacy calculation unit The flowchart which shows the detailed operation | movement step of step S203 shown in FIG. Reference diagram showing an example of the data structure of the closeness table T (j) of the picture Pic (j) Reference diagram showing an example of a photograph input to the photograph input unit of the image processing apparatus The figure showing the face area of the person of the photograph shown in FIG. Reference diagram showing two photos taken under the same conditions except for zoom magnification Reference diagram showing a photograph that crops a part of the photo focusing on a person 6 is a flowchart showing operation steps of a database management unit, a data extraction unit, and an output unit of the image processing apparatus according to the first embodiment. Reference diagram showing an example of a photo search input screen that accepts P's name Reference diagram showing an extracted data list L1 that lists photos extracted from the data storage unit in the database management unit, and a sorted data list L2 in which the data in the extracted data list L1 is rearranged Reference diagram of screen with photos arranged in order of familiarity Functional block diagram of an image processing apparatus according to Embodiment 2 FIG. 5 is a top view of the positional relationship between the camera and the head of the subject person when the photograph shown in FIG. 5 is taken. The figure which shows the change of the coefficients E1 and E2 by the direction of the head face shown in FIG. The flowchart showing the operation steps of the database management unit, the data extraction unit, and the output unit when changing the representative familiarity of the extracted photo using the related information of the photo stored in the data storage unit Functional block diagram of an image processing apparatus to which a communication history holding unit according to Embodiment 3 is added Reference diagram when three people are photographed in a line up and down at the time of shooting Functional block diagram of image processing apparatus

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Photo input part 11,1301 Closeness calculation part 12 Database management part 13 Data storage part 14 Extraction condition input part 15 Data extraction part 16 Output part 21 Person detection part 22,1302 Parameter calculation part 23 Distance measurement part 50 Photograph 51,52 Subject person 100, 1300, 1700, 1900 Image processing device 1303 Number processing unit 1304 Face direction determination unit 1305 Expression determination unit 1701 Communication history holding unit 1901 Distance sensor

Claims (16)

An image processing apparatus that performs image processing using image data,
Storage means for storing the image data;
Person detecting means for detecting a person as a subject from the image data;
A distance measuring means for calculating a distance between each person when two or more persons are detected by the person detecting means;
Parameter calculating means for calculating a parameter related to the intimacy between the persons based on the distance;
An image processing apparatus comprising: data extraction means for extracting image data stored in the storage means based on the parameters. The person detection means includes a face area detection unit that detects a face area of the person as a subject,
The distance measuring means calculates a distance between the face areas detected by the face area detecting unit;
The image according to claim 1, wherein the parameter calculation unit calculates a parameter indicating that the closeness between the persons as subjects is high when the distance is short in proportion to the distance. Processing equipment. The distance measuring means calculates the distance by normalizing the distance between the centroids of the two face areas detected by the face area detecting unit with the size of the face area of the person who is the subject. The image processing apparatus according to claim 2, wherein: The image processing apparatus further includes:
Comprising pixel number detecting means for detecting the number of pixels of the image data;
The image processing apparatus according to claim 1, wherein the parameter calculation unit increases or decreases the parameter related to familiarity according to the number of pixels of the image data. The image processing apparatus further includes:
A number-of-people detection means for detecting the number of persons who are subjects of the image data;
The image processing apparatus according to claim 1, wherein the parameter calculation unit increases or decreases a parameter related to intimacy according to the number of persons detected by the number of persons detection unit. The image processing apparatus further includes:
A face orientation judging means for judging the orientation of the face of the person who is the subject of the image data;
The image processing apparatus according to claim 1, wherein the parameter calculation unit increases or decreases the parameter related to familiarity based on a face orientation of the person detected by the face orientation determination unit. The image processing apparatus further includes:
Equipped with facial expression determination means for determining the facial expression of the person who is the subject of the image data;
The image processing apparatus according to claim 1, wherein the parameter calculation unit increases or decreases the parameter related to intimacy based on the facial expression of the person detected by the facial expression determination unit. The image processing apparatus further includes:
A distance sensor for measuring the actual distance between the persons,
The image processing apparatus according to claim 1, wherein the parameter calculation unit calculates the parameter related to intimacy based on a distance detected by the distance sensor. The image processing apparatus further includes:
Communication history holding means for holding a communication history is provided,
The image processing apparatus according to claim 1, wherein the parameter calculation unit increases or decreases the parameter related to intimacy based on a communication history held by the communication history holding unit. The image processing apparatus further includes:
A history holding means for holding at least one of a storage history, a reference history, and a correction history of the image data;
The image processing apparatus according to claim 1, wherein the parameter calculation unit increases or decreases the parameter related to familiarity based on a history held by the history holding unit. The storage means further stores a closeness table in which a relationship between at least the face area and the parameters related to the closeness is described.
The image processing apparatus further includes:
Input means for inputting extraction conditions from the storage means of the image data;
The image processing apparatus according to claim 2, wherein the data extraction unit extracts image data satisfying the extraction condition from the storage unit with reference to the closeness table. An image processing method used in an image processing apparatus that performs image processing using image data,
A storing step for storing the image data;
A person detecting step of detecting a person as a subject from the image data;
A distance measuring step of calculating a distance between each person when two or more persons are detected in the person detecting step;
A parameter calculating step for calculating a parameter related to the intimacy between the persons based on the distance;
A data extraction step of extracting the image data stored in the storage step based on the parameter. The person detection step includes a face area detection step of detecting a face area of the person as a subject,
In the distance measuring step, the distance between the face areas detected in the face area detecting step is calculated,
13. The parameter calculation step according to claim 12, wherein when the distance is short in proportion to the distance, a parameter indicating that the closeness between the persons who are subjects is high is calculated. Image processing method. In the distance measurement step, the distance is calculated by normalizing the distance between the centroids of the two face areas detected by the face area detection unit with the size of the face area of the person as a subject. The image processing method according to claim 13. A program for causing a computer to realize the image processing method according to any one of claims 12 to 14. A computer-readable recording medium, wherein the program according to claim 15 is recorded.

Images