JP2008178042A - Image search device, imaging device, control method of image search device, program, and storage medium - Google Patents

Abstract

An object of the present invention is to easily and flexibly search for whether or not a target subject candidate is included in image data.
An upper category subject candidate image satisfying the upper category condition is extracted under the upper category condition, and a lower category subject candidate image satisfying the lower category condition is selected under the lower category condition. An image search device that displays that an image of a subject that satisfies the search condition is included in the image data to be captured when an image of a candidate for a lower category subject is selected.
[Selection] Figure 1

Description

  The present invention relates to an image search device, an imaging device, a control method for an image search device, a program, and a storage medium that search for a specific subject included in image data.

There has been known a technique for automatically recording image data when various subjects (particularly people) included in the image data are in a desired state (pause, facial expression, etc.). For example, an automatic imaging system described in Patent Document 1, an image evaluation apparatus described in Patent Document 2, a digital camera described in Patent Document 3, and a camera described in Patent Document 4 by the present applicant are proposed. Has been. Further, the face image recording apparatus described in Patent Document 5 discloses a configuration in which one person is imaged a plurality of times, and face image data with good image quality is selected and recorded. Further, in the technique disclosed in Patent Document 6, a method of searching for an image using a hierarchically classified category and keyword and a classification tree of image features is proposed.
JP 2004-294498 A JP 2004-46591 A Japanese Patent Laid-Open No. 2001-211377 JP 2001-51338 A JP 2003-99763 A Patent Registration 03683481

  In the above-described conventionally proposed techniques, an image of a specific person desired by the user is searched from image data in which a plurality of various persons and objects exist as subjects, and the specific person's There is no automatic search for an image being in a particular state. An efficient method related to an image search technique capable of displaying the result or recording image data including an image in which a specific person is in a specific state on a recording medium is proposed. It has not been. Specifically, a person's image is specified from the input image data and the person's state is evaluated, or in the group photo shooting, only the image of the interested person to be taken is specified. An efficient method for conducting state assessment has not been proposed yet.

  Further, in the above-described conventional technique, the image input unit is not controlled so that the imaging state of the image of the input image data is suitable for extraction or selection of a subject of a specific condition or a specific category. . Therefore, there is a problem that there is no guarantee that image data containing subject candidates belonging to a predetermined category can be recognized regardless of the imaging environment. Furthermore, the above-described conventional technique has a problem in that various image search conditions or image search narrowing conditions cannot be set flexibly.

  Accordingly, an object of the present invention is to provide an image search device that can easily and flexibly search for whether or not an image of a target subject candidate is included in image data, and an imaging device equipped with the image search device. There is.

In order to achieve the above object, an image search device according to an embodiment of the present invention provides:
Image input means for inputting image data;
Search condition setting means for setting a subject search condition including a higher category condition and a lower category condition associated with the higher category condition;
Subject candidate extraction means for extracting upper category subject candidates that satisfy the upper category condition from subjects included in the image data;
A selection means for selecting, as the extracted upper category subject candidate, a lower category subject candidate that satisfies the lower category condition associated with the upper category condition;
Display means for displaying that the image data includes a candidate candidate that satisfies the search condition when the lower category subject candidate is selected.

In order to achieve the above object, an imaging apparatus according to another embodiment of the present invention includes:
In an imaging apparatus comprising image input means for inputting image data, display means, and recording means,
Search condition setting means for setting a subject search condition including a higher category condition and a lower category condition associated with the higher category condition;
Subject candidate extraction means for extracting upper category subject candidates that satisfy the upper category condition from subjects included in the image;
A selection means for selecting, as the extracted upper category subject candidate, a lower category subject candidate that satisfies the lower category condition associated with the upper category condition;
An image search device comprising: display means for displaying that the image data includes a candidate subject that satisfies the search condition when the lower category subject candidate is selected. .

In order to achieve the above object, a method for controlling an image search apparatus according to still another embodiment of the present invention includes:
An image input process for inputting image data;
A search condition setting step for setting a subject search condition including a higher category condition and a lower category condition associated with the higher category condition;
A subject candidate extraction step of extracting a higher category subject candidate that satisfies the higher category condition from subjects included in the image;
A selection step of selecting a lower category subject candidate that satisfies the lower category condition associated with the upper category condition as the extracted upper category subject candidate;
A display step of displaying that the image data includes a candidate candidate that satisfies the search condition when the lower category subject candidate is selected.

  According to the present invention, it is possible to provide an image search device that can easily and flexibly search for whether or not a target subject candidate is included in image data, and an imaging device equipped with the image search device. It becomes.

<Embodiment 1>
Hereinafter, an image search apparatus according to Embodiment 1 of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram illustrating a main part of an imaging apparatus 100 including an image search apparatus according to Embodiment 1 of the present invention. The imaging apparatus 100 includes an imaging optical system 1, an imaging device 2 composed of a CCD or CMOS sensor, and a sensor peripheral circuit 3 (including an AD conversion circuit, a sensor signal processing circuit, and a sensor drive circuit (not shown)). These circuits constitute an image input unit.

  Furthermore, the video signal from the image sensor 2 is AD-converted by the sensor peripheral circuit 3 and converted into image data. The imaging apparatus 100 includes a storage circuit 4 such as a RAM or a cache memory that temporarily holds image data obtained by such conversion, and an imaging mode setting unit 5. Although not shown, the imaging optical system 1 includes an exposure control mechanism and a focus detection control mechanism.

  The search condition setting unit 6 includes a search condition data storage unit (ROM, RAM, etc.) 7 for storing image search condition data, and allows the user to set upper category conditions and lower category conditions that are image search conditions. It is. The user selects a search condition stored in the search condition data storage unit 7 by using the operation unit 8 to select a GUI for directly inputting the upper category condition and the lower category condition which are search conditions, and sets the search condition.

  For this purpose, the search condition setting unit 6 includes a display display control circuit for generating respective GUIs, and these GUIs are displayed on the display unit 9. Various data for specifying the upper category condition and the lower category condition input by the user are stored in the search condition data storage unit 7 in association with the upper category condition and the lower category condition.

  The subject candidate extraction unit 10 reads the image data temporarily stored in the storage circuit 4 and extracts an image of a specific subject candidate (for example, a face) that satisfies the upper category condition included in the image data. The image of the specific subject candidate (eg, face) extracted here is referred to as a higher category subject candidate here.

  The subject candidate selection unit 11 also selects subject candidate images that satisfy the designated lower category condition (a face registered in advance by a specific person) from the extracted upper category subject candidate images. The subject candidate selected here is referred to as a lower category subject candidate here.

  These upper category conditions and lower category conditions are supplied and set by the search condition setting unit 6 together with feature data related to the subject candidate extraction unit 10 and the subject candidate selection unit 11.

  In addition, the imaging apparatus 100 includes an imaging condition control circuit 12, an image data compression / decompression circuit 13, a removable recording medium such as a flash memory, or a recording / reproducing circuit 14 that performs recording on a recording medium such as a hard disk. The image data recorded on the recording medium can be reproduced by the recording / reproducing circuit 14 and displayed on the display unit 9 via the image data compression / decompression circuit 13.

  The operation of the imaging apparatus 100 is basically controlled according to the control unit 15. A non-volatile memory 16 is connected to the control unit 15, and the imaging device 100 is controlled according to a control program stored in the non-volatile memory 16. Similarly, the operation of the embodiment of the present invention is also controlled. The flash unit 17 is provided in the imaging apparatus 100 as necessary.

  The imaging condition control circuit 12 controls imaging conditions such as focus, exposure time, and white balance for the imaging optical system 1, the imaging device 2, and the sensor peripheral circuit. Therefore, the exposure control mechanism and the focus detection control mechanism included in the imaging optical system 1 are controlled to control the exposure state and the focus state, and the sensor peripheral circuit 3 controls the white balance. When manually setting the imaging conditions, a cross key, a determination key, a power switch, a release switch, a zoom lever, and the like provided on the operation unit 8 are basically used.

  The distinction among the control unit 15, the subject candidate extraction unit 10, the subject candidate selection unit 11, and the imaging condition control circuit 12 is for convenience of explanation. Therefore, the present invention can be implemented as an implementation by software processing on a single processor or as a multi-core chip. Here, the meaning of one processor includes a CPU, a configurable processor, a DSP, a media processor, and the like, and the meaning of a multi-core chip includes a circuit as a hard IP, a soft IP, or a mixture of both.

  In the first embodiment, it is assumed that after the upper category subject candidate image is extracted by the face detection process based on the upper category condition, the lower category subject candidate image is selected by, for example, the personal identification process based on the lower category condition. That is, since the current image data includes an image of a subject that matches the conditions, this image data is determined as image data to be captured.

  Thereby, under the control of the control unit 15, the corresponding current image data from the image sensor 2 constituting the image input unit is operated to be recorded on the recording medium through the recording / reproducing circuit 14. The image data compression / decompression circuit 13 is a circuit that executes standardized compression encoding algorithms such as MPEG versions for moving images and JPEG versions for still images. In this way, it is possible to reliably record image data including a subject desired by the user.

  The imaging mode setting unit 5 inputs information on the set (selected) imaging mode and displays the information on the display unit 9. Like the control unit 15 and the imaging condition control circuit 12, the imaging mode setting unit 5 uses various processors and the like. It can also be implemented. Therefore, the control unit 15, the imaging condition control circuit 12, and the imaging mode setting unit 5 may be configured to be mounted on the same processor.

  As described above, the imaging condition control circuit 12 is for manually or automatically controlling focus conditions, exposure conditions, white balance, and the like. In the first embodiment, first, the imaging mode is set by the imaging mode setting unit 5 by the user. Accordingly, the exposure control mechanism, the focus detection control mechanism, and the sensor peripheral circuit 3 are controlled so that image data from which upper category subject candidates satisfying the upper category condition determined by the search condition setting unit 6 can be easily obtained. For this purpose, the imaging condition control circuit 12 controls a focus condition, an exposure condition, a white balance condition, and the like.

  FIG. 2 is a circuit block of the subject candidate extraction unit 10. The subject candidate extraction unit 10 includes a feature extraction circuit 101 that performs a predetermined type of feature extraction in accordance with the upper category condition, and a determination circuit 102 that determines whether an image output from the feature extraction circuit 101 satisfies the upper category condition. , And the upper category condition input circuit 103. The feature extraction circuit 101 receives the current image data temporarily stored in the storage circuit 4 and detects, for example, edge features of a plurality of specific direction components, and outputs subject feature data. The upper category condition input circuit 103 receives the upper category condition and its feature data given from the search condition setting unit 6 and supplies them to the determination circuit 102.

  The determination circuit 102 performs a matching process between the feature data of the image detected from the feature extraction circuit 101 and the feature data of the image of the upper category condition from the upper category condition input circuit 103, and performs upper processing that satisfies the upper category condition. An image of a category subject candidate is extracted. In this case, images of a plurality of upper category subject candidates may be extracted. Note that the subject candidate extraction unit 10 may be implemented as a hardware IP as a functional circuit block or a software IP deployed and implemented in a predetermined processor.

  Returning to FIG. 1, the display unit 9 displays the current image data, which is an input image from the image sensor 2, via the storage circuit 4, and displays a marker at the position of the extracted upper category subject candidate image. Display is made. For example, a rectangular frame figure is displayed at the face position of the extracted upper category subject candidate in accordance with the face size. Such display control is executed by the control unit 15 based on the output of the determination result from the determination circuit 102.

  FIG. 3 shows a circuit block of the subject candidate selection unit 11. As illustrated, the subject candidate selection unit 11 includes a lower category condition input circuit 112 to which the lower category condition set by the search condition setting unit 6 and its feature data are input. Furthermore, it has an upper category subject candidate input circuit 111 to which the feature data of the upper category subject candidate extracted by the subject candidate extraction unit 10 is input. Further, it comprises a determination circuit 113 that performs a matching process on the feature data of the lower category condition image from the inputted feature data of the upper category subject candidate image to determine the lower category subject candidate image satisfying the lower category condition. The configuration of the subject candidate selection unit 11 is also implemented as a hard IP or a soft IP, similar to the subject candidate extraction unit 10.

  Hereinafter, for example, the operation of the first embodiment will be described assuming that the image of the lower category condition is “capturing a scene where the specific person A laughs” and the image of the upper category condition is “detection of a human face”.

  First, when a plurality of images of upper category subject candidates satisfying the upper category condition are extracted, feature data of images of upper category subject candidates that are partial images of the face area are input to the upper category subject candidate input circuit 111 in the raster scan order. Entered.

  The determination circuit 113 processes the feature data of the partial image (the face partial image in the first embodiment) that is the upper category subject candidate input from the upper category subject candidate input circuit 111. Then, a program to be executed is selectively activated based on the image feature data of the lower category condition input from the search condition setting unit 6 through the lower category condition input circuit 112.

  For example, in the case of an image with a lower category condition of “capturing a scene where a specific person A laughs”, the determination circuit 113 performs personal identification processing and smile detection processing as a facial expression detection program on the feature data of the upper category subject candidate image. Are executed together. In the personal identification processing, face image authentication processing is executed on the feature data of the upper category subject candidate image. That is, by the matching process, it is determined whether or not the feature data of the face portion of the input upper category subject candidate image matches the feature data of the image specifying the specific person A.

  In the smile detection process, image feature data related to “smile” extracted by the feature extraction circuit 101 of the subject candidate extraction unit 10 is input, a predetermined facial expression is detected based on the change pattern, and the lower category subject candidate is detected. Select an image. It goes without saying that well-known methods may be used for facial expression detection algorithms such as “boring”, “excitement”, “confused”, “anger”, “surprise”, etc., as well as “smile”.

  Assume that the output from the subject candidate selection unit 11 determines that an image of a lower category subject candidate that satisfies the lower category condition has been selected. In this case, the imaging condition control circuit 12 performs at least one of focus control, exposure control, and white balance control on the subject area of the selected lower category subject candidate image. In this case, the current imaging condition is set to be suitable for the extraction of the upper category subject candidate image and the selection of the lower category subject candidate image. For this reason, it is not always an imaging condition for obtaining image data suitable for imaging and recording. Therefore, it is necessary to record image data after obtaining an imaging condition for obtaining image data suitable for imaging and recording. is there.

  FIG. 4 is a diagram summarizing the basic processing flow of the imaging apparatus 100 that is the image search apparatus according to the first embodiment. First, in step 401 after the start of the process, the user sets an imaging mode using the imaging mode setting unit 5, and further uses the operation unit 8 to enter the search candidate setting unit 10, subject candidate extraction unit 10 into the search condition setting unit 6. Upper category conditions and lower category conditions used in the selection unit 11 are set.

  In this case, for example, a GUI menu screen of the upper category condition is displayed on the display unit 9. Here, when the user selects a desired upper category condition using the operation unit 8, a GUI menu screen of a lower category condition related to the upper category condition is displayed. Similarly, the image of the lower category condition can be selected by the user. In this case, when the lower category condition is set on the GUI menu screen, the upper category condition can be automatically set. A set of image feature data of the upper category condition and the lower category condition is stored in the search condition data storage unit 7.

  As another configuration, the user does not select and set the upper category condition and the lower category condition, but selects and sets the desired subject condition (corresponding to the lower category condition). Then, it is possible to automatically set the desired subject condition and the feature data set structure of the image satisfying the upper category condition and the lower category condition so that the upper category condition and the lower category condition are set automatically. It is.

  For example, if “capturing a scene where a specific individual A laughs” is set as a subject condition, the upper category condition is set to “face detection”, the lower category condition is set to “scene where a specific individual A laughs”, and the like. Is done. As described above, the data link structure between the upper category condition and the lower category condition condition may be described in a predetermined format and stored in the search condition data storage unit 7 together with the feature data.

  In general, there are many cases where a category condition that describes an object to be imaged can be described using a hierarchical structure of two or more layers. In the example of “capturing a scene where Mr. A of a specific individual laughs”, if “detection and imaging of face” is the highest category condition, “detection of Mr. A” is set as the lower category condition, and lower The category condition can be described in a three-layer structure following “detect smile”. Or, as a two-layer structure, “Mr. A's detection (imaging)” is described in the upper category condition and “Smile detection (imaging)” is described in the lower category condition. Alternatively, a hierarchical structure such as “smile detection (imaging)” as the upper category condition and “detection of Mr. A (imaging)” as the lower category condition may be given. Which can be performed more efficiently depends on the situation at the time of imaging.

  In general, the upper category condition is a condition for limiting the type of a rough subject as a preceding stage of the lower category condition. The lower category conditions further limit the conditions of the subject. In other words, conditions related to appearance, personal name, age, clothes, dogs, cats, and other conditions, or specific movement routes / movement patterns, specific facial expressions / gestures / hand gestures / behaviors / walking patterns, etc. Is included. Alternatively, those related to normality / abnormality may be included.

  Note that the upper category conditions are not limited or fixed to category conditions such as person extraction and face extraction, but the subject category condition corresponding to the final final image data input category condition. It may be determined according to the fineness.

  After roughly narrowing down the upper category and extracting the upper category subject candidates, the lower category subject candidate image that matches the specific condition corresponding to the lower category condition is selected from the images of the upper category subject candidates. . By determining an image under the conditions set in this way, efficient processing, that is, speeding up of processing can be brought about in a variety of subjects and combinations of various states in general.

  The expression of the category condition describing the imaging of the subject may have a hierarchical branch structure as shown in FIG. 5 as the lower category condition belonging to the higher category condition. That is, there is an expression in which a plurality of lower category conditions are arranged immediately below the upper category conditions as described above. In addition, as shown in FIG. 5, the category located in the upper layer of the two or more branch hierarchy structure is a higher category, and different categories at the same hierarchical level are expressed as categories at the same level. May be. In this case, the subcategory includes a supplementary category (for example, an image of a person wearing a bag, an image of Mr. A's family, etc.) with an element that characterizes an upper category (for example, an image of a person) as a representative example. It is done.

  Returning to the flowchart of FIG. 4, the process proceeds to step S <b> 402, and imaging condition control processing based on the imaging mode is performed. Thereafter, the process proceeds to step S403, where the subject candidate extraction unit 10 executes subject candidate extraction processing based on the set upper category condition. Thereby, subject candidate extraction processing is executed on the image data generated based on the output from the image sensor 2. In step S404, subject candidate extraction determination is executed based on the subject candidate detection level at each point of the image output as a result of the subject candidate extraction process executed in the previous step S403. This process is executed by the determination circuit 102 of the subject candidate extraction unit 10.

  In step S404, when an image of the upper category subject candidate (person's face) that satisfies the upper category condition is not detected in the current image data, the process proceeds to step S405, and processing for determining imaging condition control is executed. As a result, when it is determined that the process is finished, the entire process is finished. On the other hand, if it is determined in step S405 that the process has not ended, the process returns to the imaging condition control process in step S402.

  Thereafter, the subject candidate selection unit 11 performs a process for selecting an image of a subject candidate that satisfies the lower category condition set in step S401. Then, the process proceeds to step S407, and the determination circuit 113 executes determination processing for determining subject candidate selection. If it is determined that the matching lower category subject candidate image has been selected as a result of the determination process in step S407, the process proceeds to step S408 in which the imaging condition is set again, and image data suitable for imaging and recording is obtained. An imaging condition control process is performed. This process is executed by the imaging condition control circuit 12 according to the output of the determination circuit 113.

  After the imaging condition is set to a condition that results in image data suitable for recording, the image data corresponding to the frame in which the lower category subject candidate is detected is read from the storage circuit 4 and predetermined by the image data compression / decompression circuit 13. The process including the compression process is performed. Thereafter, the data is recorded on the recording medium by the recording / reproducing circuit 14. On the other hand, if no subject candidate image satisfying the lower category condition is selected from the upper category subject candidate images in step S407, the entire process is terminated.

  Hereinafter, each processing operation will be described in more detail. First, the imaging condition control process in the imaging condition control circuit 12 in step S402 in FIG. 4 will be described with reference to the flowchart of the process shown in FIG. Specifically, in the imaging condition control process in step S402, if the upper category condition is that an image of “face” is detected, first, in step S601, an imaging condition estimation process is performed and estimated. Imaging condition control is executed according to the imaging conditions.

  Specifically, the process proceeds to step S602, and the focus condition, the exposure condition, and the white balance condition are set as the estimation of the imaging condition, and the focus condition set value and the exposure condition set value in the normal mode are set as the initial values of the estimated values. Thereafter, the process proceeds to step S603, and search and control of optimal imaging conditions are performed by scanning search of each condition (stepwise increase or decrease of imaging condition parameters). Note that the optimum imaging condition means that imaging is performed as image data that facilitates extraction of the upper category subject candidates with parameters such as brightness, color tone, and contrast set in advance for the designated subject. The normal mode refers to an imaging mode other than the input mode of subject image data as shown in the first embodiment.

  When it is difficult to detect an image of a human face under backlight illumination, the exposure condition setting value in the normal mode is set as an initial value in the imaging condition control process of step S402 in FIG. In this way, the face detection process is executed each time. As the step width setting method, for example, an LUT is used as a value obtained by averaging pixel values of input image data in the vicinity of the center of the image data or in a local area centered on the exposure correction photometric spot. And stored in a predetermined memory.

  The determination of the backlight illumination condition performed as one of the imaging environment condition estimation processing in step S601 in FIG. 6 may be performed using a known method (for example, a method described in Japanese Patent Application No. 0504939 by the applicant). Even when a face is detected as an upper category subject candidate, the exposure condition and white balance condition at that time are not necessarily optimal. You may determine a stop. Specifically, a memory color of a preferred subject is registered and stored in advance as memory color data in a predetermined memory.

  Further, as the imaging condition change process stop condition in the search control process for the optimum imaging condition in step S603, the memory color is closest to the memory color in the control range such as the exposure condition, or a difference from the memory color is preset. It controls so that it may become below the threshold value. In this control, the imaging condition control circuit 12 controls exposure conditions and white balance conditions.

  Next, the subject candidate extraction process executed in step S403 in FIG. 4 will be described. The determination circuit 102 in the subject candidate extraction unit 10 shown in FIG. 2 includes image feature data output from the feature extraction circuit 101 and feature data of higher category conditions input from the search condition setting unit 6. Matching process. As a result, it is a module for extracting an image of a person's head, face, upper body, or whole body and identifying a user (a person registered in advance). Therefore, it is mounted as a software module or a circuit module.

  The algorithm used in the determination circuit 102 may be a known method such as human head or face extraction in the first embodiment (see, for example, Japanese Patent No. 3078166 by the present applicant). Generally, it is stored in a memory as a feature data file of an extraction software library or an extraction parameter corresponding to the category condition of each subject candidate, and the determination circuit 102 reads the feature data file stored in the memory and performs an algorithm Execute.

  Note that color information and motion vector information can be used as the types of low-order features used for the extraction of the human head image and the facial part feature extraction process. Furthermore, other low-order feature quantities, for example, geometric features having a specific range of directional components and a specific range of spatial frequencies, or local features as disclosed in Japanese Patent No. 3078166 of the present applicant. Needless to say, a characteristic feature may be used.

  Next, the subject candidate selection process in step S406 in FIG. 4 will be described. The subject candidate selection process in step S406 is executed by a CPU, a media processor, a configurable processor, and other various arithmetic circuits. In this process, it is determined whether or not there is a subject candidate that satisfies the lower category condition from the upper category subject candidates obtained by the subject candidate extraction process in step S403. Specifically, the image area is set according to the lower category condition with the candidate point of the upper category subject candidate as the center, the image feature data of the area is input, and the determination circuit 113 performs the determination process.

  An example of the upper category condition and the lower category condition is shown in Table 1 and Table 2 shown in FIGS. Under these conditions, there may be a person who falls into a specific category related to the user's acquaintance (registered person), age and gender, clothes, hairstyle, and the like. Or, the ratio of a person's face is in a specific state (for example, eyes are open, facing the camera, laughing) at a certain ratio or more among a plurality of faces. It is done. The ratio above a certain value is, for example, 8 or more out of 10. However, the ratio above a certain level varies depending on the selected upper category and lower category. These correspond to more detailed conditions as compared with the upper category conditions (for example, person detection imaging). As already described, it is possible to realize high-speed and efficient processing by narrowing down subjects under higher category conditions and determining whether or not there are subjects that satisfy more detailed conditions among the narrowed-down subjects.

  Next, the imaging condition control process shown in step S408 of FIG. 4 will be described using the flowchart shown in FIG.

  If an image of a lower category subject candidate is selected in step S407 in FIG. 4, it is determined that the current image data is image data worth recording. In this case, prior to recording, it is determined in step S801 whether the target area of the image in which the image of the lower category subject candidate of the image data is suitable as the image data to be captured. If it is appropriate, the process proceeds to step S409 in FIG. 4, and the image data obtained under the current imaging conditions is recorded on the recording medium by the recording / reproducing circuit 14 to complete the imaging.

  However, if it is determined to be inappropriate in step S801, the process proceeds to step S802, and the imaging condition of the control target area is reset. The setting of the control target area is changed according to the image position of the selected lower category subject candidate. Thereafter, the process proceeds to step S803, where the imaging condition control circuit 12 controls the imaging condition according to the changed imaging condition and the control target area, and the determination is again made in step S801. This operation is repeated until it becomes appropriate, and if it becomes appropriate, this process is terminated. Then, the process proceeds to step S409 in FIG. 4, and image data obtained under the imaging conditions whose settings have been changed is recorded on the recording / reproducing circuit. In step 14, the image is recorded on a recording medium, and imaging is completed.

  According to the first embodiment described above, it is possible to provide a search device that allows a subject who wishes to capture an image to freely search for image data in a state in which the imaging is desired. Furthermore, an image pickup apparatus that can avoid recording unnecessary image data can be provided by being incorporated in the image pickup apparatus.

<Embodiment 2>
The second embodiment is an image pickup apparatus that is an image search device as in the first embodiment, and the main components are the same as those shown in FIG. 1, and therefore, in the following description, the components refer to FIG. 1. . Here, as in the first embodiment, the case where the subject category conditions are decomposed into various category conditions and combined is handled.

  In particular, the second embodiment is characterized in that category conditions can be selected and combined in advance in a menu selection format so that the details of the conditions can be set easily. FIG. 9 shows an example of the GUI of the search condition setting screen displayed on the display unit 9. As described in the first embodiment, this screen is displayed under the control of the search condition setting unit 6. In addition, as a display device on which the search condition setting screen is displayed, a display device of another predetermined terminal different from the display unit 9 provided in the imaging device 100 according to the second embodiment can be used.

  Here, the hierarchical structure of the category conditions is shown in the figure, and the user directly instructs with a touch panel or the like while viewing the search condition setting screen, or selects with the operation unit 8 including a cross key. By specifying and setting the lowest layer of the hierarchical structure, the image data including the subject to be actually imaged is specified.

  For example, in a state where a category of clothing under the upper category condition of person is selected by the user, an image relating to clothing or a character string expressing the clothing in characters is displayed as an option for the image of the lower category condition. The category hierarchy may be set in advance, but the user can also specify the hierarchy.

  Thus, at least one higher category condition of the image determined according to the subject condition (lower category condition or narrowed category condition) set by the user using the search condition setting screen is automatically generated to extract subject candidates. Processing and selection processing are executed. These processes are executed by the subject candidate extraction unit 10 and the subject candidate selection unit 11 described in the first embodiment.

  When a subject candidate image satisfying the set subject condition is selected, current image data including the subject candidate may be encoded and automatically recorded on the recording medium by the recording / reproducing circuit 14. Specifically, an example of an image in which a plurality of category conditions are combined, such as “image of Mr. A (male) sitting with a muddy expression” will be described below as an example.

  In this case, the image condition relating to the subject identification (being Mr. A) and the image condition relating to the subject state (condition relating to the expression and posture) are combined. In this case, as the upper category condition, a part of elements constituting the composite condition such as detection of a sitting person, detection of Mr. A, detection of a person with a dull expression, or the like is set as an upper category condition of an image. Alternatively, the superordinate concept (for example, human detection imaging and male detection imaging) of the elements constituting the composite condition is set as the upper category condition of the image.

  At this time, the image of the upper category subject candidate corresponding to the image extraction result based on the upper category condition can be displayed with a marker or the like superimposed on the display unit 9. At this time, the user determines whether or not the image of the upper category subject candidate extracted under the upper category condition is appropriate by looking at the image. If it is determined as inappropriate, the user can change the upper category condition on the menu screen. Can be done. Such various operations are appropriately performed by the user using the operation unit 8 in the search condition setting unit 6.

  In the second embodiment described above, when the search condition is set by the search condition setting unit 6 by operating the operation unit 8, the user can easily select images that satisfy the upper category condition and the lower category condition while looking at the menu screen. It becomes possible to set.

<Embodiment 3>
In the first and second embodiments, a predetermined search condition is input from the search condition setting unit 6 using the GUI and the operation unit 8, but the search condition is extracted from newly prepared image data. A configuration is also possible. FIG. 10 shows the third embodiment, and shows only the part corresponding to the search condition setting unit 6 of the first and second embodiments. The other components are basically the same as those in the first and second embodiments, and are not shown in the figure.

  In the figure, reference numeral 61 denotes a search condition image input unit, which includes an image reader. The image read by the search condition image input unit 61 includes a so-called candidate image to be searched and is stored in the storage circuit 62. The image data stored in the storage circuit 62 is extracted by analyzing the feature data of the subject candidate image included in the image analysis unit 63 and stored in the feature data storage circuit 64. The feature data storage circuit 64 outputs upper category conditions, lower category conditions, and feature data of each image. These operations are executed under the control of the control unit 65.

  In the search condition setting unit 60 of FIG. 10, a search condition image input unit 61 inputs image data (such as a typical image of a subject to be searched or an image based on a sketch by a user) as a search condition. Therefore, the image reading apparatus is included. The image data input by the search condition image input unit 61 is stored in the storage circuit 62. In this case, the search condition image input unit 61 does not necessarily need to use the image reading device to capture the image data. If there is already imaged data of the subject to be retrieved, it may be retrieved from an external device. Is possible. Any image data is stored in the storage circuit 62. The image data stored in the storage circuit 62 is supplied to the image analysis unit 63 in order to extract feature data of a subject to be searched.

  The image analysis unit 63 analyzes the feature data of the subject to be searched included in the image data of the subject to be searched, and extracts the upper category condition and the lower category condition. The extracted upper category condition and lower category condition are the character display in the hierarchical category expression as shown in FIG. 9 or the image of the image data of the similar sample on the display unit 9 in FIG. Let the user confirm by displaying them side by side.

  Further, as shown in FIG. 9, the user can further narrow down the upper category conditions and the lower category conditions from the category conditions displayed in a hierarchical structure. The series of processing is executed by the control unit 65 in accordance with an instruction from the control unit 15 in FIG.

  In the third embodiment, as described above, image data of a subject to be searched is input, feature data of the subject to be searched is extracted, and upper category conditions and lower category conditions are specified. It becomes possible. In this case, the upper category condition and the lower category condition of the subject can be specified as a specific hierarchy level in the branch structure (tree structure) of the category expressed hierarchically, Narrowing is also possible. For example, the category “Mr. A who is laughing” has passed through subject candidate extraction processing at each node shown in the hierarchical branch structure as shown in FIG. Is searched.

  As described above, by previously storing or generating a search data structure that hierarchically classifies and expresses search categories, the search process can be narrowed down from the upper category conditions of the hierarchical structure to the lower category conditions. The user can set an image with an appropriate search condition while checking the intermediate result by checking the intermediate result as the search result data. In the case of the third embodiment, in a search condition such as “search for image of Mr. A who is laughing”, “image of Mr. A” is attached to the upper category condition, and “image of a smiling expression” is incidental to Mr. A. Set as an image of the lower category condition. First, Mr. A's image is searched and extracted, and then an image of a smiling expression is searched. Note that the user can set which image is to be the image of the upper category condition.

  FIG. 11 shows a basic flow of image search processing in the image search apparatus according to the third embodiment. First, in step S1101 after the start of processing, search conditions are set by inputting image data. This is performed by the image reading device provided in the search condition image input unit 61 shown in FIG. However, it may be taken in as an image of a search condition from an external memory or the like. In any case, the image includes the subject to be searched.

  Thereafter, the process proceeds to step S1102, where the image analysis unit 63 and the control unit 65 analyze the hierarchical structure of the search category, and extract feature data of the upper category condition and the lower category condition for searching for the subject. In step 1103, the hierarchical structure of the category condition is displayed. The hierarchical structure of categories may be the same as that illustrated in FIG. The user confirms the hierarchical structure of the category conditions displayed on the display unit 9, and if it is determined that correction is necessary in step S1104, the user returns to step S1103 and corrects the search result as necessary.

  Subsequently, in step S1105, extraction of subject candidate images based on the upper category condition is performed from the image data to be captured at present. The extraction result is displayed on the display unit 9 as shown in step S1106. If the result is not satisfied in step S1107, the process returns to step S1103, the analysis result is corrected as necessary, and the search is performed again. If satisfied, the process proceeds to step S1108 to select an image of a subject candidate based on the lower category condition from the image data to be captured at present, and the selection result is displayed in step S1109. That is, the image of the candidate for the lower category subject that satisfies the selected lower category condition in the image data to be captured currently displayed on the display unit 9 is displayed in an identifiable manner. If this result is not satisfied, the process returns from step S1110 to step S1103, and the operation is repeated after correction. However, if satisfied, a series of search processing ends.

  As described above, in the third embodiment, the case where the state of the specific person or animal is detected and the automatic imaging or searching is performed has been described. However, it goes without saying that the target subject may be applied to the detection of a specific article and the automatic input of an image other than a person or animal, for example, a vehicle or other article. .

  According to the third embodiment of the present invention described above, an image input as a search condition, typically an image of a subject candidate that satisfies the search condition such as an image including a subject to be searched or a sketch image by a user. Can be extracted and selected.

  The present invention can also be implemented by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus. That is, the object of the present invention can also be achieved by reading and executing a program code stored in a storage medium by a computer (or CPU or MPU) of the system or apparatus. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS or the like running on the computer performs actual processing based on an instruction of the program code. Some or all may be done. Thus, the present invention includes a case where the functions of the above-described embodiments are realized by the processing.

  Furthermore, in the present invention, the program code read from the storage medium can be written in a memory provided in a function expansion board inserted in the computer or a function expansion unit connected to the computer. Therefore, the function of the above-described embodiment is realized by performing part or all of the actual processing by the CPU provided in the function expansion board or function expansion unit based on the written program code instruction. Is also included.

1 is a block diagram of an imaging device in which an image search device according to Embodiment 1 of the present invention is incorporated. It is a block diagram which shows the structure of the to-be-photographed object candidate extraction part of the image search device which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the to-be-photographed object candidate selection part of the image search device which concerns on Embodiment 1 of this invention. It is a flowchart figure explaining operation | movement of the image search device which concerns on Embodiment 1 of this invention. It is a conceptual diagram which shows the example of the hierarchical structure of the category conditions used with the image search device which concerns on Embodiment 1 of this invention. It is a flowchart figure explaining operation | movement of the image search device which concerns on Embodiment 1 of this invention. It is a figure which shows the table | surface of the category conditions used with the image search device which concerns on Embodiment 2 of this invention. It is a flowchart figure explaining operation | movement of the image search device which concerns on Embodiment 2 of this invention. It is a figure of the screen displayed of the hierarchical structure of the category conditions used with the image search device which concerns on Embodiment 2 of this invention. It is a block diagram which shows the structure of the search condition setting part used with the image search device which concerns on Embodiment 3 of this invention. It is a flowchart figure explaining operation | movement of the image search device which concerns on Embodiment 3 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging optical system 2 Image pick-up element 3 Sensor peripheral circuit 4 Memory circuit 5 Imaging mode setting part 6, 60 Search condition setting part 7 Search condition data storage part 8 Operation part 9 Display part 10 Subject candidate extraction part 11 Subject candidate selection part 12 Imaging condition control circuit 13 Image data compression / decompression circuit 14 Recording / reproducing circuit 15, 65 Control unit 16 Non-volatile memory 61 Search condition image input unit 62 Storage circuit 63 Image analysis unit 64 Feature data storage circuit 101 Feature extraction circuit 102, 113 Determination circuit 103 Upper Category Condition Input Circuit 111 Upper Category Subject Candidate Input Circuit 112 Lower Category Condition Input Circuit

Claims (19)

Image input means for inputting image data;
Search condition setting means for setting a subject search condition including a higher category condition and a lower category condition associated with the higher category condition;
Subject candidate extraction means for extracting upper category subject candidates that satisfy the upper category condition from subjects included in the image data;
A selection means for selecting, as the extracted upper category subject candidate, a lower category subject candidate that satisfies the lower category condition associated with the upper category condition;
An image search apparatus comprising: a display unit configured to display that the candidate data satisfying the search condition is included in the data image when the lower category subject candidate is selected.   The image search apparatus according to claim 1, wherein the upper category condition includes a condition for specifying a person.   The image search apparatus according to claim 1, wherein the search condition setting unit includes a search condition data storage unit that stores the upper category condition and the lower category condition.   The image search apparatus according to claim 2, wherein the lower category condition includes a condition for specifying a state of a person specified by the upper category condition.   The image search apparatus according to claim 2, wherein the lower category condition includes a condition for specifying a face state of a person specified by the upper category condition.   The lower category condition includes a condition in which the lower category subject candidate having the specified face state of the person specified by the upper category condition exists at a certain ratio or more in the upper category subject candidate. The image search apparatus according to claim 5, wherein:   The image search apparatus according to claim 1, wherein the display unit displays a graphic or a marker indicating the lower category subject candidate together with the image data.   The image search apparatus according to claim 1, wherein the display unit further displays the search condition. The search condition setting unit is a search condition image input unit that inputs a search condition image, and an image that analyzes the search condition image and outputs the upper category condition and lower category condition of the subject included in the search condition image An analysis means,
The image search apparatus according to claim 1, wherein the search condition is set based on an output related to the upper category condition and the lower category condition from the image analysis unit. The search condition data storage means stores the search conditions in a hierarchical structure,
The image search apparatus according to claim 1, wherein the display unit displays the search conditions in a hierarchical manner. The upper category condition is a condition for specifying the type of subject,
The image search device according to any one of claims 1 to 10, wherein the lower category condition is a condition for specifying a state of a subject. The upper category condition is a condition for specifying an animal other than a human subject.
The image according to any one of claims 1 to 10, wherein the lower category condition is a condition for at least one of a type of a subject, a state of a subject, and the number of subjects. Search device. In an imaging apparatus comprising image input means for inputting image data, display means, and recording means,
Search condition setting means for setting a subject search condition including a higher category condition and a lower category condition associated with the higher category condition;
Subject candidate extraction means for extracting upper category subject candidates that satisfy the upper category condition from subjects included in the image data;
A selection means for selecting, as the extracted upper category subject candidate, a lower category subject candidate that satisfies the lower category condition associated with the upper category condition;
An image search device comprising: display means for displaying that the image data includes a candidate subject that satisfies the search condition when the lower category subject candidate is selected. Imaging device.   The imaging condition control means for controlling any one of an exposure control mechanism and a focus detection control mechanism included in the image input means according to a result of extraction of the upper category subject candidate. The imaging device according to 13. Furthermore, a recording means is provided,
15. The imaging apparatus according to claim 14, wherein the recording unit records the image data in which the lower category subject candidate is selected on a recording medium.   The imaging apparatus according to claim 14, wherein the imaging condition control unit controls the exposure control mechanism or the focus detection control mechanism according to a result of selection of the lower category subject candidate. An image input process for inputting image data;
A search condition setting step for setting a subject search condition including a higher category condition and a lower category condition associated with the higher category condition;
A subject candidate extraction step of extracting a higher category subject candidate that satisfies the higher category condition from subjects included in the image data;
A selection step of selecting a lower category subject candidate that satisfies the lower category condition associated with the upper category condition as the extracted upper category subject candidate;
And a display step of displaying that the image data includes a candidate candidate that satisfies the search condition when the lower category subject candidate is selected.   A non-transitory computer-readable storage medium storing a program for causing a computer to execute the procedure of the control method for an image search apparatus according to claim 17.   A computer-readable storage medium storing a program for causing a computer to execute the procedure of the control method for an image search device according to claim 17.

Images