JP2018018339A - Determination apparatus, determination method, and determination program - Google Patents

Abstract

An image ranking is appropriately determined. A search device according to the present application includes an acquisition unit and a determination unit. The acquisition unit acquires a query in image search. The determination unit determines the ranking of each image based on information regarding the size of the target corresponding to the query in each image. For example, the determination unit determines the ranking of each image according to information regarding the object occupancy ratio in each image based on information regarding the size of the object in each image and information regarding the size of each image. [Selection] Figure 3

Description

  The present invention relates to a determination device, a determination method, and a determination program.

  Conventionally, various techniques relating to image retrieval have been provided. For example, a technique related to image search in consideration of the contents and layout of an object is provided. Conventionally, a technique related to image feature extraction using a neural network has been provided. For example, a technique for identifying an object included in an image is provided by a convolutional neural network.

JP 2011-197892 A

  However, with the above-described conventional technology, it is difficult to appropriately determine the ranking of images to be provided as a search result for a query. For example, just based on the contents of an object (target) included in an image, the ranking of the image is not always determined appropriately.

  The present application has been made in view of the above, and an object thereof is to provide a determination device, a determination method, and a determination program that appropriately determine the ranking of images in a search result for a query.

  The determination device according to the present application includes an acquisition unit that acquires a query in an image search, and a determination unit that determines a ranking of each image based on information on a size of a target corresponding to the query in each image. It is characterized by that.

  According to the aspect of the embodiment, there is an effect that the ranking of the image in the search result for the query can be appropriately determined.

FIG. 1 is a diagram illustrating an example of a search process according to the embodiment. FIG. 2 is a diagram illustrating an example of a search process according to the embodiment. FIG. 3 is a diagram illustrating a configuration example of the search device according to the embodiment. FIG. 4 is a diagram illustrating an example of a learning information storage unit according to the embodiment. FIG. 5 is a diagram illustrating an example of the image information storage unit according to the embodiment. FIG. 6 is a diagram illustrating an example of the list information storage unit according to the embodiment. FIG. 7 is a diagram illustrating an example of the occupation rate information storage unit according to the embodiment. FIG. 8 is a flowchart illustrating an example of the estimation of the occupation ratio of the object in the image according to the embodiment. FIG. 9 is a flowchart illustrating an example of ranking determination according to the embodiment. FIG. 10 is a diagram illustrating an example of learning processing according to the embodiment. FIG. 11 is a diagram illustrating an example of learning processing according to the embodiment. FIG. 12 is a flowchart illustrating an example of a learning process according to the embodiment. FIG. 13 is a diagram illustrating an example of ranking determination according to the embodiment. FIG. 14 is a hardware configuration diagram illustrating an example of a computer that implements the function of the search device.

  Hereinafter, a determination apparatus, a determination method, and a mode for executing a determination program according to the present application (hereinafter referred to as “embodiment”) will be described in detail with reference to the drawings. Note that the determination device, the determination method, and the determination program according to the present application are not limited to the embodiment. In the following embodiments, the same portions are denoted by the same reference numerals, and redundant description is omitted.

(Embodiment)
[1. Search process)
An example of search processing according to the embodiment will be described with reference to FIGS. 1 and 2. 1 and 2 are diagrams illustrating an example of search processing according to the embodiment. Specifically, FIG. 1 is a diagram illustrating an example of estimation of the occupancy rate of a target in each of image groups to be searched, which will be described later. 1 uses a learning device (model) that outputs (estimates) an occupancy rate of an object in an image, and estimates the occupancy rate of the object in each image group to be searched.

  Specifically, FIG. 2 is a diagram illustrating an example of providing search results obtained by ranking (ranking) images including objects corresponding to a query according to an occupation ratio when a query is acquired. Further, the learning device LE that outputs (estimates) the occupancy ratio of the object in the image is generated by using the input image and correct information indicating the occupancy ratio of the object in the input image, which will be described in detail later. Moreover, in the example shown in FIG.1 and FIG.2, although the object which estimates an occupation rate shows the case of a cat, a target is not only a cat but other living things, such as a dog and a human, a plant, a car, etc. Various objects may be used. In addition, the object referred to here may include various objects as long as they can be identified, and may include various phenomena such as fire and sea waves. In FIG. 2, for ease of explanation, the search apparatus 100 determines the ranking (rank) of images including the target “cat” using only the click rate by the user for the image including the target “cat” as an index. Show the case. Specifically, in FIG. 2, the search device 100 divides the cat occupancy rate in the image into a plurality of sections, and uses the average click rate by the user for the image corresponding to the occupancy ratio of each section as an index. The ranking of images including “cat” is determined.

  Here, the learning device used by the search device 100 will be briefly described. The learning device used by the search apparatus 100 is, for example, a learning device in which a plurality of nodes that output calculation results for input data are connected in multiple layers, and has learned the features of an image abstracted by supervised learning. It is a learning device. For example, the learning device is a neural network in which layers having a plurality of nodes are connected in multiple stages, and may be a DNN (Deep Neural Network) realized by a so-called deep learning technique. Image features include not only the specific features that appear in the image, such as the presence / absence of characters in the image, color, composition, etc., but also what the object is being imaged and what kind of user the image is It is a concept that also includes the characteristics of an abstracted (meta-) image such as the atmosphere of the image.

  For example, the learning device is generated by the following learning method using a deep learning technique. For example, in the learning device, the connection coefficient between the nodes is initialized, and images having various characteristics are input. The learning device is generated by a process such as back propagation (error back propagation method) for correcting a parameter (connection coefficient) so that an error between the output from the learning device and the input image is reduced. For example, the learning device is generated by performing a process such as back propagation so as to minimize a predetermined loss function such as an error function. By repeating the processing as described above, the learning device can output an output that can better reproduce the input image, that is, a feature of the input image.

  Note that the learning method of the learning device is not limited to the method described above, and any known technique can be applied. Information used when learning by the learning device may use a data set of various images such as an image and a target included in the image. Information used when learning by the learning device includes an image including one target and a set of information indicating that there is one target, and a plurality of images and targets including a plurality of (for example, two) targets. A set of information indicating (for example, two), an image that does not include a target, a set of information that indicates that a target is not included (zero), and the like may be used. In addition, any method can be applied to the image input method to the learning device, the format of data output from the learning device, the content of features that are explicitly learned by the learning device, and the like. In other words, the search device 100 can use any learning device as long as it can calculate a feature value indicating a feature abstracted from an image.

  In FIG. 1, the search device 100 uses a learning device LE based on a so-called convolutional neural network (CNN) that repeats convolution and pooling of a local region of an input image. Hereinafter, the convolutional neural network may be referred to as CNN. For example, in addition to the function of extracting and outputting features from an image, the learning device LE by CNN has invariance of output with respect to positional variations such as characters and imaging objects included in the image. For this reason, the learning device LE can calculate the abstract feature of the image with high accuracy.

  For example, in FIG. 1, the search device 100 uses a learning device LE that is a discriminator (model) that outputs information related to the occupancy rate of a target (cat) in an image. That is, in FIG. 1, it is assumed that the search device 100 uses the learning device LE that has been generated by the predetermined learning process as described above. Note that FIG. 1 shows a case where the search device 100 uses a learning device LE that identifies a target included in an image. However, the search device 100 can perform any learning as long as the occupancy of a predetermined target can be estimated. A vessel may be used. Further, when generating (learning) the learning device LE, a predetermined loss function, correct answer information, and the like are used, but details will be described later.

  As shown in FIG. 2, the search system 1 includes a terminal device 10 and a search device 100. The terminal device 10 and the search device 100 are communicably connected via a predetermined communication network (not shown) in a wired or wireless manner. In addition, the search system 1 illustrated in FIG. 2 may include a plurality of terminal devices 10 and a plurality of search devices 100.

  The terminal device 10 is an information processing device used by a user. The terminal device 10 is realized by, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like. In the example shown in FIG. 2, the case where the terminal device 10 is a smartphone is shown. Hereinafter, the terminal device 10 may be referred to as a user. That is, hereinafter, the user can be read as the terminal device 10.

  The search device 100 is an information processing device that estimates an occupancy ratio of a predetermined object in an acquired image using a learning device LE that outputs the occupancy ratio of the predetermined object in an input image. The search device 100 is a determination device that determines the ranking of each image based on information about the size of the target corresponding to the query in each image.

  First, the estimation of the occupancy rate of the target (cat) in each image to be searched by the search device 100 will be described with reference to FIG. For example, the images IM11 to IM15 shown in FIG. 1 are stored in the image information storage unit 122 (FIG. 5). Hereinafter, when the images IM11 to IM15 and the like are described without being distinguished, they may be described as “image IM”.

  For example, the search device 100 inputs the image IM to the learning device LE to estimate the occupation ratio of the cat in the image IM. Specifically, the learning device LE to which the image IM is input outputs information indicating the occupancy ratio of the cat in the input image IM, and the search device 100 occupies the cat in the image IM output from the learning device LE. Based on the information indicating the rate, the occupation rate of the cat in the image IM is estimated. The learning device LE to which the image IM is input outputs information indicating the occupation rates of various objects in the input image IM as well as the cat. In FIG. 1 and FIG. 2, only the occupancy ratios of the two classes of cat and background are shown for simplicity of explanation, but the learning device LE may output information indicating the occupancy ratios of other classes. Good. The class here is a class in machine learning, and may be a type classified by a learning device. For example, the learning device LE may output information indicating occupancy rates for 20 classes. In this case, the learning device LE may output information indicating the occupancy rate for classes such as dogs, airplanes, and bicycles in addition to the two classes corresponding to the cat and the background. It should be noted that although the learning device LE outputs information indicating the occupancy rate of which object (class) depends on learning, details will be described later.

  Further, the search device 100 may estimate the occupancy rate of cats in the image IM by inputting all the images IM to be searched to the learning device LE, but it is assumed that the target cat is included. Only the image IM may be input to the learning device LE, and information indicating the occupancy rate of the cat in the image IM may be output. For example, the search device 100 inputs only the image IM associated with the tag “cat” among the images IM in the image information storage unit 122 (FIG. 5) to the learning device LE, and occupies the cat in the image IM. Information indicating the rate may be output. For example, when the tag “cat” is associated with the images IM11 to IM15 and the like illustrated in FIG. 1, the search device 100 inputs the images IM11 to IM15 and the like to the learning device LE, and occupies the cats in the image IM. May be output.

  In the example of FIG. 1, the search device 100 inputs the image IM11 to the learning device LE (step S11-1). For example, the search device 100 acquires the image IM11 from the image information storage unit 122 (FIG. 5), and inputs the image IM11 to the learning device LE. Thereafter, the search device 100 estimates the occupancy rate of the cat in the image IM11 based on the output of the learning device LE (step S12-1). For example, as shown in the estimation information OC11, the search device 100 estimates the occupancy rate of the cat in the image IM11 as 80% (0.8) based on the output of the learning device LE, and occupies the background rate in the image IM11. Is estimated to be 18% (0.18). For example, the search device 100 estimates the occupancy rate of each target so that the total occupancy rate of each target (class) is 100%. For example, the search device 100 estimates that the total occupancy rate of each target (class) other than a cat or background is 2%. In the present embodiment, the occupation ratio is expressed as a percentage “%” (for example, 80%), but may be a decimal point (for example, 0.8). For example, the learning device LE may output a numerical value within a range of 0 to 1 indicating the occupancy rate of each target (class).

  In the example of FIG. 1, the search device 100 inputs the image IM12 to the learning device LE (step S11-2). For example, the search device 100 acquires the image IM12 from the image information storage unit 122 (FIG. 5), and inputs the image IM12 to the learning device LE. Thereafter, the search device 100 estimates the occupancy rate of the cat in the image IM12 based on the output of the learning device LE (step S12-2). For example, as shown in the estimation information OC12, the search device 100 estimates the occupancy rate of the cat in the image IM12 as 60% and estimates the occupancy rate of the background in the image IM12 as 36% based on the output of the learning device LE. To do.

  In the example of FIG. 1, the search device 100 inputs the image IM13 to the learning device LE (step S11-3). For example, the search device 100 acquires the image IM13 from the image information storage unit 122 (FIG. 5), and inputs the image IM13 to the learning device LE. Thereafter, the search device 100 estimates the occupancy rate of the cat in the image IM13 based on the output of the learning device LE (step S12-3). For example, as shown in the estimation information OC13, the search device 100 estimates the occupancy rate of the cat in the image IM13 as 20% and estimates the occupancy rate of the background in the image IM13 as 75% based on the output of the learning device LE. To do.

  In the example of FIG. 1, the search device 100 inputs the image IM14 to the learning device LE (step S11-4). For example, the search device 100 acquires the image IM14 from the image information storage unit 122 (FIG. 5), and inputs the image IM14 to the learning device LE. Thereafter, the search device 100 estimates the occupancy rate of the cat in the image IM14 based on the output of the learning device LE (step S12-4). For example, as shown in the estimation information OC14, the search device 100 estimates the occupancy rate of the cat in the image IM14 as 90% and estimates the occupancy rate of the background in the image IM14 as 9% based on the output of the learning device LE. To do.

  In the example of FIG. 1, the search device 100 inputs the image IM15 to the learning device LE (step S11-5). For example, the search device 100 acquires the image IM15 from the image information storage unit 122 (FIG. 5), and inputs the image IM15 to the learning device LE. Thereafter, the search device 100 estimates the occupancy rate of the cat in the image IM15 based on the output of the learning device LE (step S12-5). For example, as shown in the estimation information OC15, the search device 100 estimates the occupancy rate of the cat in the image IM15 as 70% and estimates the occupancy rate of the background in the image IM15 as 27% based on the output of the learning device LE. To do.

  The search device 100 also inputs other images IM associated with the tag “cat” to the learning device LE, and estimates the occupation ratio of the cats in each image IM. Thereafter, the search device 100 generates list information indicating the occupancy rate of cats in each estimated image IM (step S13). In the example of FIG. 1, the search device 100 generates list information LT11 indicating the occupation ratio of cats in the images IM11 to IM15 and the like. In the example of FIG. 1, the case where the search device 100 generates the list information LT11 is described as an example for the sake of explanation, but the occupancy rate of cats in each image IM estimated in steps S12-1 to S12-5 and the like. May be stored in the image information storage unit 122 (FIG. 5) in association with each image.

  Next, providing search results by the search device 100 when a query is acquired will be described with reference to FIG. First, the search device 100 acquires a query from the terminal device 10 used by the user U1 (step S14). In the example of FIG. 2, the search device 100 acquires a character string “cat” as a query. Note that the search device 100 may acquire a query image instead of a character string. When an image is acquired as a query, the search device 100 inputs the acquired image to the learning device LE. Then, the search device 100 may estimate the target included in the image as a cat based on the output of the learning device LE. The query image here means an image indicating a condition specified in the search, and may include a processing request (inquiry) for extracting an image satisfying a predetermined condition with the query image.

  Thereafter, the search device 100 extracts an image corresponding to the query “cat”. For example, the search device 100 extracts the image IM with the tag “cat” from the image information storage unit 122 (see FIG. 5). Thereafter, the search device 100 acquires occupancy information corresponding to the query. For example, the search device 100 acquires occupancy rate information corresponding to the query “cat” from the occupancy rate information storage unit 124. The occupancy rate information here indicates information related to the occupancy rate of the object in the image when the image provided to the user as a target search result is selected by the user.

  For example, as shown in the occupancy rate information storage unit 124 in FIG. 2, an image including the target “cat” has a click rate rank of “5” when the occupancy rate of the cat in the image is 0-20%. Yes, indicates the lowest click rate. For example, 0-20% indicates a range greater than 0 and 20 or less. An image including the target “cat” indicates that the click rate ranking is “3” when the occupancy rate of the cat in the image is 20-40%. For example, 20-40% indicates a range greater than 20 and 40 or less.

  An image including the target “cat” indicates that the click rate rank is “2” when the occupation ratio of the cat in the image is 40-60%. For example, 40-60% indicates a range greater than 40 and 60 or less. An image including the target “cat” has a click rate rank of “1” when the cat occupation ratio in the image is 60-80%, indicating that the click rate is the highest. For example, 60-80% indicates a range greater than 60 and 80 or less. An image including the target “cat” indicates that the click rate rank is “4” when the occupancy ratio of the cat in the image is 80-100%. For example, 80-100% indicates a range greater than 80 and 100 or less.

  Thus, in the example of FIG. 2, the image including the target “cat” indicates that the probability of the user clicking is the highest when the occupation ratio of the cat in the image is 60-80%. That is, in the example of FIG. 2, when the click rate by the user is used as an index, when the target is “cat”, the optimal occupation rate is within a range of 60-80%.

  Thereafter, the search device 100 determines the ranking of the image IM (step S15). For example, as shown in the ranking information LT12, the search device 100 determines the ranking so that the ranking of the image IM included in the range of “60-80”%, which is the range of the occupation rate with the highest click rate ranking, is high. To do. For example, the search device 100 places the image IM15 having an occupancy of the median “70%” of 60-80% as the first place, places the image IM19 closest to “70%” next to the image IM15 as the second place, and sets the image IM19. Next, the image IM17 closest to “70%” is ranked third, and the image IM11 closest to “70%” next to the image IM17 is ranked fourth.

  Thereafter, the search device 100 provides a search result based on the determined ranking to the user U1 (step S16). In the example of FIG. 2, the search device 100 displays search results in display order based on the ranking in which the image IM15 is ranked first, the image IM19 is ranked second, the image IM17 is ranked third, and the image IM11 is ranked fourth. Provided to the terminal device 10 used by U1. For example, the search device 100 transmits a search result based on the determined ranking to the terminal device 10 used by the user U1.

  As described above, the search device 100 appropriately estimates the manner in which the target is included in the image, that is, the proportion of the target in the image, by using the learning device that outputs the occupation ratio of the target in the image. be able to. In addition, when the search device 100 acquires a query, the search device 100 determines the ranking of the image based on information related to a user operation on an image including an object corresponding to the query. Thereby, when the search device 100 acquires a query, the search device 100 can appropriately determine the ranking of the image including the target corresponding to the query. As illustrated in FIG. 2, the search device 100 determines the ranking of the image according to the click rate ranking for the image including the target corresponding to the query, thereby obtaining a search result that more appropriately reflects the user's intention. Can be provided. Note that the search device 100 may determine the ranking of images using various information as appropriate without being limited to the information related to the occupation ratio. For example, the search device 100 may determine the ranking so that the ranking of an image in which the target is clearly reflected is high. For example, the search device 100 may determine the ranking in consideration of information related to the image quality (number of pixels) of the image.

  In the above example, the search device 100 determines the ranking of the image according to the click rate of the user with respect to the image including the query “cat”, but the search device 100 uses various information as appropriate. You may decide the ranking. The search apparatus 100 may determine the ranking of the image based on other various features and respective weights, with the occupation ratio of the object as one feature. For example, the search device 100 may determine the ranking of images in consideration of the click rate of each image, the click rate for each user, and the like. Moreover, although the case where the search device 100 determines the ranking based on the click rate for each target has been described in the above example, the search device 100 may determine the ranking based on the click rate for each category. For example, when determining the ranking of the image including the target “cat”, the search device 100 may determine the ranking based on, for example, the click rate of the entire target belonging to the category “animal”. For example, when determining the ranking of an image including the target “cat”, the search device 100 determines the ranking based on the click rate of the entire target belonging to the category “animal” including targets other than cats such as dogs and horses. May be. Accordingly, the search device 100 can appropriately determine the ranking even when there is little information regarding the user operation on the cat image.

  Further, for example, the search device 100 determines the ranking of the image using the average of the occupancy ratios of the objects included in the image selected by the user as search results corresponding to the query “cat” in the past. May be. For example, the search device 100 provides the user with a search result corresponding to the query “cat” in the past, and ranks the images based on the distribution of the occupancy rates of a plurality of objects included in each image clicked by the user. You may decide. For example, the search device 100 is provided to the user as a search result corresponding to the query “cat” in the past, and the average distribution of the occupancy ratios of a plurality of objects clicked by the user (hereinafter referred to as “optimal distribution”). ) May be determined as the optimum value of the image corresponding to the query “cat”. Further, for example, the search device 100 may generate a model that estimates the optimum value of the occupancy rate of each object in the image by learning. In this case, the search device 100 may determine the ranking of the image based on the optimum value of the occupation rate of the target estimated by the model. Further, the search device 100 may determine the ranking so that images corresponding to the occupation ratios of the click rate ranks “1” to “5” are repeatedly arranged. For example, the search apparatus 100 sets the ranking of images corresponding to the occupation ratio of 60-80% to the first and sixth ranks, and ranks the images corresponding to the occupation ratio of 40-60% to the second and seventh ranks. The ranking of images corresponding to the rate 20-40% is 3rd or 8th, etc. The ranking of images corresponding to the occupation rate 80-100% is 4th or 9th, etc., corresponding to the occupation rate 0-20% The ranking of the image may be 5th or 10th. In this case, the search device 100 can determine the ranking so that the occupancy ratio of images having consecutive ranks varies. Thereby, the search apparatus 100 can determine the ranking of images so that images with various occupation ratios are arranged.

  For example, the search device 100 can determine the KL divergence between the distribution of the occupancy ratio of a plurality of targets included in each image corresponding to the query “cat” extracted from the image information storage unit 122 (see FIG. 5), and the optimal distribution. Image ranking may be determined based on the index value. For example, the search device 100 may determine a higher image ranking as the KL divergence value between the distribution of the occupancy ratio of the plurality of objects included in each image corresponding to the query “cat” and the optimal distribution is smaller. Good. Note that the search device 100 is not limited to KL divergence, and the index ranking may be determined by appropriately using various index values as long as the index value is based on the similarity of distribution in each image.

  Moreover, although the case where the search device 100 uses one learning device LE is shown in the above-described example, the search device 100 may use a learning device for each target. For example, the search device 100 may use a learning device for a dog separately from a learning device for a cat. Further, in the above-described example, the case where the information regarding the target occupancy is used for determining the ranking of the image in the image search with respect to the input of the query is shown. However, any information regarding the target occupancy can be used. Information regarding the occupation rate of the target may be used in the service. For example, the search device 100 may use information regarding the occupancy rate of the object in determining the display order (ranking) of products in electronic commerce or the like. For example, the search device 100 may use information regarding the target occupancy rate in determining the display order (ranking) of the search results of the products on a shopping site or the like. For example, the search device 100 may determine the ranking so that the ranking of an image in which a target product is large, that is, an image with a large occupancy rate of the product is high. For example, the search device 100 may determine the ranking so that the ranking of images in which other objects other than the target product are not reflected is high.

[2. (Configuration of search device)
Next, the configuration of the search device 100 according to the embodiment will be described with reference to FIG. FIG. 3 is a diagram illustrating a configuration example of the search device 100 according to the embodiment. As illustrated in FIG. 3, the search device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The search device 100 includes an input unit (for example, a keyboard and a mouse) that receives various operations from an administrator of the search device 100 and a display unit (for example, a liquid crystal display) for displaying various types of information. May be.

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card). The communication unit 110 is connected to the network by wire or wirelessly, and transmits and receives information to and from the terminal device 10 included in the search system 1, for example.

(Storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. As illustrated in FIG. 3, the storage unit 120 according to the embodiment includes a learning information storage unit 121, an image information storage unit 122, a list information storage unit 123, and an occupation rate information storage unit 124.

(Learning information storage unit 121)
The learning information storage unit 121 according to the embodiment stores various types of information related to learning. For example, in FIG. 4, the learning information storage unit 121 stores learning information (model) related to the learning device LE generated by a predetermined learning process. FIG. 4 shows an example of the learning information storage unit 121 according to the embodiment. The learning information storage unit 121 illustrated in FIG. 4 stores “weight (w _ij )”.

For example, in the example illustrated in FIG. 4, “weight (w ₁₁ )” is “0.2”, and “weight (w ₁₂ )” is “−0.3”. In the example illustrated in FIG. 4, “weight (w ₂₁ )” is “0.5”, and “weight (w ₂₂ )” is “1.3”.

The “weight (w _ij )” may be, for example, a synapse coupling coefficient from the neuron y _i to the neuron x _j in the learning device LE. The learning information storage unit 121 is not limited to the above, and may store various types of information according to the purpose. Moreover, the search apparatus 100 may memorize | store the information regarding a some learning device, when using the learning device properly for every object. For example, when using a learning device for a dog separately from a learning device for a cat, the search device 100 may also store information related to the learning device for the dog.

(Image information storage unit 122)
The image information storage unit 122 according to the embodiment stores various types of information related to images. FIG. 5 shows an example of the image information storage unit 122 according to the embodiment. For example, the image information storage unit 122 stores an image to be searched. The image information storage unit 122 illustrated in FIG. 5 includes items such as “image ID”, “image”, and “tag”.

  “Image ID” indicates identification information for identifying an image. “Image” indicates image information. Specifically, “image” indicates an image to be searched. In FIG. 5, an image identified by the image ID is illustrated for explanation, but a file path name indicating the storage location of the image may be stored as “image”. “Tag” is information attached in association with an image, and indicates a target included in the image.

  For example, in the example illustrated in FIG. 5, the image identified by the image ID “IM11” is the image IM11 including a cat, and indicates that the tag “cat” is associated and stored. Note that a tag attached to each image may be attached by an administrator who registers the image, or may be automatically attached by identifying a target using various image recognition techniques as appropriate. .

  The image information storage unit 122 is not limited to the above, and may store various types of information according to the purpose. For example, the image information storage unit 122 may store information related to the date and time when the image is generated. Further, for example, the image information storage unit 122 may store information related to a target included in the image. For example, the image information storage unit 122 may store the acquired original image.

(List information storage unit 123)
The list information storage unit 123 according to the embodiment stores various types of information related to the occupancy ratio of objects in the image. For example, the list information storage unit 123 stores various types of information regarding the occupancy rate of the object in the image for each object. FIG. 6 is a diagram illustrating an example of the list information storage unit according to the embodiment. In the example illustrated in FIG. 6, the list information storage unit 123 stores information (table) for each target that is included, such as the list information LT11 and the list information LT21. For example, the list information LT11 and the list information LT21 have items such as “image ID” and “occupancy (%)”.

  “Image ID” indicates identification information for identifying an image. “Image” indicates image information. The “occupation rate (%)” of the list information LT11 includes items such as “cat” and “background”. Further, “occupation rate (%)” of the list information LT21 includes items such as “dog” and “background”.

  For example, as shown in the list information LT11 in FIG. 6, the image (IM11) identified by the image ID “IM11” has the occupation rate of the target “cat” is “80”% and the occupation of the target “background” The rate is “18”%. For example, as shown in the list information LT11 in FIG. 6, the image (IM12) identified by the image ID “IM12” has an occupancy rate of “60”% for the target “cat” and the target “background”. Indicates that the occupation ratio is “36”%.

  The list information storage unit 123 is not limited to the above, and may store various types of information according to the purpose. For example, the list information storage unit 123 may store information regarding the position in the target image. For example, the list information storage unit 123 may store various pieces of information regarding which part of the image is located in the center, top, bottom, right, left, whole, or the like. For example, the list information storage unit 123 may store an image corresponding to each image ID, a file path name indicating the storage location of the image, and the like. In the example of FIG. 6, the list information storage unit 123 shows a case where the table is divided and stored for each target such as the list information LT11 and the list information LT21. You may memorize | store in one table. For example, the list information storage unit 123 may store all items in “occupation rate (%)” in one table.

(Occupancy rate information storage unit 124)
The occupancy rate information storage unit 124 according to the embodiment stores various types of information regarding the occupancy rate of the target in the image. For example, the occupancy rate information storage unit 124 stores various types of information related to the optimal value of the occupancy rate for each target. FIG. 7 is a diagram illustrating an example of the occupation rate information storage unit according to the embodiment. In the example illustrated in FIG. 7, the occupation rate information storage unit 124 includes items such as “target”, “occupation rate (%)”, and “click rate ranking”.

  “Target” indicates information for identifying a target. In “target”, an ID for identifying the target may be stored. “Occupancy rate (%)” indicates the range of the occupation rate. “Click rate ranking” indicates the ranking of the click rate corresponding to each range of the occupation rate.

  For example, in the example shown in FIG. 7, an image including the target “cat” has a click rate rank of “5” when the occupancy rate of the cat in the image is “0-20” (%), and is most clicked. The rate is low. An image including the target “cat” indicates that the click rate ranking is “3” when the occupation ratio of the cat in the image is “20-40” (%). An image including the target “cat” indicates that the click rate rank is “2” when the occupation ratio of the cat in the image is “40-60” (%). An image including the target “cat” has a click rate rank of “1” when the cat occupation ratio in the image is “60-80” (%), indicating that the click rate is the highest. An image including the target “cat” indicates that the click rate ranking is “4” when the occupation ratio of the cat in the image is “80-100” (%).

  Note that the occupation rate information storage unit 124 is not limited to the above, and may store various types of information according to the purpose. For example, the occupation rate information storage unit 124 may store information in which the image clicked by the user is associated with the occupation rate of the target in the image.

(Control unit 130)
Returning to the description of FIG. 3, the control unit 130 is a controller, and is stored in a storage device inside the search device 100 by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. Various programs (corresponding to an example of a determination program) are implemented by using the RAM as a work area. The control unit 130 is a controller, and is realized by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

  As shown in FIG. 3, the control unit 130 includes an acquisition unit 131, a learning unit 132, an estimation unit 133, an extraction unit 134, a determination unit 135, and a provision unit 136. Information described below Implement or execute a processing function or action.

(Acquisition part 131)
For example, the acquisition unit 131 acquires various types of information. For example, the acquisition unit 131 acquires an image. For example, the acquisition unit 131 acquires an image from the image information storage unit 122. In FIG. 1, the acquisition unit 131 acquires the images IM11 to IM15 and the like from the image information storage unit 122. Note that the acquisition unit 131 may acquire the images IM11 to IM15 and the like from an external information processing apparatus.

  The acquisition unit 131 acquires a query for image search. In FIG. 2, the acquisition unit 131 acquires a query from the terminal device 10 used by the user U1. In FIG. 2, the acquisition unit 131 acquires a character string “cat” as a query. For example, the acquisition unit 131 may acquire a query image instead of a character string.

(Learning unit 132)
The learning unit 132 learns various information. The learning unit 132 generates various information by learning. For example, the learning unit 132 learns a learning device (model). In other words, the learning unit 132 generates a learning device (model) by performing learning. For example, the learning unit 132 learns the learning device LE. For example, the learning unit 132 learns a learning device based on a combination of an image and an occupation ratio of a predetermined target in the image. The learning unit 132 learns the learning device so as to minimize a predetermined evaluation function. Details of the learning process performed by the learning unit 132 will be described later.

(Estimation unit 133)
The estimation unit 133 estimates various information. For example, the estimation unit 133 is acquired by the acquisition unit 131 based on a learning device that outputs an occupation ratio of a predetermined target in the input image according to the input image and an image acquired by the acquisition unit 131. Estimate the occupancy of a given object in the image. For example, the estimation unit 133 estimates the occupancy rate of a predetermined target in the image acquired by the acquisition unit 131 based on a learning device that is a neural network learned from predetermined data (teacher data). For example, the estimation unit 133 occupies the predetermined object in the image acquired by the acquisition unit 131 based on a learning device that is a neural network learned by a combination of the image and the occupation ratio of the predetermined object in the image. Estimate the rate. For example, the estimation unit 133 estimates an occupancy ratio of a predetermined target in the image acquired by the acquisition unit 131 based on a learning device that is a neural network that performs convolution processing and pooling processing.

  In FIG. 1, the estimation unit 133 estimates the occupancy rate of the cat in the image IM by inputting the image IM to the learning device LE. Specifically, the learning device LE to which the image IM is input outputs information indicating the occupancy rate of the cat in the input image IM, and the estimation unit 133 occupies the cat in the image IM output from the learning device LE. Based on the information indicating the rate, the occupation rate of the cat in the image IM is estimated. For example, the estimation unit 133 estimates the occupancy rate of the cat in the image IM11 based on the output of the learning device LE. In FIG. 1, the estimation unit 133 estimates the occupancy rate of the cat in the image IM11 as 80% based on the output of the learning device LE as shown in the estimation information OC11, and the occupancy rate of the background in the image IM11 as 18%. Estimated. Moreover, in FIG. 1, the estimation part 133 estimates the occupancy rate of the cat in image IM12-IM15 etc. based on the output of the learning device LE. For example, the estimation unit 133 estimates the occupancy of the cat in each image IM by inputting the image IM associated with the tag “cat” to the learning device LE.

  Note that when the search device 100 acquires information on the occupation ratio stored in the list information storage unit 123 from an external information processing device, the search device 100 may not include the learning unit 132 or the estimation unit 133. . That is, the search device 100 that provides a search result for a query and the information processing device that performs a learning process or the like may be separate.

(Extractor 134)
The extraction unit 134 extracts various information. For example, the extraction unit 134 extracts information about an image from the image information storage unit 122 or the list information storage unit 123. For example, the extraction unit 134 extracts an image that satisfies a condition from a predetermined image group stored in the image information storage unit 122 or the list information storage unit 123. In FIG. 2, the extraction unit 134 extracts an image corresponding to the query “cat”.

(Determining unit 135)
The determination unit 135 determines various information. For example, the determination unit 135 determines the ranking of each image based on information regarding the size of the target corresponding to the query in each image. For example, the determination unit 135 determines the ranking of each image according to information regarding the object occupancy ratio in each image based on information regarding the size of the object in each image and information regarding the size of each image. For example, the determination unit 135 determines the ranking of each image based on information regarding the occupation ratio that varies depending on the target. For example, the determination unit 135 determines the ranking of the plurality of images based on the occupancy rate of the target corresponding to the query in each of the plurality of images estimated by the estimation unit 133.

  For example, the determination unit 135 determines the ranking of each image based on information regarding the occupation ratio that varies depending on the target. For example, the determination unit 135 is an image provided to the user as a target search result, and is based on information about the occupancy rate determined according to the occupancy rate of the object in the image selected by the user. Determine the ranking.

  In FIG. 2, the determination unit 135 determines the ranking of the image IM. For example, the determination unit 135 determines the ranking so that the ranking of the image IM included in the range of “60-80”%, which is the occupancy range with the highest click rate ranking, is high. For example, the determination unit 135 sets the image IM15 having the median value “70%” in the range of “60-80”% as the first position and the image IM19 closest to “70%” next to the image IM15 as the second position. The image IM17 closest to “70%” next to the image IM19 is ranked third, and the image IM11 closest to “70%” next to the image IM17 is ranked fourth.

  Further, for example, the determination unit 135 determines the ranking of the image using the average of the occupancy ratios of the objects included in the image selected by the user as search results corresponding to the query “cat” in the past. May be. For example, the determination unit 135 provides the ranking of the images based on the distribution of the occupancy ratios of a plurality of objects provided to the user as search results corresponding to the query “cat” in the past and included in each image clicked by the user. You may decide. For example, the determination unit 135 provides an optimal distribution, which is an average distribution of the occupancy ratios of a plurality of objects of all images clicked by the user as search results corresponding to the query “cat” in the past. Image ranking may be determined as the optimum value of the image corresponding to “cat”.

  For example, the determination unit 135 is based on an index value such as KL divergence between the distribution of the occupation ratio of a plurality of targets included in each image corresponding to the query “cat” extracted from the image information storage unit 122 and the optimal distribution. Image ranking may be determined. For example, the determination unit 135 may determine the higher the ranking of the image as the KL divergence value between the distribution of the occupation ratio of the plurality of objects included in each image corresponding to the query “cat” and the optimal distribution is smaller. Good. Note that the determination unit 135 is not limited to KL divergence, and may use various index values to determine the ranking of images as long as the index value is based on the similarity of distribution in each image.

  For example, the determination unit 135 may determine the ranking of each image according to the difference in the occupation ratio of each image in each ranking of the image. For example, the determination unit 135 may determine the ranking of each image so that the difference in the occupancy rate of each successive image in each ranking of the image satisfies a predetermined condition. For example, the determination unit 135 may determine the ranking of the images so that the difference in the occupancy rate of the object in the image IM between successive rankings is 5% or more. For example, when the occupancy rate of the cat in the image IM ranked first is “70”%, the deciding unit 135 determines that the occupancy rate of the cat in the image IM ranked second is greater than “75”%, The image IM with the second ranking may be determined so as to be less than “65”%.

  For example, the determination unit 135 may determine the ranking of each image depending on whether or not each image includes a target other than the search target. For example, when the search target is “cat” and the image includes an object other than a cat (for example, “dog”, etc.), the determination unit 135 causes each of the images to have a low ranking. A ranking may be determined. For example, the determination unit 135 may determine the ranking of each image according to whether or not character information is included in each image. For example, the determination unit 135 may determine the ranking of each image so that the ranking of the image in which character information is included in each image is low. In this case, for example, the determining unit 135 may determine whether character information is included in each image by appropriately using various conventional techniques for recognizing characters.

  For example, the determination unit 135 may determine the ranking of each image according to the position of the search target in each image. For example, the determination unit 135 determines the ranking of each image so that the ranking of the image in which the target is located on the right is high when the click rate is high when the position of the target corresponding to the query is on the right. Also good. In this case, for example, the determination unit 135 may determine the position of the search target in each image based on information indicating the position of the target in each image acquired by the acquisition unit 131. For example, the determination unit 135 may determine the position of the search target in each image based on information indicating the position of the target in each image output from the learning device that holds the position information. Note that the above is an example, and the determination unit 135 may use any information as long as it can determine the position of the search target in each image.

(Providing unit 136)
The providing unit 136 provides various types of information to an external information processing apparatus. For example, the providing unit 136 provides information based on the ranking of a plurality of images determined by the determining unit 135. In FIG. 2, the providing unit 136 provides the search result based on the determined ranking to the user U1. For example, the providing unit 136 uses the search result in the display order based on the ranking in which the image IM15 is ranked first, the image IM19 is ranked second, the image IM17 is ranked third, and the image IM11 is ranked fourth. Provided to the terminal device 10. For example, the providing unit 136 transmits a search result based on the determined ranking to the terminal device 10.

[3. Occupancy rate estimation process flow)
Here, the procedure of the occupation rate estimation process performed by the search device 100 according to the embodiment will be described with reference to FIG. FIG. 8 is a flowchart illustrating an example of the estimation of the occupation ratio of the object in the image according to the embodiment.

  As shown in FIG. 8, the search device 100 acquires an image (step S101). In FIG. 1, the search device 100 acquires images IM11 to IM15 and the like. Thereafter, the search device 100 inputs the image acquired in step S101 to the learning device (step S102). In FIG. 1, the search device 100 inputs the acquired images IM11 to IM15 and the like to the learning device LE.

  Thereafter, the search device 100 estimates the occupation ratio of the target in the input image based on the output of the learning device (step S103). In FIG. 1, the search device 100 estimates the occupancy rate of cats in the input images IM11 to IM15 and the like based on the output of the learning device LE.

  Thereafter, the search device 100 generates image list information including the occupation ratio (step S104). In FIG. 1, the search device 100 generates list information LT11 indicating the occupancy rate of cats in the images IM11 to IM15 and the like.

[4. (Ranking decision process flow)
Next, the procedure of the ranking determination process performed by the search device 100 according to the embodiment will be described with reference to FIG. FIG. 9 is a flowchart illustrating an example of ranking determination according to the embodiment.

  As shown in FIG. 9, the search device 100 acquires a query (step S201). In FIG. 2, the search device 100 acquires a character string “cat” as a query. In step S201, the search device 100 may acquire a query image instead of a character string.

  Thereafter, the search device 100 extracts an image corresponding to the query (step S202). For example, the search device 100 extracts the image IM with the tag “cat” from the image information storage unit 122.

  Thereafter, the search device 100 acquires occupancy information corresponding to the query (step S203). In FIG. 2, the search device 100 acquires occupancy rate information corresponding to the query “cat” from the occupancy rate information storage unit 124.

  Thereafter, the search device 100 determines the ranking based on the image corresponding to the query and the occupation rate information corresponding to the query (step S204). In the example of FIG. 2, the search device 100 determines the ranking so that the ranking of the image IM included in the range of “60-80”% that is the range of the occupation rate with the highest click rate ranking is high.

  Thereafter, the search device 100 provides a search result based on the determined ranking (step S205). In the example of FIG. 2, the search device 100 displays search results in display order based on the ranking in which the image IM15 is ranked first, the image IM19 is ranked second, the image IM17 is ranked third, and the image IM11 is ranked fourth. Provided to the terminal device 10 used by U1.

[5. Learning process)
Here, the learning process in the learning unit 132 of the search device 100 will be described with reference to FIGS. 10 and 11. 10 and 11 are diagrams illustrating an example of the learning process according to the embodiment.

  First, it demonstrates using FIG. In the example illustrated in FIG. 10, the search device 100 teaches a combination of an image IM21 including a cat and information RO21 indicating the occupation ratio of a cat or the like in the image (hereinafter, sometimes described as “correct answer information RO21”). Obtained as data (step S21). In FIG. 10, only the occupancy ratios of the two classes of cat and background are illustrated in the correct answer information RO21 for the sake of simplicity, but the correct answer information RO21 corresponds to the class output by the learning device LE. Occupancy rate may be included. For example, when the learning device LE outputs information indicating occupancy rates for 20 classes, the correct answer information RO21 includes information indicating occupancy rates of other classes in addition to the occupancy rates of two classes of cat and background. But you can. In a case where the learning device LE outputs information indicating an occupation rate regarding a class such as a dog, an airplane, or a bicycle in addition to two classes corresponding to a cat and a background, the correct answer information RO21 includes an occupation rate “0” of the dog, Information indicating the plane occupancy “0” or the bicycle occupancy “0” may be included.

  Then, an image IM21 including a cat is input to the learning device LE (step S22). Thereafter, information indicating the occupation rate of the target as indicated by the output information OC21-1 is output from the learning device LE (step S23). In FIG. 10, only the occupancy ratios of the two classes of cat and background are shown for simplicity of explanation, but the learning device LE may output information indicating the occupancy ratios of other classes. For example, when the learning device LE outputs information indicating the occupancy rates for 20 classes, the learning device LE occupies the occupancy rates for classes such as dogs, airplanes, and bicycles in addition to two classes corresponding to cats and backgrounds. May be output.

  As described above, for example, the learning unit 132 learns and generates the learning device LE using a deep learning technique. For example, the learning unit 132 uses a combination of an image and an occupation ratio of a predetermined target in the image as teacher data. For example, the learning unit 132 performs backpropagation (error back propagation method) for correcting a parameter (connection coefficient) so that an error between an output from the learning device LE and an occupation ratio of a predetermined target included in the teacher data is reduced. The learning device LE is learned by performing the above processes. For example, the learning unit 132 generates the learning device LE by performing processing such as back propagation so as to minimize a predetermined error (loss) function.

  For example, the learning unit 132 uses an error function L as shown in the following equation (1). As shown in the following equation (1), the learning unit 132 uses cross entropy as an error function in the case of an N-class classification problem, for example. The error function L may be any function as long as it represents the certainty of the identification result. For example, the error function L may be entropy obtained from the identification probability. For example, the error function L may be any function as long as it indicates the recognition accuracy of the learning device LE.

  Here, “x” in the above formula (1) and the following formulas (2) to (4) indicates an image. For example, in the example shown in FIG. 10, “x” in the above formula (1) and the following formulas (2) to (4) corresponds to the image IM. Also, 1 to N assigned to the variable “n” correspond to each class identified (classified) by the learning device LE. For example, the learning device LE corresponding to the above equation (1) indicates that N classes are identified. For example, “Cat” and “Background” correspond to each class.

In addition, “t _n (x)” in the above formula (1) and the following formulas (3) and (4) represents the occupancy rate of the object corresponding to the class n (any one of 1 to N) in the image IM21. Show. For example, “t _n (x)” in the above formula (1) indicates the occupation rate of the object corresponding to the class n as shown in the correct answer information RO21. In this case, for example, when the target corresponding to class 1 is “cat”, “t ₁ (x)” is “0.53 (53%)”. Further, for example, “t _n (x)” in the above formula (1) may be defined to take 1 only in the correct class and to take 0 otherwise.

Further, “p _n (x)” in the above formula (1) and the following formulas (2) and (3) is a learning device for the object corresponding to the class n (any one of 1 to N) in the image IM21. The occupation rate based on the output of LE is shown. For example, “p _n (x)” in the above formula (1) indicates the occupation rate of the object corresponding to the class n output by the learning device LE as shown in the output information OC21-1. In this case, for example, when the target corresponding to class 1 is “cat”, “p ₁ (x)” is “0.64 (64%)”.

Further, “p _n (x)” in the above equation (1) is defined by a Softmax function as shown in the following equation (2) with the probability of class n with respect to x.

The function “f _n ” in the above equation (2) is a score of class n output from the CNN (learning device LE). “Θ” is a parameter of CNN (learning device LE). The function “exp” is an exponential function. In this case, the gradient of the error function L (1) shown in the above equation (1) is calculated by the following equation (3).

As shown in the above equation (3), in all classes 1 to N, when p _n (x) = t _n (x), the gradient of the error function L (x) becomes 0 and becomes an extreme value. . For example, the learning unit 132 performs feedback processing so that the gradient of the error function L (x) becomes 0 (step S24). For example, when the learning unit 132 repeats the processing as described above, the learning device LE can appropriately output the occupation ratio of the target in the input image. FIG. 10 is for visually showing that the processing is repeated so as to minimize the error function L and the like in order to make the output of the learning device LE close to the correct answer information RO21. It may be done automatically.

As described above, if “t _n (x)” is defined as the occupation rate of class n when the entire image is 1, for example, it is expressed by the following equation (4).

“Δ _{i, j} ” is the Kronecker delta and “j _p ” is the label for pixel p. For example, “j _p ” is a label corresponding to one of a plurality of classes such as a cat and a background. For example, the label for each pixel indicates which object the pixel is for. If the label of a pixel is a label corresponding to a cat, this indicates that the pixel is related to the object “cat”. For example, the denominator of the above equation (4) is the total value of all the pixels of the image “x”. Further, for example, the numerator of the above formula (4) is the number of pixels labeled with the class n in the image “x”. Thus, “t _n (x)” in the above formula (4) is a value between 0 and 1. The occupancy rate of the object corresponding to each class in the image is calculated by the above equation (4). Note that the labeling of each pixel may be performed by, for example, the administrator of the search device 100 or the owner of the image.

  Next, a description will be given with reference to FIG. In FIG. 11, the search device 100 illustrates a case where learning is performed by a learning device LE31 that is different from the learning device LE. In the example illustrated in FIG. 11, the search apparatus 100 includes an image IM31 including a person and a bottle, and information RO31 (hereinafter, sometimes referred to as “correct answer information RO31”) indicating the occupation ratio of the person and the bottle in the image. Are obtained as teacher data (step S31). Then, the image IM31 including a person and a bottle is input to the learning device LE31 (step S32). Thereafter, information indicating the occupation rate of the target as indicated by the output information OC31-1 is output from the learning device LE31 (step S33). Also in the example shown in FIG. 11, as in the example shown in FIG. 10, the learning unit 132 uses the above equations (1) to (3) so that the gradient of the error function L (x) becomes zero. A feedback process is performed (step S34). For example, when the learning unit 132 repeats the above-described processing, the learning device LE31 can appropriately output the occupation ratio of the object in the input image even when there are a plurality of objects. FIG. 11 is for visually showing that the processing is repeated so as to minimize the error function L and the like in order to bring the output of the learning device LE31 closer to the correct answer information RO31. In the learning device LE31, FIG. It may be done automatically.

[6. Learning process flow)
Here, the procedure of the learning process performed by the search device 100 according to the embodiment will be described with reference to FIG. FIG. 12 is a flowchart illustrating an example of a learning process according to the embodiment.

  As illustrated in FIG. 12, the search device 100 acquires an image and correct information regarding the occupancy rate of each target in the image (step S301). In FIG. 10, the search device 100 acquires an image IM <b> 21 including a cat and correct answer information RO <b> 21 indicating the occupation ratio of a cat or the like in the image. Thereafter, the search device 100 inputs the image acquired in step S301 to the learning device (step S302). In FIG. 10, the search device 100 inputs the acquired image IM21 to the learning device LE.

  After that, the search device 100 learns so that an error between the occupation rate based on the output of the learning device and the occupation rate of the correct answer information becomes small (step S303). In FIG. 10, the search device 100 learns based on the occupancy rate of each target indicated in the output information OC21-1 based on the output of the learning device LE and the occupancy rate of each target indicated in the correct answer information RO21.

  Thereafter, when the predetermined condition is satisfied (step S304: Yes), the search device 100 ends the process. For example, when the error between the occupation rate based on the output of the learning device and the occupation rate of the correct answer information is within a predetermined threshold or when a predetermined time has elapsed after starting learning, the search device 100 If the condition is satisfied, the process may be terminated. In addition, when the predetermined condition is not satisfied (step S304: No), the search device 100 repeats the process of step S303. For example, the search device 100 is predetermined when the error between the occupation rate based on the output of the learning device and the occupation rate of the correct information is larger than a predetermined threshold or when a predetermined time has not elapsed since the start of learning. The process of step S303 may be repeated assuming that the above condition is not satisfied. Note that the learning process described above is merely an example, and the search device 100 may perform learning according to various procedures.

[7. (Ranking)
As described above, the search device 100 may determine the ranking of images using various information as appropriate. For example, the search device 100 may determine the ranking of each image according to the diversity of the occupation ratio between the images in the ranking of each image. With respect to this point, determination of ranking by the search device 100 according to the embodiment will be described with reference to FIG. FIG. 13 is a diagram illustrating an example of ranking determination according to the embodiment.

  In the example illustrated in FIG. 11, as in the example illustrated in FIG. 2, the ranking of images is determined based on the occupation rate information storage unit 124 and the list information LT <b> 11 indicating the occupation rate of cats. In the example illustrated in FIG. 11, the search device 100 determines the ranking of images so that the occupancy ratio of consecutive images varies in the ranking of images (Step S <b> 41). For example, as shown in the ranking information LT41, the search device 100 determines the ranking so that the images corresponding to the occupancy rates with the click rate ranks “1” to “5” are repeatedly arranged in the order of 1st to 5th. To do.

  In the example of FIG. 11, the search device 100 determines the ranking of the image IM15 corresponding to the occupation rate 60-80% with the click rate rank “1” as the first rank. For example, the search device 100 determines the ranking of the image IM15 in which the occupancy rate of the cat is 70% within the range of the occupancy rate of 60-80% as the first ranking. In the example of FIG. 11, the search device 100 determines the ranking of the image IM41 corresponding to the occupation rate 40-60% with the click rate ranking “2” as the second ranking. For example, the search device 100 determines the ranking of the image IM41 whose cat occupancy rate is 51% within the range of the occupancy rate 40-60% as the second ranking.

  In the example of FIG. 11, the search device 100 determines that the ranking of the image IM45 corresponding to the occupation rate of 20-40% with the click rate ranking “3” is third. For example, the search device 100 determines the ranking of the image IM45 in which the cat occupation ratio is 28% within the range of the occupation ratio of 20-40% as the third place. In the example of FIG. 11, the search device 100 determines the ranking of the image IM14 corresponding to the occupation rate 80-100% with the click rate ranking “4” as the second ranking. For example, the search device 100 determines the ranking of the image IM14 in which the cat occupancy rate is 90% within the range of the occupancy rate 80-100% as the fourth place. In the example of FIG. 11, the search device 100 determines the ranking of the image IM49 corresponding to the occupation rate 0-20% with the click rate rank “5” as the fifth rank. For example, the search device 100 determines the ranking of the image IM49 in which the cat occupation ratio is 11% within the range of the occupation ratio 0-20% as the fifth place. Then, the search device 100 repeatedly determines the click rate order from the sixth place in the ranking in order starting from “1”.

  In the example of FIG. 11, the search device 100 determines that the ranking of the image IM19 corresponding to the occupation rate 60-80% with the click rate ranking “1” is sixth. For example, the search device 100 determines the ranking of the image IM19 in which the occupancy rate of the cat is 65% within the range of the occupancy rate of 60-80% as the 6th place. In the example of FIG. 11, the search device 100 determines the ranking of the image IM47 corresponding to the occupation rate of 40-60% with the click rate ranking “2” as the seventh ranking. For example, the search device 100 determines the ranking of the image IM47 whose cat occupancy rate is 55% within the range of the occupancy rate 40-60% as the seventh place. Although not shown in the example of FIG. 11, the search device 100 ranks the image corresponding to the occupancy rate 20-40% as the eighth rank and ranks the image corresponding to the occupancy ratio 80-100% as the ninth rank. And the ranking of the image corresponding to the occupation rate of 0-20% is 10th.

  As described above, the search device 100 determines the ranking so that images corresponding to the occupancy rates with the click rate ranks “1” to “5” are repeatedly arranged, thereby occupying consecutive images in the image ranking. The ranking of images can be determined so that the rate varies. Note that the above-described determination of the ranking is an example, and the search apparatus 100 determines the ranking of the image using any information as long as the ranking of the image can be determined so that images with various occupation ratios are arranged. Also good.

[8. effect〕
As described above, the search device 100 according to the embodiment includes the acquisition unit 131 and the determination unit 135. The acquisition unit 131 acquires a query for image search. The determination unit 135
The ranking of each image is determined based on the information regarding the size of the target corresponding to the query in each image ("cat" in the embodiment, the same applies hereinafter).

  Thereby, the search device 100 according to the embodiment can appropriately determine the ranking of the image in the search result for the query based on the information regarding the size of the target corresponding to the query in each image.

  Further, in the search device 100 according to the embodiment, the determination unit 135 determines each image according to information regarding the occupancy rate of each object based on information regarding the size of the object in each image and information regarding the size of each image. Determine the ranking.

  Thereby, the search device 100 according to the embodiment uses the information about the occupancy rate of each object based on the information about the size of the object in each image and the information about the size of each image, so that The ranking of the image can be determined appropriately.

  Further, in the search device 100 according to the embodiment, the determination unit 135 determines the ranking of the image based on information on the occupation ratio that varies depending on the target.

  Thereby, the search device 100 according to the embodiment can appropriately determine the ranking of the image in the search result with respect to the query, based on the information on the occupation ratio that varies depending on the target.

  In the search device 100 according to the embodiment, the determination unit 135 is an image provided to the user as a target search result, and is determined according to the target occupancy in the image selected by the user. The ranking of each image is determined based on the information regarding.

  Thereby, the search device 100 according to the embodiment is an image provided to the user as a target search result, and is based on information on the occupancy rate determined according to the occupancy rate of the target in the image selected by the user. Thus, it is possible to appropriately determine the ranking of images in the search result for the query.

  Also, in the search device 100 according to the embodiment, the determination unit 135 determines the ranking of each image according to the diversity of the occupation ratio between the images in the ranking of each image.

  Thereby, the search device 100 according to the embodiment appropriately determines the ranking of the image in the search result for the query by determining the ranking of the image based on the difference in the occupation ratio of each image in the ranking of each image. be able to.

  Further, in the search device 100 according to the embodiment, the determination unit 135 determines the ranking of each image so that the difference between the occupancy rates of consecutive images in the ranking of each image satisfies a predetermined condition.

  Thereby, the search device 100 according to the embodiment determines the ranking of each image so that the difference in the occupancy rate of each successive image in the ranking of each image satisfies a predetermined condition. The ranking of the image can be determined appropriately.

  Further, in the search device 100 according to the embodiment, the determination unit 135 determines the ranking of each image according to whether or not each image includes a target other than the target.

  Thereby, the search device 100 according to the embodiment appropriately determines the ranking of the images in the search result for the query by determining the ranking of each image according to whether each image includes a target other than the search target. Can be determined.

  In the search device 100 according to the embodiment, the determination unit 135 determines the ranking of each image according to whether or not each image includes character information.

  Thereby, the search device 100 according to the embodiment appropriately determines the ranking of the images in the search result for the query by determining the ranking of each image according to whether or not the character information is included in each image. be able to.

  In addition, the search device 100 according to the embodiment includes a providing unit 136. The providing unit 136 provides information based on the ranking of each image determined by the determining unit 135.

  Thereby, the search device 100 according to the embodiment can provide the user with more appropriate information by providing information based on the ranking of the image determined using the information regarding the target size.

[9. Hardware configuration)
The search device 100 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in FIG. 14, for example. FIG. 14 is a hardware configuration diagram illustrating an example of a computer that implements the function of the search device. The computer 1000 includes a CPU 1100, RAM 1200, ROM 1300, HDD (Hard Disk Drive) 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

  The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each unit. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, a program depending on the hardware of the computer 1000, and the like.

  The HDD 1400 stores programs executed by the CPU 1100, data used by the programs, and the like. The communication interface 1500 receives data from other devices via the network N and sends the data to the CPU 1100, and provides the data generated by the CPU 1100 to other devices via the network N.

  The CPU 1100 controls an output device such as a display and a printer and an input device such as a keyboard and a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. In addition, the CPU 1100 outputs the generated data to the output device via the input / output interface 1600.

  The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads such a program from the recording medium 1800 onto the RAM 1200 via the media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. Etc.

  For example, when the computer 1000 functions as the search device 100 according to the embodiment, the CPU 1100 of the computer 1000 implements the function of the control unit 130 by executing a program loaded on the RAM 1200. The CPU 1100 of the computer 1000 reads these programs from the recording medium 1800 and executes them. However, as another example, these programs may be acquired from other devices via the network N.

  Although the embodiments of the present application have been described in detail with reference to the drawings, these are merely examples, and various modifications and improvements have been made based on the knowledge of those skilled in the art including the aspects described in the disclosure line of the invention. The present invention can be implemented in other forms.

[10. Others]
In addition, among the processes described in the above embodiment, all or part of the processes described as being automatically performed can be performed manually, or the processes described as being performed manually can be performed. All or a part can be automatically performed by a known method. In addition, the processing procedures, specific names, and information including various data and parameters shown in the document and drawings can be arbitrarily changed unless otherwise specified. For example, the various types of information illustrated in each drawing is not limited to the illustrated information.

  Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured.

  Further, the above-described embodiments can be appropriately combined within a range in which processing contents are not contradictory.

  In addition, the “section (module, unit)” described above can be read as “means” or “circuit”. For example, the acquisition unit can be read as acquisition means or an acquisition circuit.

1 Search System 100 Search Device (Determination Device)
REFERENCE SIGNS LIST 121 Learning information storage unit 122 Image information storage unit 123 List information storage unit 124 Occupancy rate information storage unit 130 Control unit 131 Acquisition unit 132 Learning unit 133 Estimation unit 134 Extraction unit 135 Determination unit 136 Providing unit 10 Terminal device N network

Claims (11)

An acquisition unit for acquiring a query in image search;
A determination unit that determines a ranking of each image based on information on a size of a target corresponding to the query in each image;
A determination device comprising: The determination unit
The ranking of each image is determined according to information on the occupancy rate of the target in each image based on information on the size of the target in each image and information on the size of each image. The determination device according to claim 1. The determination unit
The determination apparatus according to claim 2, wherein the ranking of each image is determined based on information regarding an occupation ratio that varies depending on the target. The determination unit
The image provided to the user as a search result of the target, and the ranking of each image is determined based on information on the occupancy determined according to the occupancy of the target in the image selected by the user The determination device according to claim 3, wherein: The determination unit
5. The determination device according to claim 2, wherein the ranking of each image is determined in accordance with the diversity of the occupation ratio between the images in the ranking of the images. The determination unit
The determination apparatus according to claim 5, wherein the ranking of each image is determined so that a difference in occupancy between the images that are consecutive in the ranking of the images satisfies a predetermined condition. The determination unit
The determination apparatus according to claim 1, wherein the ranking of each image is determined according to whether or not each image includes a target other than the target. The determination unit
The determination apparatus according to claim 1, wherein ranking of each image is determined according to whether or not character information is included in each image. A providing unit for providing information based on the ranking of each image determined by the determining unit;
The decision device according to claim 1, further comprising: A decision method performed by a computer,
An acquisition step of acquiring a query in image search;
A determination step of determining a ranking of each image based on information about a size of a target corresponding to the query in each image;
A determination method characterized by comprising: An acquisition procedure for acquiring a query in image search;
A determination procedure for determining a ranking of each image based on information about a size of an object corresponding to the query in each image;
A determination program characterized by causing a computer to execute.

Images