JP2017016539A - Commodity shelf recognition device, commodity shelf recognition method, program and image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of more preferably collecting pieces of commodity information of commodities which are sold.SOLUTION: A commodity shelf recognition device comprises: first recognition means for recognizing a commodity based on an image, then recognizing a price of the commodity based on a peripheral area of the commodity image area; and second recognition means for based on feature information of the peripheral area of the image area of the first commodity whose price is recognized by the first recognition means, and feature information of the peripheral area of an image area of a second commodity included in the image, recognizing a price of the second commodity.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a product shelf recognition device, a product shelf recognition method, a program, and an image processing device.

  As a market trend survey, in order to grasp the sales status of products in stores, vending machines, etc., activities are being carried out to collect information on the products sold at those stores. However, since there are many stores and vending machines, there is a problem that when product information is collected manually, enormous work time and labor costs are required. Therefore, a method for automatically collecting such product information from an image taken by a camera is required.

  Conventionally, for example, OCR (Optical Character Reader) for recognizing characters in an image is widely known.

  Patent Document 1 uses a SIFT (Scale Invariant Feature Transform) feature that robustly identifies a subject in an image (for example, a product in a store) against a change in shooting size, angle, and shielding. An image recognition device is described.

  By using the OCR and the technology described in Patent Document 1 together, it is possible to automatically recognize a product and a price associated therewith from an image of a store shelf or a vending machine.

US Pat. No. 6,711,293 JP 2013-167773 A JP 2014-044480 A

  However, if some of the characters that make up the price are blocked by the product or the shadow of the product, the region corresponding to one character is cut out accidentally from the captured image, or the characters in each cut out region are It may not be recognized correctly. Therefore, the character recognition result of OCR is not always reliable.

  In order to solve the above problems, a product shelf recognition apparatus according to an aspect of the present invention includes a first recognition unit that recognizes a product from an image and recognizes the price of the product from a peripheral region of the image area of the product. , Based on the feature information of the peripheral region of the image region of the first product whose price is recognized by the first recognition means and the feature information of the peripheral region of the image region of the second product included in the image, Second recognition means for recognizing the price of the second product.

  The product shelf recognition method according to an aspect of the present invention recognizes a product from an image, recognizes the price of the product from a peripheral region of the image area of the product, and detects the price of the first product for which the price is recognized. The price of the second product is recognized based on the feature information of the peripheral region of the image region and the feature information of the peripheral region of the image region of the second product included in the image.

  In addition, the image processing apparatus according to an aspect of the present invention recognizes a product from an image, and recognizes product information related to the product from a peripheral region of the image area of the product, Based on the feature information of the peripheral region of the image region of the first product for which the product information has been recognized by the recognition means and the feature information of the peripheral region of the image region of the second product included in the image. Second recognition means for recognizing product information related to the product.

  Note that a computer program that realizes each of the above-described apparatuses or methods by a computer, and a computer-readable storage medium that stores the computer program are also included in the scope of the present invention.

  According to the present invention, even if a part of product information such as a part of characters constituting a price is shielded by a product or a shadow of the product, it can be accurately recognized.

It is a functional block diagram which shows an example of a function structure of the goods shelf recognition apparatus which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating an example of the picked-up image which the goods shelf recognition apparatus which concerns on the 1st Embodiment of this invention receives. It is a functional block diagram which shows an example of a function structure of the 1st recognition part 101 of the club location recognition apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of a goods recognition result, a price search image area | region, and a character recognition result. It is a functional block diagram which shows an example of a function structure of the effectiveness determination part 103 of the goods shelf recognition apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows an example of the flow of operation | movement of the goods shelf recognition apparatus which concerns on the 1st Embodiment of this invention. It is a functional block diagram which shows an example of a function structure of the goods shelf recognition apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows an example of the flow of operation | movement of the goods shelf recognition apparatus which concerns on the 2nd Embodiment of this invention. It is a functional block diagram which shows an example of a function structure of the goods shelf recognition apparatus which concerns on the 3rd Embodiment of this invention. It is a figure which shows an example of operation | movement of the pattern generation part 301 of the goods shelf recognition apparatus which concerns on the 3rd Embodiment of this invention. It is a flowchart which shows an example of the flow of operation | movement of the goods shelf recognition apparatus which concerns on the 3rd Embodiment of this invention. It is a functional block diagram which shows an example of a function structure of the goods shelf recognition apparatus which concerns on the 4th Embodiment of this invention. It is a flowchart which shows an example of the flow of operation | movement of the goods shelf recognition apparatus which concerns on the 4th Embodiment of this invention. It is a functional block diagram which shows an example of a function structure of the goods shelf recognition apparatus which concerns on the 5th Embodiment of this invention. It is a figure which shows an example of operation | movement of the pattern generation part 501 of the goods shelf recognition apparatus which concerns on the 5th Embodiment of this invention. It is a flowchart which shows an example of the flow of operation | movement of the goods shelf recognition apparatus which concerns on the 5th Embodiment of this invention. It is a functional block diagram which shows an example of a function structure of the goods shelf recognition apparatus which concerns on the 6th Embodiment of this invention. It is a flowchart which shows an example of the flow of operation | movement of the goods shelf recognition apparatus which concerns on the 6th Embodiment of this invention. It is a functional block diagram which shows an example of a function structure of the goods shelf recognition apparatus which concerns on the 7th Embodiment of this invention. It is a functional block diagram which shows an example of a function structure of the image processing apparatus which concerns on the 8th Embodiment of this invention. It is a figure which illustrates illustartively the hardware constitutions of the computer (information processing apparatus) which can implement | achieve each embodiment of this invention. It is the figure which showed typically an example of operation | movement of the goods shelf recognition apparatus which concerns on the 1st Embodiment of this invention.

  First, in order to facilitate understanding of the embodiments of the present invention, the background of the present invention will be described.

  When collecting product information by automatically recognizing products and their associated prices from images taken from store shelves and vending machines, some of the characters that make up the prices are shadows of the products and products. In situations such as being shielded, OCR character recognition does not always recognize the price accurately.

  On the other hand, by using the SIFT feature amount described in Patent Document 1, it is possible to reduce a reduction in recognition accuracy even if a part of characters constituting the price is blocked by a product or a shadow of the product. However, since there are many variations of the fonts of characters that make up the prices of products sold at stores and vending machines and the spacing between those characters, it is impractical to prepare all variations in advance. . In the following description, “character” may mean one character or a character string.

  According to the embodiment of the present invention described below, the above-described problems are solved, and the price can be recognized with high accuracy.

  In addition, techniques for narrowing down a plurality of recognized product candidates with high accuracy by matching (searching) two different viewpoints are described in Patent Document 2 and Patent Document 3, but the techniques described in these documents are described above. The problem of shielding is not solved.

<First Embodiment>
A first embodiment of the present invention will be described in detail with reference to the drawings. In the present embodiment, information for explaining a product (article) being sold, which is associated with the product, is referred to as product information. The merchandise information includes, for example, identification information for identifying a merchandise such as a merchandise name and accompanying information associated with the merchandise. The accompanying information of the product includes, for example, information indicating the price of the product. The accompanying information of the product may include, for example, information indicating a situation where the product is sold (sales status). The information indicating the sale status of a product is information indicating whether the product is a cold product or a warm product, for example. The accompanying information of the product is not limited to this, and may include information indicating whether the product is on sale or sold out. The information indicating the sales status of the product may be a character string, a color, or a shape.

  In addition, the product shelf in the present embodiment may be a store fixture in which a product is displayed or a vending machine. The same applies to the second and subsequent embodiments.

(Functional configuration)
A product shelf recognition apparatus 10 according to the present embodiment will be described with reference to FIG. FIG. 1 is a functional block diagram illustrating an example of a functional configuration of the commodity shelf recognition apparatus 10 according to the present embodiment. As shown in FIG. 1, the product shelf recognition apparatus 10 includes a first recognition unit 101, a first storage unit 102, an effectiveness determination unit 103, a second storage unit 104, and a second recognition unit 105. And. The product shelf recognition apparatus 10 shown in FIG. 1 is a configuration unique to the present invention, and the product shelf recognition apparatus 10 shown in FIG. 1 may have a member that is not shown in FIG. Needless to say.

  The first recognizing unit 101 receives an image obtained by photographing a store shelf or a vending machine, which is a target for product recognition, and uses the feature information stored in the first storage unit 102 to capture the image. It recognizes the characters that make up the product and the price that accompanies it in the image, and supplies them to the validity determination unit 103.

  The captured image is an image captured by an imaging device such as a camera, for example. The imaging device may be a terminal having an imaging function such as a mobile phone terminal, a smart phone, a digital camera, or a tablet, or may be a monitoring camera installed in a store.

  The first recognition unit 101 receives this captured image from, for example, an imaging device. In addition, the method in which the 1st recognition part 101 receives a picked-up image is not specifically limited. For example, the first recognition unit 101 may receive a captured image from an imaging device connected to the product shelf recognition device 10 using a USB (Universal Serial Bus) cable or the like. For example, the 1st recognition part 101 may receive a captured image from the imaging device connected with the goods shelf recognition apparatus 10 via the network.

  The first recognition unit 101 may receive a captured image from, for example, a storage device in which captured images are stored. Note that the first recognition unit 101 may receive position information indicating a position where the captured image is captured together with the captured image.

  In this embodiment, for convenience of explanation, the captured image is described as a single still image. However, for example, an image captured continuously in the time direction, such as a moving image, may be used.

  In this embodiment, a store shelf or vending machine to be photographed is one or a plurality of products or an object (for example, a sample of a product) indicating one or a plurality of products displayed. There may be. Further, the housing or fixture including the shelf to be photographed may have a transparent panel on the front surface. In the present embodiment, the vending machine to be imaged may be one in which an image showing one or a plurality of products is displayed on the display. The method for indicating the products to be sold by the store shelves or vending machines is not particularly limited.

  FIG. 2 shows an example of a captured image used in this embodiment. FIG. 2 is a diagram for explaining an example of a captured image received by the commodity shelf recognition apparatus 10 according to the present embodiment. In addition, in FIG. 2, the picked-up image when image | photographing the vending machine is shown as the example. As shown in FIG. 2, the photographed image includes a plurality of product samples 2101. Further, the photographed image includes, for example, an area 2102 indicating the price of a product specified by the product sample 2101 and a button 2103 for purchasing the product, which are associated with each product sample 2101. Here, the product sample 2101 indicates a product sold by a vending machine including the product sample 2101. Hereinafter, an image of the product sample 2101 is referred to as a product image. In FIG. 2, it is assumed that the alphabet in each product sample 2101 represents a product name.

  In the present embodiment, the photographed image is an image obtained by photographing one vending machine as shown in FIG. 2, but may be an image obtained by photographing a plurality of vending machines. Further, the photographed image may be a photograph of a part of one vending machine. Needless to say, the same may be applied to the case of photographing a store shelf.

  The first storage unit 102 stores the product indicated by the product sample 2101 included in the photographed image and feature information for recognizing the price from the part 2102 indicating the product price. Specifically, in the first storage unit 102, at least one of a product or product sample image and a feature amount included in the product or product sample image is stored in association with information for identifying the product. ing. The information for identifying the product is, for example, an identifier for identifying the product or a product name. Note that the feature information stored in the first storage unit 102 may be information necessary for identifying a product. Specifically, for example, the brightness value, edge component, or HOG (Histogram Of Gradient) of a product image and a character image may be used. The information for identifying the product may be, for example, the above-described information and a distance image including a distance value from the imaging device to each pixel. By using the distance image, the shape of the product can be estimated from the image, and thus products having the same design but different shapes, such as canned drinks and plastic bottle drinks, can be uniquely recognized.

  Further, the first storage unit 102 stores character data necessary for recognizing characters constituting the price of a product for each font. The character data stored in the first recognizing unit 101 may be data used for character recognition, and the type thereof is not particularly limited.

  Further, the first storage unit 102 may store information related to the conditions of a vending machine (a vending machine that is a photographing target) that is a target for product recognition. Hereinafter, a vending machine that is a target for product recognition is referred to as a target vending machine. Information on vending machine conditions is also referred to as vending machine condition information. The vending machine condition information is, for example, information indicating conditions regarding the price of the product handled by the vending machine described above, or a condition that a character string indicating the price of the product indicated by the product sample 2101 exists below the product sample 2101. It may include information about Note that the vending machine condition information is not limited to this, and the first storage unit 102 may include information indicating a position where the vending machine is installed. The information indicating the position may be information received by the first recognition unit 101.

  In addition, the information for recognizing the product, the character data, and the vending machine condition information may be stored in the same storage device (for example, the first storage unit 102) or different from each other. It may be stored in a storage device.

  The first storage unit 102 may be built in the target vending machine, or may be realized by a storage device that is separate from the target vending machine.

  Next, with reference to FIG. 3, the detail of the 1st recognition part 101 of the goods shelf recognition apparatus 10 which concerns on this embodiment is demonstrated. FIG. 3 is a functional block diagram illustrating an example of a functional configuration of the first recognition unit 101 of the commodity shelf recognition apparatus 10 according to the first embodiment of the present invention. As shown in FIG. 3, the first recognition unit 101 includes a product recognition unit 1011 and a character recognition unit 1012.

  The product recognition unit 1011 refers to the feature information for recognizing the product stored in the first storage unit 102, recognizes the subject (product sample 2101 in this embodiment) included in the received captured image, A price search image area is determined based on the recognition result, and the price search image area is supplied to the character recognition unit 1012. The recognition result will be described later.

  At this time, for example, the product recognition unit 1011 cuts out a local region including the product sample 2101 from the photographed image, compares the local region with the feature information stored in the first storage unit 102, and compares the local region. You may recognize the goods contained in.

  In addition, the product recognition unit 1011 may recognize without cutting out a local region including the product from the captured image, for example. In that case, the product recognition unit 1011 detects a number of characteristic points (feature points) from the photographed image, and based on the coordinate value, scale (size), and angle (direction) of each feature point, the individual feature points A local feature amount may be calculated for each local region including. The product recognizing unit 1011 includes the calculated local feature amount in the captured image by comparing the set of local feature amounts with the set of local feature amounts calculated in the same manner from the product image and stored in the first storage unit 102. You may be able to recognize products In addition, the method in which the goods recognition part 1011 recognizes goods is not specifically limited.

  The recognition result output by the product recognition unit 1011 will be specifically described with reference to FIG. The product recognition unit 1011 may output, for example, information indicating the recognized product sample 2101 (for example, product name) and image area information indicating the product image area 2301 associated with the product as a recognition result. . Here, the image area information is, for example, position information of the product image area 2301. For example, as shown in FIG. 4, if the product image area 2301 is rectangular, the position information may be the coordinate values of the four corners, the coordinate value of any one of the four corners, and the product The width and height of the image area 2301 may be used.

  In addition, the product recognition unit 1011 determines the price search image area 2302 based on the above recognition result. The price search image area 2302 may be, for example, image data around the product image area 2301 as shown in FIG. For example, as shown in FIG. 2, when it is known that there is a product price under the product sample 2101, only the image region under the product image region 2301 may be set as the price search image region 2302.

  Next, returning to FIG. 3, the character recognition unit 1012 will be described. The character recognition unit 1012 refers to the feature information for recognizing the characters constituting the price stored in the first storage unit 102 from the received price search image region 2302, and is included in the price search image region 2302 Recognize characters Here, the characters are, for example, numbers from 0 to 9, symbols such as “¥” and “yen”, and kanji. The character recognition unit 1012 outputs a recognition result including position information and type of the recognized character in the captured image. The character type may be, for example, a number, a symbol such as “¥” or “yen”, or an identifier that can uniquely identify a Chinese character. The character recognizing unit 1012 may recognize the characters constituting the price from the price search image area using a well-known character recognition technique. In addition, the method in which the character recognition part 1012 recognizes a character is not specifically limited.

  The recognition result output by the character recognition unit 1012 will be specifically described with reference to FIG. The character recognizing unit 1012 includes, for example, information indicating the recognized character 2304 (numbers such as “0” and “9”, symbols such as “¥”, or kanji characters such as “yen”), the recognized character 2304, The information indicating the similarity to the characters constituting the price and the character image area information indicating the character image area 2303 associated with the character stored in the one storage unit 102 may be output as the recognition result. Good.

  The information indicating the similarity of the character 2304 is extracted from, for example, the feature amount constituting the feature information for recognizing the character constituting the price stored in the first storage unit 102 and the price search image area 2302. It may be calculated from the distance between feature amounts between the feature amounts constituting the feature information. In the present embodiment, for convenience of explanation, it is assumed that both are more similar as the distance between feature amounts approaches 0 and the degree of similarity is higher.

  Here, the character image area information is, for example, position information of the character image area 2303 of the character 2304 in the price search image area 2302. For example, as shown in FIG. 4, if the character image area 2303 is a rectangle, the position information may be the coordinate values of the four corners, the coordinate value of one of the four corners, and the character It may be represented by the width and height of the image area 2303.

  Next, returning to FIG. 1, the validity determination unit 103 will be described. The validity determination unit 103 receives the recognition result from the first recognition unit 101, detects the price using them, and determines whether the detected price is appropriate as the price of the product included in the photographed image. To do. The effectiveness determination unit 103 extracts feature information for recognizing the image regions from the image regions with high effectiveness and supplies them to the second storage unit.

  With reference to FIG. 5, the detail of the effectiveness determination part 103 of the goods shelf recognition apparatus 10 which concerns on this embodiment is demonstrated. FIG. 5 is a functional block diagram illustrating an example of a functional configuration of the validity determination unit 103 of the commodity shelf recognition apparatus 10 according to the first embodiment of the present invention. As shown in FIG. 5, the validity determination unit 103 includes a key character detection unit 1031, a price detection unit 1032, and a feature extraction unit 1033.

  The key character detection unit 1031 detects the key character from the received recognition result and supplies the key character to the price detection unit 1032. Here, the key character may be, for example, a symbol attached to the beginning or end of the price, such as “¥” or “yen”.

  The price detection unit 1032 searches for a character recognition result other than the key characters around the key character received from the key character detection unit 1031. When the price detection unit 1032 finds a character recognition result other than the key characters, the price detection unit 1032 determines the validity of whether the price constituted by them is appropriate as the price of the product included in the photographed image. As a result of the determination, the price detection unit 1032 supplies image area information with a highly effective price to the feature extraction unit 1033.

  Here, an example of the operation of the price detection unit 1032 will be described in detail with reference to FIG. In FIG. 4, the “¥” mark is a key character, and the character image area 2303 arranged on the right side indicates a character recognition result other than the key character. For example, the price detection unit 1032 may detect a set of characters “1”, “3”, and “0” arranged in a line on the right side of the key character as a price including the character.

  Further, for example, when the key character detection unit 1031 cannot detect the key character, the price detection unit 1032 may detect the price based on the positions of the plurality of character image areas 2303. Specifically, for example, if characters “1”, “3”, and “0” are arranged at regular intervals, the price detection unit 1032 may detect “130” as the price of the product.

  For example, the price detection unit 1032 determines the validity of whether or not the price is appropriate as the price of the product included in the photographed image, based on the similarity between the characters constituting the price. In this case, for example, when the similarity is the distance between feature amounts, the price detection unit 1032 determines that the effectiveness is higher as the distance between feature amounts is smaller and the effectiveness is lower as the distance between feature amounts is larger. Good. Specifically, for example, the price detection unit 1032 may set the effectiveness to 1 if the sum of the similarities, that is, the sum of the distances between the feature amounts is less than the threshold, and set the effectiveness to 0 if the sum is greater than or equal to the threshold. In the present embodiment, for convenience of explanation, it is assumed that the effectiveness is expressed in a range from 0 to 1, and the effectiveness is closer to 1.

  In addition, for example, when the sum of the similarities is less than the threshold value, the price detecting unit 1032 may perform an interpolation process according to the sum of the similarities and set a numerical value from 0 to 1 as the validity. Specifically, for example, when the similarity is a distance between feature amounts, the price detection unit 1032 sets the effectiveness when the sum of the similarity degrees, that is, the sum of the distances between the feature amounts is 0, and the distance between the feature amounts. The effectiveness when the sum of the values is a threshold value may be set to 0, and the interval between them may be linearly interpolated.

  The price detection unit 1032 may determine the validity of the price detected based on the price range of the product of the target vending machine with reference to the vending machine condition information, for example. Specifically, for example, if the maximum price of a product handled by the target vending machine is 170 yen and the minimum value is 120 yen, the effectiveness is valid if the detected price is less than 120 yen or greater than 170 yen. May be 0, otherwise the effectiveness may be 1.

  Further, for example, the price detection unit 1032 may determine the validity by calculating the sum after multiplying the similarity of each character by a weighting factor based on the arrangement of characters constituting the price. Specifically, for example, in the case of a vending machine, the hundreds digit is often “1” as the price range of the product for sale. Therefore, the price detection unit 1032 sets the weighting factor to 1.0 if the hundreds of the characters constituting the price is “1”, otherwise sets the weighting factor to 0.5, and sets this weight to a similar value. You may multiply every degree.

  In addition, the price detection unit 1032 may determine the validity based on, for example, the width or height of the character image area 2303 indicating individual characters constituting the price. Specifically, for example, when the heights of the plurality of character image areas 2303 shown in FIG. 4 are equal, the validity may be set to 1, and otherwise, the validity may be set to 0. For example, the price detection unit 1032 may set the validity to 1 if the difference in height between the plurality of character image areas 2303 is within a predetermined threshold, and set the validity to 0 otherwise.

  Further, the price detection unit 1032 may increase the effectiveness when a key character can be detected, and decrease the effectiveness when a key character cannot be detected, for example.

  The position information (price image area information) of the image area of the price determined to be highly effective may be expressed by, for example, the coordinate values of the four corners of the rectangle that circumscribes all the characters 2304 constituting the price. Alternatively, the price image area information may be expressed by the coordinate value of any one of the four corners of the rectangle and the width and height of the rectangle.

  The price detection unit 1032 is comprehensively based on two or more criteria among the above-described distance between feature amounts, vending machine condition information, character arrangement, character image area width or height, and presence / absence of key character detection. Effectiveness as a price may be determined.

  Next, returning to FIG. 5, the feature extraction unit 1033 will be described. The feature extraction unit 1033 receives the price image region, extracts feature information for recognizing the price included in the price image region, and supplies the feature information to the second storage unit 104. Note that the feature information supplied to the second storage unit 104 may be information necessary to identify the price. Specifically, for example, the luminance value, edge component, or HOG of the price image region may be used, or the RGB value of the price image region may be used.

  Next, returning to FIG. 1, the second storage unit 104 will be described. The second storage unit 104 stores the above-described information for recognizing the price included in the price image area.

  The second storage unit 104 may store vending machine condition information similar to that of the first storage unit 102.

  The feature information for recognizing the price and the vending machine condition information may be stored in the same storage device (for example, the second storage unit 104), or may be stored in different storage devices. It may be done.

  The second storage unit 104 may be built in the target vending machine, or may be realized by a storage device separate from the target vending machine.

  The second recognizing unit 105 refers to the feature information for recognizing the price stored in the second storage unit 104 and recognizes the price included in the received captured image.

  At this time, the second recognition unit 105 may use a method similar to that of the first recognition unit 101, for example. Note that the method by which the second recognition unit 105 recognizes the price is not particularly limited.

  Here, with reference to FIG. 22, an example of an operation when the luminance value of the price image area is stored as feature information in the second storage unit 104 will be described. FIG. 22 is a diagram schematically illustrating an example of the operation of the commodity shelf recognition apparatus 10 according to the present embodiment. In the example shown in FIG. 22, all characters are visible in the price in the area 2202 indicating the price of the product specified by the product sample 2101 in the upper stage. On the other hand, a part of the price in the area 2202 indicating the price of the product specified by the lower product sample 2101 is shielded by the product sample 2101.

  In the example shown in FIG. 22, the first recognizing unit 101 recognizes characters constituting the product and the price by the above-described method, and supplies the recognition result to the validity determining unit 103. The validity determination unit 103 detects the price image region 2601 from the supplied recognition result by the method described above. The validity determination unit 103 supplies the detected luminance value of the price image area 2601 (in this example, an image including “¥ 150”) to the second storage unit 104 as feature information for recognizing the price. . Then, the second recognition unit 105 refers to the luminance value of the price image area 2601 stored in the second storage unit 104 and searches for an image area that matches the price image area included in the captured image 2100. . That is, in this example, the price image “¥ 150” and the captured image 2100 are collated, and “¥ 150” included in the captured image 2100 is searched.

  Here, in the example illustrated in FIG. 22, the second recognition unit 105 displays a similar region 2602 determined to be similar between the captured image 2100 and the price image region 2601 with a dotted circle. A pair 2603 of similar regions between the photographed image 2100 and the price image region 2601 is indicated by a solid line. Further, the price image area 2604 recognized by the second recognition unit 105 is displayed as a solid rectangle.

  The recognition result of the merchandise shelf recognition apparatus 10 according to the present embodiment may be displayed on a display device or stored in a storage device. The merchandise shelf recognition apparatus 10 and the display device or storage device may be connected by a USB cable or may be connected via a network. Moreover, the merchandise shelf recognition apparatus 10 may superimpose the recognition result on the captured image 2100 and display it on the display device or store it in the storage device.

(Operation flowchart)
Next, the flow of operations of the product shelf recognition apparatus 10 according to this embodiment will be described with reference to FIG. FIG. 6 is a flowchart showing an example of the operation flow of the commodity shelf recognition apparatus 10 according to the present embodiment.

  First, the first recognition unit 101 refers to feature information for recognizing a product stored in the first storage unit 102, and recognizes a product included in the received captured image (step S1001). Next, the first recognition unit 101 sets a peripheral area of the product as a price search image area (step S1002). Next, the first recognition unit 101 refers to the feature information for recognizing characters stored in the first storage unit 102, and recognizes the characters in the price search image area (step S1003). Next, the validity determination unit 103 detects a key character and a price from the recognized characters (step S1004), and determines whether the price has been detected (step S1005).

  If the price cannot be detected (No in step S1005), it is determined whether there is any other product that can be recognized (step S1006). If there is no other recognized product (No in step S1006), the process ends. When there is another product that can be recognized (step S1006 Yes), the process proceeds to step S1003.

  When the price can be detected (Yes in step S1005), the validity determination unit 103 calculates the validity of the detected price as the price of the product included in the captured image (step S1007). Next, the validity determination unit 103 determines whether the price is highly effective (step S1008).

  When the effectiveness is low (No in step S1008), the process proceeds to step S1006.

  If the effectiveness is high (step S1008 Yes), a feature for recognizing the price is extracted from the price image area (step S1009) and stored in the second storage unit 104 (step S1010). Next, the second recognition unit 105 refers to the feature information for recognizing the price stored in the second storage unit 104, recognizes the price included in the received captured image (step S1011), The process proceeds to step S1006.

(Effect of the invention)
As described above, the merchandise shelf recognition apparatus 10 according to the present embodiment recognizes the price included in the captured image using information obtained from an area having high effectiveness as the price included in the captured image. As a result, the product shelf recognition apparatus 10 can accurately recognize the product information of the product being sold because it can be recognized accurately even if a part of the characters constituting the price is blocked by the product or the shadow of the product. Can do.

<Second Embodiment>
With reference to FIG. 7, the goods shelf recognition apparatus 20 which concerns on the 2nd Embodiment of this invention is demonstrated. FIG. 7 is a functional block diagram illustrating an example of a functional configuration of the commodity shelf recognition apparatus 20 according to the present embodiment. The merchandise shelf recognition apparatus 20 according to the present embodiment supplies the recognition result of the second recognition unit to the second storage unit 104. In the following description, elements similar to those already described are denoted by the same reference numerals, and description thereof will be omitted as appropriate. This also applies to the third and subsequent embodiments.

(Functional configuration)
As shown in FIG. 7, the product shelf recognition apparatus 20 according to the present embodiment includes a first recognition unit 101, a first storage unit 102, an effectiveness determination unit 103, a second storage unit 104, 2 recognition units 201. The operation of the second recognition unit 201 is partly different from that of the first embodiment.

  The second recognizing unit 201 refers to the feature information for recognizing the price stored in the second storage unit 104, and recognizes the price included in the received captured image. At this time, the second recognition unit 105 may use a method similar to that of the first recognition unit 101, for example.

  Then, the second recognition unit 201 supplies an image area including the price included in the captured image to the second storage unit.

(Operation flowchart)
Next, the flow of operations of the product shelf recognition apparatus 20 according to the present embodiment will be described with reference to FIG. FIG. 8 is a flowchart showing an example of the operation flow of the commodity shelf recognition apparatus 20 according to the present embodiment. Steps S2001 to S2011 shown in FIG. 8 are the same as steps S1001 to S1011 shown in FIG.

  After execution of steps S2001 to S2011 shown in FIG. 8, the second recognition unit 201 determines whether the price included in the captured image has been recognized (step S2012).

  When the price can be recognized (step S2012 Yes), the process proceeds to step S2010. When the price cannot be recognized (No in step S2012), the process proceeds to step S2006.

(Effect of the invention)
As described above, the merchandise shelf recognition apparatus 20 according to the present embodiment recognizes the price included in the captured image using information obtained from the region having high effectiveness as the price included in the captured image, and recognizes the recognition. The result is stored in the storage unit. Thereby, the merchandise shelf recognition apparatus 20 can collect price region image information of various variations with different illumination fluctuations, camera sensor noises, and the like even at the same price. Therefore, according to this embodiment, robustness against illumination fluctuations and camera sensor noise can be improved as compared to the first embodiment.

<Third Embodiment>
With reference to FIG. 9, a product shelf recognizing device 30 according to a third embodiment of the present invention will be described. FIG. 9 is a functional block diagram illustrating an example of a functional configuration of the commodity shelf recognition apparatus 30 according to the present embodiment. The product shelf recognition apparatus 30 according to the present embodiment generates a price pattern by combining feature information extracted from a character area in an arbitrary price image area and feature information extracted from a character area in another price image area To do.

(Functional configuration)
As illustrated in FIG. 9, the product shelf recognition apparatus 30 according to the present embodiment includes a first recognition unit 101, a first storage unit 102, an effectiveness determination unit 103, a pattern generation unit 301, and a second A storage unit 104 and a second recognition unit 105 are provided.

  The validity determination unit 103 is the same as the element in the first embodiment. In the present embodiment, the validity determination unit 103 supplies the price generation region feature information to the pattern generation unit 301.

  The pattern generation unit 301 receives the feature information of the price image region, and generates a new price pattern by combining the feature information extracted from the character regions in the different price image regions in the received feature information. The pattern generation unit 301 supplies feature information extracted from the generated new price pattern to the second storage unit 104.

  An example of the operation of the pattern generation unit 301 will be described in detail with reference to FIG. FIG. 10 is a diagram illustrating an example of the operation of the pattern generation unit 301 of the product shelf recognition device 30 according to the present embodiment.

  The pattern generation unit 301 selects feature information of an arbitrary character image area 2402 in the price image area 2401 and generates a price pattern 2403. In the example illustrated in FIG. 10, the pattern generation unit 301 includes the feature information extracted from the character image area 2402 indicating “¥” in the price image area 2401 specified by the product sample 2404 and the price specified by the product sample 2405. Feature information extracted from the character image region 2402 indicating “3” in the image region 2401 and character image region 2402 indicating “2” and “0” in the price image region 2401 specified by the product sample 2406 A price pattern 2403 of “¥ 320” is generated in combination with the feature information.

  Thus, even if the first recognition unit 101 cannot recognize only “¥ 320” in the captured image, the “¥ 120” and “¥ 130” recognized by the first recognition unit 101 are used. Since “¥ 320” can be generated, the second recognition unit 105 can recognize “¥ 320”.

(Operation flowchart)
Next, with reference to FIG. 11, the flow of operation of the product shelf recognition apparatus 30 according to the present embodiment will be described. FIG. 11 is a flowchart showing an example of the operation flow of the commodity shelf recognition apparatus 30 according to the present embodiment. Steps S3001 to S3009 and steps S3011 to S3012 shown in FIG. 11 are the same as steps S1001 to S1009 and steps S1010 to S1011 shown in FIG.

  After execution of steps S3001 to S3009 shown in FIG. 11, the pattern generation unit 301 combines feature information extracted from characters in an arbitrary price image region and feature information extracted from characters in another price image region. A price pattern is generated (step S3010). Thereafter, the process proceeds to step S3011.

(Effect of the invention)
As described above, the product shelf recognition device 30 according to the present embodiment uses the information obtained from the region having high effectiveness as the price included in the photographed image, and features extracted from characters in an arbitrary price image region. A price pattern is generated by combining information and feature information extracted from characters in another price image area. Thereby, even if the 1st recognition part cannot recognize the price of the goods which have only one, for example, since the price of the goods concerned can be recognized in the 2nd recognition part, from the 1st embodiment Product information can also be suitably collected.

<Fourth Embodiment>
With reference to FIG. 12, a product shelf recognition apparatus 40 according to a fourth embodiment of the present invention will be described. FIG. 12 is a functional block diagram illustrating an example of a functional configuration of the commodity shelf recognition apparatus 40 according to the present embodiment. The product shelf recognition apparatus 40 according to the present embodiment generates a price pattern by combining feature information extracted from characters in an arbitrary price image region and feature information extracted from characters in another price image region, Furthermore, the recognition result of the second recognition unit is supplied to the second storage unit.

(Functional configuration)
As shown in FIG. 12, the product shelf recognition apparatus 40 according to the present embodiment includes a first recognition unit 101, a first storage unit 102, an effectiveness determination unit 103, a pattern generation unit 301, and a second A storage unit 104 and a second recognition unit 201 are provided.

(Operation flowchart)
Next, with reference to FIG. 13, the flow of operation of the product shelf recognition apparatus 40 according to the present embodiment will be described. FIG. 13 is a flowchart illustrating an example of an operation flow of the product shelf recognition apparatus according to the present embodiment. Steps S4001 to S4012 shown in FIG. 13 are the same as steps S3001 to S3012 shown in FIG.

  After execution of steps S4001 to S4012 shown in FIG. 13, the second recognition unit 201 determines whether or not the price included in the captured image has been recognized (step S4013).

  If the price can be recognized (step S4013 Yes), the process proceeds to step S4011. If the price cannot be recognized (No in step S4013), the process proceeds to step S4006.

(Effect of the invention)
As described above, the product shelf recognition device 40 according to the present embodiment can obtain the same effects as those of the third embodiment. In addition, the product shelf recognition device 40 according to the present embodiment recognizes the price included in the captured image using information obtained from an area having high effectiveness as the price included in the captured image, and stores the recognition result. Store in the department. As a result, it is possible to collect price region image information of various variations with different illumination fluctuations and camera sensor noises, etc. even at the same price, and thus robust against illumination fluctuations and camera sensor noises compared to the third embodiment. Can be improved.

<Fifth Embodiment>
With reference to FIG. 14, a product shelf recognizing device 50 according to a fifth embodiment of the present invention will be described. FIG. 14 is a functional block diagram illustrating an example of a functional configuration of the commodity shelf recognition apparatus 50 according to the present embodiment. The product shelf recognition apparatus 50 according to the present embodiment includes feature information extracted from characters in an arbitrary price image region, feature information extracted from characters in another price image region, and a price stored in the storage unit. The price pattern is generated by combining with the feature information for recognizing.

(Functional configuration)
As shown in FIG. 14, the product shelf recognition apparatus 50 according to the present embodiment includes a first recognition unit 101, a first storage unit 102, an effectiveness determination unit 103, a pattern generation unit 501, and a second A storage unit 104 and a second storage unit 105 are provided.

  The pattern generation unit 501 receives the feature information of the price image region, and extracts the feature information extracted from the character region in the different price image region and the price stored in the first storage unit 102 from the received feature information. A new price pattern is generated in combination with feature information for recognition. The pattern generation unit 501 supplies feature information extracted from the generated new price pattern to the second storage unit 104.

  An example of the operation of the pattern generation unit 501 will be described in detail with reference to FIG. FIG. 15 is a diagram illustrating an example of the operation of the pattern generation unit 501 of the product shelf recognition apparatus 50 according to the present embodiment.

  The pattern generation unit 501 selects feature information of an arbitrary character image area 2502 in the price image area 2501 and generates a price pattern 2503. In the example illustrated in FIG. 15, the pattern generation unit 501 includes the feature information extracted from the character image area 2502 indicating “¥” in the price image area 2501 specified by the product sample 2504 and the price specified by the product sample 2505. Feature information extracted from the character image area 2402 indicating “1” in the image area 2501, and feature information extracted from the character image area 2502 indicating “0” in the price image area 2501 specified by the product sample 2506; A price pattern 2503 “¥ 140” is generated by combining the feature information extracted from the character image information 2507 indicating “4” stored in the first storage unit 102.

  Thus, even if the first recognition unit 101 cannot recognize only “¥ 140” in the captured image, “¥ 130” that can be recognized by the first recognition unit 101 and the first storage unit. Since “¥ 140” can be generated using the feature information for recognizing “4” stored in 102, the second recognition unit 105 can recognize “¥ 140”.

(Operation flowchart)
Next, with reference to FIG. 16, the flow of operation of the product shelf recognition apparatus 50 according to the present embodiment will be described. FIG. 16 is a flowchart showing an example of the operation flow of the commodity shelf recognition apparatus 50 according to the present embodiment. Steps S5001 to S5009 and steps S5011 to S3012 shown in FIG. 16 are the same as steps S3001 to S3009 and steps S3011 to S3012 shown in FIG.

  After execution of steps S5001 to S5009 shown in FIG. 16, the pattern generation unit 501 includes feature information extracted from characters in an arbitrary price image region, feature information extracted from characters in another price image region, and first information The price pattern is generated by combining with the feature information for recognizing the price stored in the storage unit 101 (step S5010). Thereafter, the process proceeds to step S5011.

(Effect of the invention)
As described above, the product shelf recognition apparatus 50 according to the present embodiment uses the information obtained from the area having high effectiveness as the price included in the photographed image, and features extracted from characters in an arbitrary price image area. A price pattern is generated by combining information, feature information extracted from characters in another price image area, and feature information for recognizing a price stored in the storage unit. Thereby, even when there is a character constituting the price of the product that the first recognition unit could not recognize, the second recognition unit can recognize the price of the product including the character. Goods information can be collected more suitably than a form.

<Sixth Embodiment>
With reference to FIG. 17, a product shelf recognizing device 60 according to a sixth embodiment of the present invention will be described. FIG. 17 is a functional block diagram illustrating an example of a functional configuration of the commodity shelf recognition apparatus 60 according to the present embodiment. The product shelf recognition apparatus 60 according to the present embodiment includes feature information extracted from characters in an arbitrary price image region, feature information extracted from characters in another price image region, and a price stored in the storage unit. The price information is generated by combining the feature information for recognizing and the recognition result of the second recognition unit is supplied to the second storage unit.

(Functional configuration)
As illustrated in FIG. 17, the product shelf recognition apparatus 60 according to the present embodiment includes a first recognition unit 101, a first storage unit 102, an effectiveness determination unit 103, a pattern generation unit 501, and a second A storage unit 104 and a second recognition unit 201 are provided.

(Operation flowchart)
Next, with reference to FIG. 18, the flow of operation of the product shelf recognition apparatus 60 according to the present embodiment will be described. FIG. 18 is a flowchart showing an example of the operation flow of the product shelf recognition apparatus according to the present embodiment. Steps S6001 to S6012 shown in FIG. 18 are the same as steps S5001 to S5012 shown in FIG.

  After execution of steps S6001 to S6012 shown in FIG. 18, the second recognition unit 201 determines whether or not the price included in the captured image has been recognized (step S6013).

  If the price can be recognized (step S6013 Yes), the process proceeds to step S6011. If the price cannot be recognized (No in step S6013), the process proceeds to step S6006.

(Effect of the invention)
As described above, the product shelf recognition apparatus 60 according to the present embodiment can obtain the same effects as those of the fifth embodiment. In addition, the product shelf recognition device 60 according to the present embodiment recognizes the price included in the captured image using information obtained from the region having high effectiveness as the price included in the captured image, and stores the recognition result. Store in the department. As a result, it is possible to collect price region image information of various variations with different illumination fluctuations and camera sensor noises even at the same price, and thus robust against illumination fluctuations and camera sensor noises than the fifth embodiment. Can be improved.

<Seventh Embodiment>
With reference to FIG. 19, a merchandise shelf recognition apparatus 70 according to a seventh embodiment of the present invention will be described. FIG. 19 is a functional block diagram illustrating an example of a functional configuration of the commodity shelf recognition apparatus 70 according to the present embodiment. The merchandise shelf recognition apparatus 70 according to the present embodiment does not consider the validity as the price of the characters recognized by the first recognition unit. That is, the 1st recognition part of the goods shelf recognition apparatus 70 recognizes a character as a price.

(Functional configuration)
As illustrated in FIG. 19, the product shelf recognition device 70 according to the present embodiment includes a first recognition unit 101 and a second recognition unit 105.

  The first recognition unit 101 recognizes a product from the image, and recognizes the price of the product from the peripheral area of the image area of the product. The method by which the first recognition unit 105 recognizes characters is not particularly limited, and a known character recognition method may be used.

  The second recognizing unit 105 includes the feature information of the peripheral region of the image area of the first product whose price is recognized by the first recognizing unit 101, and the peripheral region of the image region of the second product included in the image. Based on the feature information, the price of the second product is recognized.

(Effect of the invention)
As described above, the product shelf recognition apparatus 70 according to the present embodiment recognizes the price included in the captured image again using the information recognized as the price of the product included in the captured image. Thereby, according to the merchandise shelf recognition apparatus 70, it is possible to accurately recognize even if a part of the characters constituting the price is blocked by the merchandise or the shadow of the merchandise. it can.

<Eighth Embodiment>
With reference to FIG. 20, an image processing apparatus 80 according to an eighth embodiment of the present invention will be described. FIG. 20 is a functional block diagram illustrating an example of a functional configuration of the image processing apparatus 80 according to the present embodiment.

(Functional configuration)
As illustrated in FIG. 20, the image processing apparatus 80 according to the present embodiment includes a first recognition unit 801 and a second recognition unit 805.

  The first recognition unit 801 recognizes a product from the image, and recognizes product information related to the product from the peripheral area of the image area of the product.

  The second recognizing unit 805 includes feature information of the peripheral region of the image region of the first product for which the product information is recognized by the first recognizing unit 801, and the peripheral region of the image region of the second product included in the image. Product information related to the second product is recognized based on the feature information.

  As described above, the product information is, for example, identification information for identifying a product such as a product name, information indicating the price of the product, information indicating whether the product being sold is a cold product or a warm product, And accompanying information associated with the product such as information indicating whether the product is on sale or sold out.

(Effect of the invention)
As described above, the image processing apparatus 80 according to the present embodiment uses the product information related to the recognized product included in the captured image, and again product information related to other products included in the captured image. Recognize Thereby, according to the merchandise shelf recognition apparatus 70, even if a part of the merchandise information is blocked by the merchandise or the shadow of the merchandise, the merchandise shelf recognition apparatus 70 can accurately recognize the merchandise information about the merchandise being sold.

<Example of hardware configuration>
Here, a configuration example of hardware capable of realizing the product shelf recognition apparatus (10, 20, 30, 40, 50, and 60) according to each embodiment described above will be described. The product shelf recognition devices (10, 20, 30, 40, 50, and 60) described above may be realized as dedicated devices, but may be realized using a computer (information processing device).

  FIG. 21 is a diagram illustrating an exemplary hardware configuration of a computer (information processing apparatus) that can implement each embodiment of the present invention.

  The hardware of the information processing apparatus (computer) 2000 shown in FIG. 21 includes a CPU (Central Processing Unit) 2001, a communication interface (I / F) 2008, an input / output user interface 2010, a ROM (Read Only Memory) 2002, a RAM ( Random Access Memory) 2003, a storage device 2005, and a drive device 2007 of a computer-readable storage medium 2006 are connected to each other via a bus 2004. The input / output user interface 2010 is a man-machine interface such as a keyboard which is an example of an input device and a display as an output device. The communication interface 2008 is configured so that the devices according to the above-described embodiments (FIGS. 1, 7, 9, 12, 14, 17, 19, and 20) are connected to an external device via the communication network 2009. It is a general communication means for communicating. In such a hardware configuration, the CPU 2001 controls the overall operation of the product shelf recognition devices (10, 20, 30, 40, 50, 60, and 70) and the image processing device 80 according to the embodiments.

  The present invention described by taking the above embodiments as an example, for example, after supplying a program (computer program) capable of realizing the processing described in each of the above embodiments to the information processing apparatus 2000 shown in FIG. The program is achieved by reading the program into the CPU 2001 and executing it. Note that the program includes, for example, various processes described in the flowcharts (FIGS. 6, 8, 11, 13, 16, and 18) referred to in the description of the above-described embodiments, or FIG. 7, 9, 12, 14, 17, 19, and 20 may be a program capable of realizing each unit (each block) shown in the apparatus.

  The program supplied to the information processing apparatus 2000 may be stored in a readable / writable temporary storage memory (2003) or a non-volatile storage device (2005) such as a hard disk drive. That is, in the storage device 2005, the program group 2005A is shown in, for example, the product shelf recognition device (10, 20, 30, 40, 50, 60, 70 and 80) and the image processing device 80 in each of the above-described embodiments. It is a program that can realize the function of each part. The various storage information 2005B includes, for example, the captured image, the vending machine position information, the information for recognizing the product, the character data, the vending machine condition information, the product information, and the like in each of the above-described embodiments. However, when the program is installed in the information processing apparatus 2000, the structural unit of each program module is a block diagram (FIG. 1, FIG. 7, FIG. 9, FIG. 12, FIG. 14, FIG. 17, FIG. 19 and FIG. 20). It is not limited to the division of each block shown in the above, and a person skilled in the art may select as appropriate when mounting.

  In the above case, the program is supplied to the apparatus by installing it in the apparatus via various computer-readable recording media (2006) such as a CD (Compact Disk) -ROM and a flash memory. Currently, a general procedure can be adopted, such as a method and a method of downloading from the outside via a communication line (2009) such as the Internet. In such a case, the present invention can be understood to be configured by a code (program group 2005A) constituting the computer program or a storage medium (2006) storing the code.

  In the above, this invention was demonstrated as an example applied to exemplary embodiment mentioned above and its Example. However, the technical scope of the present invention is not limited to the scope described in the above-described embodiments and examples. It will be apparent to those skilled in the art that various modifications and improvements can be made to the embodiment. In such a case, new embodiments to which such changes or improvements are added can also be included in the technical scope of the present invention. This is clear from the matters described in the claims.

<Other expressions of the embodiment>
A part or all of the above-described embodiment can be described as in the following supplementary notes, but is not limited thereto.

(Appendix 1)
First recognition means for recognizing a product from an image and recognizing the price of the product from a peripheral region of an image area of the product;
Based on the feature information of the peripheral region of the image region of the first product whose price is recognized by the first recognition means and the feature information of the peripheral region of the image region of the second product included in the image, A second recognition means for recognizing the price of the second product;
A product shelf recognition apparatus comprising:

(Appendix 2)
The image is an image including a store shelf or a vending machine,
The product shelf recognition device according to attachment 1.

(Appendix 3)
Further comprising validity determination means for determining the validity of the price of the product recognized by the first recognition means;
The first recognizing means recognizes a character related to the product as a price from a peripheral region of the image area of the product,
The product shelf recognition apparatus according to appendix 1 or 2, wherein the second recognizing unit uses feature information of a peripheral region of an image region of a product for which the recognized price is determined to be valid by the validity determining unit.

(Appendix 4)
Storage means for storing feature information of the peripheral area of the image area of the product validated by the validity judgment means;
The merchandise shelf recognition apparatus according to appendix 3, wherein the second recognition unit uses feature information stored in the storage unit.

(Appendix 5)
The effectiveness determination means determines the effectiveness from the sum of similarities of recognition results for each character constituting the price.
The product shelf recognition device according to appendix 3 or 4.

(Appendix 6)
The effectiveness determination means determines the effectiveness from the sum of the similarities multiplied by a weighting factor based on the position information of the recognition result.
The product shelf recognition device according to appendix 5.

(Appendix 7)
The validity determination means determines the validity based on a width or height of a region in the image of the character recognized by the first recognition means;
The product shelf recognition apparatus according to any one of appendices 3 to 6.

(Appendix 8)
The product shelf recognition device
Any one of appendices 3 to 7, further comprising a pattern generation unit configured to generate a pattern that combines feature information of the plurality of characters extracted from a region in the image including the price determined to be highly effective. The product shelf recognition device according to the item.

(Appendix 9)
The pattern generation means includes
The first recognition unit generates a pattern combining a plurality of feature information used to recognize the character from the image.
The product shelf recognition device according to appendix 8.

(Appendix 10)
Recognize the product from the image, recognize the price of the product from the peripheral area of the image area of the product,
The price of the second product based on the feature information of the peripheral region of the image region of the first product for which the price is recognized and the feature information of the peripheral region of the image region of the second product included in the image Recognize
Product shelf recognition method.

(Appendix 11)
On the computer,
A first recognition process for recognizing a product from an image and recognizing the price of the product from a peripheral region of an image area of the product;
Based on the feature information of the peripheral region of the image region of the first product whose price is recognized by the first recognition process and the feature information of the peripheral region of the image region of the second product included in the image, A second recognition process for recognizing the price of the second product;
A program that executes

(Appendix 12)
First recognition means for recognizing a product from an image and recognizing product information related to the product from a peripheral region of an image area of the product;
Based on the feature information of the peripheral region of the image region of the first product for which the product information is recognized by the first recognition means and the feature information of the peripheral region of the image region of the second product included in the image, Second recognition means for recognizing product information related to the second product;
An image processing apparatus comprising:

10, 20, 30, 40, 50, 60, 70 Product shelf recognition device 80 Image processing device 101 First recognition unit 102 First storage unit 103 Effectiveness determination unit 104 Second storage unit 105, 201 Second Recognition unit 301, 501 Pattern generation unit 1011 Product recognition unit 1012 Character recognition unit 1031 Key character detection unit 1032 Price detection unit 1033 Feature extraction unit 2001 CPU
2002 ROM
2003 RAM
2004 Bus 2005 Storage Device 2005A Program Group 2005B Various Storage Information 2006 Recording Medium 2007 Drive Device 2008 Communication Interface 2009 Network 2010 Input / Output Interface 2000 Computer 2100 Captured Image 2101, 2404, 2405, 2406, 2504, 2505, 2506 Product Sample 2102, 2202 Product price area 2103, 2203 Product purchase button 2301 Product image region 2302 Price search image region 2303, 2402, 2502 Character image region 2304 Characters 2401, 2501, 2601, 2604 Price image region 2403, 2503 Price pattern 2507 Character image information 2602 Similar region 2603 Similar region pair

Claims (10)

First recognition means for recognizing a product from an image and recognizing the price of the product from a peripheral region of an image area of the product;
Based on the feature information of the peripheral region of the image region of the first product whose price is recognized by the first recognition means and the feature information of the peripheral region of the image region of the second product included in the image, A second recognition means for recognizing the price of the second product;
A product shelf recognition apparatus comprising: The image is an image including a store shelf or a vending machine,
The merchandise shelf recognition apparatus according to claim 1. Further comprising validity determination means for determining the validity of the price of the product recognized by the first recognition means;
The first recognizing means recognizes a character related to the product as a price from a peripheral region of the image area of the product,
The product shelf recognition apparatus according to claim 1, wherein the second recognition unit uses feature information of a peripheral region of an image region of the product for which the recognized price is determined to be valid by the validity determination unit. . Storage means for storing feature information of the peripheral area of the image area of the product validated by the validity judgment means;
The merchandise shelf recognition apparatus according to claim 3, wherein the second recognition unit uses feature information stored in the storage unit. The effectiveness determination means determines the effectiveness from the sum of similarities of recognition results for each character constituting the price.
The merchandise shelf recognition apparatus according to claim 3 or 4. The validity determination means determines the validity based on a width or height of a region in the image of the character recognized by the first recognition means;
The goods shelf recognition apparatus of any one of Claims 3-5. The product shelf recognition device
The pattern generation unit according to any one of claims 3 to 6, further comprising a pattern generation unit configured to generate a pattern obtained by combining feature information of the plurality of characters extracted from an area in the image including the price determined to be highly effective. The merchandise shelf recognition apparatus according to item 1. Recognize the product from the image, recognize the price of the product from the peripheral area of the image area of the product,
The price of the second product based on the feature information of the peripheral region of the image region of the first product for which the price is recognized and the feature information of the peripheral region of the image region of the second product included in the image Recognize
Product shelf recognition method. On the computer,
A first recognition process for recognizing a product from an image and recognizing the price of the product from a peripheral region of an image area of the product;
Based on the feature information of the peripheral region of the image region of the first product whose price is recognized by the first recognition process and the feature information of the peripheral region of the image region of the second product included in the image, A second recognition process for recognizing the price of the second product;
A program that executes First recognition means for recognizing a product from an image and recognizing product information related to the product from a peripheral region of an image area of the product;
Based on the feature information of the peripheral region of the image region of the first product for which the product information is recognized by the first recognition means and the feature information of the peripheral region of the image region of the second product included in the image, Second recognition means for recognizing product information related to the second product;
An image processing apparatus comprising:

Images