JP2019045909A - Image recognition system - Google Patents

Abstract

To provide an image recognition system which specifies a commodity displayed in a store.SOLUTION: An image recognition system include: a commodity tag specification processing unit which specifies a commodity tag area in image information including a display shelf where a commodity is displayed; and a commodity-tag information specification processing unit which applies OCR recognition in the specified commodity tag area to specify information written on a commodity tag. The commodity tag specification processing unit specifies a commodity tag area by use of brightness information of image information. The commodity-tag information specification processing unit converts the specified commodity tag area to a binary value for labeling, to specify boxes, specifies specified boxes satisfying a predetermined condition, as a block, executes OCR recognition on the specified block, and executes logical consistency determination on two pieces of price information specified by the OCR recognition, to identify price excluding tax and price including tax.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image recognition system for identifying a product displayed in a store or the like.

  In the retail industry, it is known that the display condition of goods affects the sale of goods. Therefore, as a product manufacturing company or sales company, it can be connected to the development strategy and sales strategy of their own products by grasping what kind of products of their own company or other companies are displayed in the store.

  On the other hand, accurate identification of the products displayed at the store is important for achieving that. Therefore, it is conceivable that the display shelf of the store is photographed and a person manually specifies the displayed product from the image information. In this case, the product can be identified almost accurately. However, in order to continuously grasp the display status of a product, it is necessary to grasp the display status at regular intervals, but every time a person specifies a product from image information obtained by photographing a display shelf of a store Is burdensome and inefficient.

  Therefore, it is required to automatically identify the products displayed on the display shelves from the image information obtained by photographing the display shelf of the store and grasp the display condition of the products. For example, there is a method of using image recognition technology for an image obtained by photographing a display shelf of a store based on a sample image of each product. As these prior art, there exist systems which manage the display condition of goods using art as shown in the following patent documents 1 thru / or patent documents 3, for example.

Unexamined-Japanese-Patent No. 5-342230 Unexamined-Japanese-Patent No. 5-334409 International Publication 2012/029548

  The invention of Patent Document 1 is a system for assisting even a person who does not have knowledge on which display shelf a product should be displayed. Therefore, it is not possible to grasp the products actually displayed. Moreover, patent document 2 is a system which supports the input of a goods image in the shelving allocation support system which supports the display of goods. However, the system of Patent Document 2 only supports the input of a product image when using the shelving allocation support system, and even if this system is used, it is not possible to grasp the actually displayed products .

  Patent Document 3 is an invention that specifies a product to be displayed in the empty space when there is an empty space in the display shelf, or notifies that when a product to be displayed in the display shelf is misplaced. Even in the present invention, the image matching process between the product and the product displayed on the display shelf is performed to specify the product displayed on the display shelf, but the reality is that the recognition accuracy is low.

  When using an image recognition technology based on a sample image of a product for an image taken of a display shelf, the accuracy and processing load of the image recognition become a problem. For example, in a store where display cabinets are installed, the state of illumination is not uniform, and if the difference between the illumination of the sample image and the illumination is large, the accuracy of image recognition is degraded. Therefore, it may not be easy to identify the goods displayed there from the image which image | photographed the display shelf. In addition, since a large processing load occurs in image recognition, a large amount of resources are required. As a result, investment capital will increase.

  The present inventor focuses on a product tag for displaying the price etc. of the product as one method for identifying the product appearing in the image of the display shelf, and accurately displays the displayed product. We have invented an image recognition system that can be identified. Also, an image recognition system capable of identifying a displayed commodity with high accuracy by recognizing a commodity shown on a display shelf, recognizing information described in a commodity tag, and collating them. Invented.

  In the first invention, a product tag identification processing unit for specifying a product tag area in image information in which a display shelf for displaying a product is shown, and a product tag specifying processing unit performing OCR recognition processing in the specified product tag area Product tag internal information identification processing unit for identifying the processed information, the product tag identification processing unit identifies the product tag area using the lightness information of the image information, and the product tag internal information The specification processing unit specifies the box by binarizing the specified product tag area and performing labeling processing, and specifies the boxes satisfying the predetermined condition as blocks among the specified boxes, and the specified block The above-mentioned OCR recognition processing is executed for the above, and the tax exclusion price is executed by executing the judgment processing of logical consistency for the two price information specified by the above-mentioned OCR recognition. And to identify the tax-inclusive price, which is an image recognition system.

  By using the image recognition system of the present invention, the information written on the product tag can be recognized with high accuracy, so that the displayed product can be identified with high accuracy.

  According to a second aspect of the present invention, there is provided a product tag identification processing unit for specifying a product tag area in image information in which a display shelf for displaying a product is shown, and performing OCR recognition processing on a part or all of the specified product tag areas. And the product tag identification processing unit identifies the product tag area using the lightness information of the image information, and the product tag identification processing unit identifies the information described in the product tag in the product tag. It is an image recognition system.

  Similar to the first invention, by using the image recognition system of the present invention, the information written on the product tag can be recognized with high accuracy, so that the displayed product can be specified with high accuracy. it can.

  In each invention described above, the product tag identification processing unit identifies the upper side position and the lower side position of the product tag by integrating the lightness information of the image information in the horizontal direction, and the lightness information of the image information in the vertical direction The image recognition system can be configured to specify the left side position and the right side position of the product tag by integrating into the above.

  In each of the above-described inventions, the product tag identification processing unit identifies the upper side position and the lower side position of the product tag by specifying the rising and falling of the histogram obtained by integrating the lightness information of the image information in the horizontal direction. Image recognition by specifying the left side position and the right side position of the product tag as a pair by specifying the rise and fall of the histogram obtained by integrating the lightness information of the image information in the vertical direction, as a pair within the predetermined distance from the rise It can be configured as a system.

  In order to accurately recognize the information described in the product tag, it is necessary to specify the product tag area as accurately as possible. Generally, since the ground color of the product tag is white, when the display shelf is photographed, the part of the product tag is bright. Focusing on this point, as in the present invention, the product tag area can be specified with high accuracy by integrating the lightness information of the image information in the vertical direction and the horizontal direction.

  In the above-described invention, the commodity tag identification processing unit may set a tag region within a predetermined distance on the right side of the rising of the pair that can not form among the risings of the histogram obtained by integrating the lightness information of the image information in the vertical direction. If not, the tag area is identified as the product tag area, and of the fall of the histogram in which the lightness information of the image information is integrated in the vertical direction, the tag area is within a predetermined distance on the left side of the fall that could not form a pair. If not, it can be configured as an image recognition system that specifies as a product tag area.

  The product tag may be attached partially overlapping. In that case, in the histogram based on the integration of the lightness information, the rise and fall occur continuously. Therefore, by executing the process of the present invention, it is possible to specify a product tag area for a product tag that is attached in an overlapping manner.

  The sixth invention is described in the product tag by carrying out the OCR recognition process in the specified product tag area and the product tag specification processing unit for specifying the product tag area in the image information in which the display shelf for displaying the product is shown. Product tag internal information specification processing unit for specifying the processed information, and the product tag internal information specification processing unit binarizes the specified product tag area and performs labeling processing It is an image recognition system which specifies the boxes which satisfy | fill a predetermined condition among the specified and specified boxes as a block, and performs the said OCR recognition process with respect to the specified block.

  A seventh aspect of the invention relates to a product tag identification processing unit for specifying a product tag area in image information in which a display shelf for displaying a product is shown, and performing OCR recognition processing on a part or all of the specified product tag areas. And the in-product tag information specification processing unit for specifying information described in the product tag in the product tag, and the in-product tag information specification processing unit logically performs the two price information specified by the OCR recognition. This is an image recognition system that identifies tax-excluded prices and tax-included prices by executing consistency determination processing.

  Similar to the first invention, by using the image recognition system according to these inventions, the information written on the product tag can be recognized with high accuracy, so the products displayed corresponding to it can be specified with high accuracy. Can.

  In each of the above-described inventions, the in-product tag information identification processing unit executes binarization processing in the identified commodity tag area and performs labeling processing in the binarized image information, so that consecutive pixels Are grouped to identify boxes, and adjacent boxes having a predetermined similarity among the identified boxes are merged and identified as a block, and the highest block among the identified blocks is priced It can be configured as an image recognition system that performs OCR recognition processing by estimating a region.

  In order to recognize the information described in the product tag, OCR recognition processing may be executed for all of the product tag areas, but in that case a large amount of noise etc. will be detected and the accuracy will deteriorate. Connect. Moreover, processing time also increases. Therefore, it is preferable to execute the OCR recognition processing only on a part of the product tag area. The recognition accuracy can be improved by executing the OCR recognition processing on the block specifying the processing of the present invention. Also, since the price is most important to the purchaser of the goods, the highest block is often the price area. Therefore, the price can be recognized with high accuracy by estimating this block as a price area and executing the OCR recognition processing.

  In the above invention, the product tag internal information identification processing unit matches the price size and / or the tax rate consistency of the price information recognized as the tax-excluded price and the price information recognized as the tax-included price by the OCR recognition. By performing the determination process of the above, it is possible to configure as an image recognition system that specifies the tax-excluded price and the tax-included price.

  In general, the price excluding tax and the price including tax are indicated on the product tag. Therefore, the accuracy can be further improved by determining whether the logical consistency of the price recognized as the tax-excluded price and the tax-included price respectively is established.

  A tenth aspect of the invention relates to a product tag recognition processing unit for specifying product identification information in a product tag area in image information in which a display shelf for displaying a product is displayed, and product identification information of a product displayed in the image information. The recognition result of comparing the displayed product recognition processing unit to be specified, the product identification information in the specified product tag area, and the product identification information on the specified displayed product, and collating whether the recognition result matches It is an image recognition system which has a collation process part.

  According to the present invention, it is possible to collate the result of recognition of the product appearing on the display shelf with the recognition result of the information written on the product tag. This makes it possible to identify the displayed product with high accuracy.

  In the above-mentioned invention, the recognition result collating processing unit makes the section in which faces with high similarity are lined up as a group by the recognition processing in the display commodity recognition processing unit, and the commodity tag recognition in each group in each tray It can be configured as an image recognition system that associates the recognition result of the product tag by the processing unit.

  By using the present invention, the recognition result of the displayed product can be associated with the recognition result of the product tag.

  In the above-described invention, the image recognition system further includes a photographed image information regularization processing unit that executes keystone correction processing on photographed image information obtained by photographing the display shelf, and the photographed image information The normalization processing unit is configured to use the second vertex of the first captured image information, which is used in the keystone correction processing, based on the correspondence between the first captured image information and the second captured image information. The present invention can be configured as an image recognition system that projects onto captured image information and executes a keystone correction process of the second captured image information based on the projected vertex.

  Photographed image information often does not become image information photographed from the front. If the process is performed from the photographed image information itself, the accuracy is low. Therefore, it is preferable to perform keystone correction processing to correct the image. Therefore, although the person in charge must specify the vertex that is the basis of the keystone correction processing, it is burdensome. By using the present invention, it is possible to automatically specify a vertex which is a reference of the keystone correction processing, and it is possible to reduce the workload of the person in charge.

  In the above-described invention, the image recognition system further includes a position specifying processing unit for specifying a product tag arrangement area in the photographed image information which has been subjected to the keystone correction processing in the photographed image information regularization processing unit; Product tag placement area regularization processing unit for executing keystone correction processing in the product tag placement area, and the product tag identification processing unit performs processing for the image information after the keystone correction processing is performed It can be configured as an image recognition system that performs processing.

  While the surface of the display rack is vertical, the surface of the product tag is often attached upward (toward the customer) than the vertical surface. This is to make it easy for the customer to visually identify the product tag. Therefore, the product tag area can be specified with higher accuracy by performing a regularization process on the product tag placement area. This is because, for example, the rising and falling of the histogram in the lightness information (information on the amount of lightness when the image information is decomposed into two-dimensional information of hue, lightness, and saturation) become sharp. Further, according to the present invention, it is possible to improve the accuracy of the OCR recognition processing in the product tag area.

  In the above-described invention, the commodity tag placement area regularization processing unit performs edge detection in the commodity tag placement region, detects a straight line group satisfying a predetermined condition of an angle in the vertical direction, and detects the straight line group detected. Among them, it can be configured as an image recognition system that identifies at least two straight lines and executes keystone correction processing using the two identified straight lines.

  By executing the process of the present invention, the inclination angle to be used for the keystone correction process can be specified, so that the keystone correction process of the commodity tag arrangement area can be executed.

  The image recognition system of the first invention can be realized by reading and executing the image recognition program of the present invention on a computer. That is, the product tag identification processing unit identifies the product tag area in the image information in which the display shelf for displaying the product is displayed, and the computer displays the product tag by performing the OCR recognition processing in the specified product tag area. An image recognition program to be functioned as a product tag internal information specification processing unit for specifying information, wherein the product tag specification processing unit specifies a product tag area using lightness information of the image information, and The information identification processing unit binarizes the identified product tag area and performs labeling processing to identify a box, and among the identified boxes, identifies boxes satisfying a predetermined condition as blocks, and identifies the blocks. Execute the OCR recognition process on the block and logically match the two price information specified by the OCR recognition By executing the determining process, identifies the tax price and price including tax, an image recognition program.

  The image recognition system of the second invention can be realized by causing a computer to read and execute the image recognition program of the present invention. That is, the product tag identification processing unit that specifies the area of the product tag in the image information in which the display shelf displaying the product is displayed, and the computer performs OCR recognition processing on part or all of the specified product tag area. An image recognition program that causes the product tag identification information processing unit to function as an in-product tag information specification processing unit that specifies information described in a product tag by performing the product tag identification processing unit; Is an image recognition program that specifies the area of

  The image recognition system of the sixth invention can be realized by reading and executing the image recognition program of the present invention on a computer. That is, the product tag identification processing unit that specifies the area of the product tag in the image information in which the display shelf displaying the product is displayed, and the product tag by performing the OCR recognition processing in the area of the specified product tag. An image recognition program that causes the product identification information processing unit to function as a product tag information identification processing unit for identifying the information described in the item information processing unit, wherein the product tag in-data identification processing unit binarizes the area of the identified product tag and performs labeling processing The image recognition program is configured to specify a box by performing a box, specify boxes among the specified boxes that satisfy predetermined conditions as blocks, and execute the OCR recognition processing on the specified block.

  The image recognition system of the seventh invention can be realized by reading and executing the image recognition program of the present invention on a computer. That is, the product tag identification processing unit that specifies the area of the product tag in the image information in which the display shelf displaying the product is displayed, and the computer performs OCR recognition processing on part or all of the specified product tag area. An image recognition program that causes the product identification information processing unit to function as a product tag information specification processing unit that specifies information described in a product tag by performing the product tag information specification processing unit; It is an image recognition program that identifies tax-excluded prices and tax-included prices by executing logical consistency determination processing on information.

  The image recognition system of the tenth invention can be realized by reading and executing the image recognition program of the present invention on a computer. That is, a product tag recognition processing unit that specifies product identification information in a region of a product tag in image information in which a display shelf displaying a product is displayed on a computer, product identification information of a displayed product in the image information A display product recognition processing unit for specifying the product identification information for comparing the product identification information in the area of the specified product tag with the product identification information of the specified displayed product, and checking whether the recognition results match It is an image recognition program to function as a result matching processing unit.

  By using the image recognition system of the present invention, it is possible to accurately identify the products displayed on the display shelf.

It is a block diagram which shows typically an example of a system configuration | structure of the image recognition system of this invention. It is a block diagram which shows typically an example of the hardware constitutions of the computer used with the image recognition system of this invention. It is a flowchart which shows an example of the whole processing process in the image recognition system of this invention. It is a flowchart which shows an example of the processing process of the formalization process of a goods tag arrangement | positioning area | region. It is a flow chart which shows an example of a processing process of specification processing of goods tag picture information. It is a flow chart which shows an example of a processing process of specification processing of information in goods tag. It is a figure which shows an example of the emplacement image information which performed the emplacement process to picked-up image information. It is a figure which shows another example of the emplacement image information which performed the emplacement process to photography image information. It is a figure which shows an example of the state which designated the goods tag arrangement | positioning area | region to the emplacement image information with respect to the emplacement image information of FIG. It is a figure which shows an example of the state which designated the goods tag arrangement | positioning area | region to the emplacement image information with respect to the emplacement image information of FIG. It is a figure which shows an example of the image information which performed the regularization process with respect to the goods tag arrangement | positioning area | region in FIG. It is a figure which shows the state which detected the straight line of the orthogonal | vertical direction used as the reference | standard of a keystone correction process in a goods tag arrangement | positioning area regularization process part. It is a figure which shows typically the process which specifies the upper side position of a goods tag area | region in a goods tag specific process part, and a lower side position. It is a figure which shows typically the process which specifies the position of the horizontal direction of the goods tag area | region in a goods tag specific process part. It is a figure which shows typically the state which specified the goods tag area | region from the image information of the embellished goods tag arrangement | positioning area | region of FIG. It is a figure which shows typically the image information which binarized the goods tag area | region in the process in the information processing part in a goods tag. It is a figure which shows typically the state which produced | generated the box in the process in a product tag internal information identification process part. It is a figure which shows typically the state which produced | generated the block in the process in a product tag internal information specific process part. It is a figure which shows an example of a goods name with the minimum edit distance among the goods names described in the goods tag and the goods name memorize | stored in the goods dictionary of a result of OCR recognition, and an edit distance. It is a figure which shows an example of the correspondence of the edit distance which may be decided for every character number of the last candidate. It is a figure showing typically an example of photography picture information. It is a figure showing typically an example of photography picture information. FIG. 8 is a block diagram schematically showing an example of a system configuration of an image recognition system in a second embodiment. It is a block diagram which shows typically an example of a structure of a goods tag recognition process part. It is a block diagram which shows typically an example of a structure of a display goods recognition process part. 15 is a flowchart illustrating an example of the entire processing process of the image recognition system according to the second embodiment. It is a figure which shows typically the Nth process in a goods identification information specific process part. 5 illustrates an example of sample image information stored in a sample image information storage unit. It is a figure which shows the state in which the tray area and the goods tag arrangement | positioning area | region were specified. It is a figure which shows the state in which the tray area and the goods tag arrangement | positioning area | region were specified. An example of the state which set the feature-value collection area | region with respect to N-1st imaging | photography image information is shown. It is a figure which shows an example of the state which set the feature-value extraction area | region with respect to N-th imaging | photography image information. It is a figure which shows the relationship of the pair of the N-1st feature-value collection area | region and the N-th feature-value collection area | region. It is a figure which shows the state which projects the shelf position C in N-1st imaging | photography image information as a shelf position D in the N-th imaging | photography image information by the function F. FIG. It is a figure showing an example of photography picture information. It is a figure which shows the state which set the feature-value extraction area | region to N-1st imaging | photography image information. It is a figure shown in the state where the feature-quantity extraction area was set to the N-th shot image information. It is a figure which shows an example of the state which specified shelf positions D1 to D4 in N-th captured image information.

  An example of a system configuration of the image recognition system 1 of the present invention is shown in FIG. The image recognition system 1 uses a management terminal 2 and a photographed image information input terminal 4.

  The management terminal 2 is a computer used by an organization such as a company that operates the image recognition system 1. The photographed image information input terminal 4 is a terminal for inputting image information obtained by photographing a display shelf of a store.

  The management terminal 2 and the photographed image information input terminal 4 in the image recognition system 1 are realized using a computer. FIG. 2 schematically shows an example of the hardware configuration of the computer. The computer can input information by using an arithmetic device 70 such as a CPU that executes arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, and a display device such as a display that displays information. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 that transmits and receives the processing result of the arithmetic device 70 and information stored in the storage device 71 via a network such as the Internet or LAN.

  When the computer includes a touch panel display, the display device 72 and the input device 73 may be integrally configured. The touch panel display is often used, for example, in portable communication terminals such as tablet computers and smartphones, but is not limited thereto.

  The touch panel display is an apparatus in which the functions of the display device 72 and the input device 73 are integrated in that input can be directly performed by a predetermined input device (such as a pen for a touch panel) or a finger on the display.

  The photographed image information input terminal 4 may include a photographing device such as a camera in addition to the above-described devices. A portable communication terminal such as a mobile phone, a smartphone, or a tablet computer can also be used as the captured image information input terminal 4.

  The respective means in the present invention may only logically distinguish their functions, and may physically or virtually form the same area. The processing in each means of the present invention can also change the processing order as appropriate. Also, part of the processing may be omitted. For example, the formalization process can be omitted. In that case, it is possible to execute processing for image information that has not been subjected to orthographic processing.

  The management terminal 2 in the image recognition system 1 can transmit and receive information to and from the captured image information input terminal 4 via a network.

  The image recognition system 1 includes a photographed image information input acceptance processing unit 20, a photographed image information storage unit 21, a photographed image information regularization processing unit 22, a position specification processing unit 23, and a commodity tag placement area cutout processing unit A commodity tag arrangement area regularization processing unit 25, a commodity tag specification processing unit 26, and a commodity tag information identification processing unit 27 are provided.

  The photographed image information input reception processing unit 20 receives an input of image information (photographed image information) of a display shelf of a store photographed with the photographed image information input terminal 4 and stores it in a photographed image information storage unit 21 described later. In addition to the photographed image information, it is preferable that the photographed image information, the shop identification information such as the photographing date, the shop name, the image information identification information for identifying the image information and the like be received from the photographed image information input terminal 4 together.

  The photographed image information storage unit 21 stores the photographed image information received by the photographed image information input reception processing unit 20, the photographing date and time, the store identification information, the image information identification information, and the like in association with each other. The photographed image information may be image information to be subjected to the keystone correction processing, and when one display shelf is photographed with a plurality of sheets, the image information combined as one image information is also included. . Further, image information after the distortion correction processing has been performed is also included in the captured image information.

  The photographed image information regularization processing unit 22 generates orthographic image information obtained by performing trapezoidal correction processing on photographed image information stored in the photographed image information storage unit 21. The keystone correction process is a correction process performed so that the shelf of the display shelf reflected in the photographed image information is horizontal and the product tag for the product displayed there is vertical.

  The keystone correction process performed by the photographed image information regularization processing unit 22 receives input of designation of four vertices in the photographed image information, and executes the keystone correction process using each vertex. The four apexes for receiving the designation may be the four apexes of the shelf of the display rack or the four apexes of the shelf position of the display rack. Also, it may be four vertices of a group of two or three trays. Any four points can be specified as the four vertices.

  FIGS. 7 and 8 show an example of photographed image information (for example, orthographic image information) subjected to the orthographic process. FIG. 7 shows right-placed image information in which products of beverage cans such as beer are displayed on upper and lower two tiers as display shelves. FIG. 8 shows a hanging shelf for displaying a product such as a toothbrush and the like on a display shelf, and is right-placed image information displaying the product in two upper and lower stages.

  The position specifying processing unit 23 is a region where there is a possibility that a product tag may be attached to the fixed image information in which the keystone correction processing is performed on the photographed image information in the photographed image information normalization processing unit 22 (product tag arrangement Identify the area). That is, although the display shelf is reflected in the photographed image information and the right-placed image information, the display shelf may have the area of the shelf on which the product is displayed and the possibility that the product tag for the product displayed there is attached There is a product tag placement area with Therefore, the product tag placement area is specified from the orthographic image information. As the specification of the product tag arrangement area, the operator of the management terminal 2 may manually specify the product tag arrangement area, and the position specification processing unit 23 may receive it, or the product tag for which the input is received manually at the first time The commodity tag placement area may be automatically specified for the second and subsequent times based on the information of the placement area.

  FIG. 9 schematically shows a state in which an input of designation of a product tag arrangement area is received with respect to the orthographic image information of FIG. Further, FIG. 10 schematically shows a state in which an input of specification of a product tag arrangement area is received with respect to the orthographic image information of FIG.

  The product tag placement area cutout processing unit 24 cuts out the image information of the product tag placement area specified by the position specification processing unit 23 as product tag placement area image information. The product tag placement area cutout processing unit 24 may actually cut out as image information, or may virtually cut out as image information without actually cutting out as image information. To virtually cut out the image information means to execute the processing with the specified area, for example, the range of the commodity tag arrangement area as the processing target.

  The product tag placement area unification processing unit 25 executes a keystone correction process of unifying the product tag placement area image information cut out by the product tag placement area cutting processing unit 24. While the surface of the display rack is vertical, the surface of the product tag often faces upward from the vertical surface so as to be easily viewed by the customer. Therefore, the recognition accuracy is improved by normalizing the image information of the product tag placement area. FIG. 11 shows an example of the image information of the commodity tag arrangement region which has been formalized with respect to the image information of the commodity tag arrangement region of FIG. 11 (a) is image information of a product tag placement area obtained by normalizing the image information of the product tag placement area of the upper tray in FIG. 9, and FIG. 11 (b) is a bottom tray in FIG. It is the image information of the goods tag arrangement field which carried out regularization of the image information of the goods tag arrangement field.

  The product tag placement area regularization processing unit 25 performs the following process to regularize the image information of the product tag placement area. That is, in the image information of the product tag placement area, edge detection is performed, and at a location near both ends on the left and right, a contour line close to the vertical with a certain length or more (for example, 70 degrees to 110 degrees; The outline in the angle of a predetermined range from) is specified. In addition, although it is preferable to extract the outline of the location close | similar to the both ends on either side, it does not limit to it. In the case of the image information of the product tag arrangement area of FIG. 9, as shown in FIG. 12, L1 to L4 are specified respectively. 12 (a) is the product tag arrangement area of the upper shelf in FIG. 9, and FIG. 12 (b) is the product tag arrangement area of the lower shelf in FIG. Note that the outlines L1 to L4 to be specified are not actually drawn on the image information of the commodity tag arrangement area. Then, the keystone correction processing is performed on the image information of the product tag placement area so that L1 and L2 in FIG. 12A and L3 and L4 in FIG. 12B respectively become vertical lines. By executing such a process, the image information of the product tag arrangement area can be orthographically obtained, and the image information of the regularized product tag arrangement area shown in FIG. 11 can be obtained. In addition, by executing the processing of the product tag placement area regularization processing unit 25, the accuracy of the product tag identification processing unit 26 and the in-product tag information identification processing unit 27 can be improved, so that processing is executed. Although it is preferable, it can be omitted. In that case, the product tag identification processing unit 26 and the product tag internal information specification processing unit 27 are executed on the product tag placement area cut out by the product tag placement area cutout processing unit 24.

  The product tag identification processing unit 26 identifies the region (product tag region) of each product tag from the normalized product tag arrangement region image information. Mainly two types of methods can be used to specify the product tag area. The first method is a method of identifying the product tag area based on the outline, and the second method is to match the overall characteristics such as the distribution of the whole brightness and darkness with the image information of the product tag template Is a method of specifying a product tag area. Methods other than the first method and the second method can also be used.

  The first method is a method utilizing the fact that the ground color (background color) of the product tag is rich in white (brighter than the background). That is, first, a histogram is generated by integrating the lightness information of the image information in the horizontal direction among the image information of the product tag placement area that has been formalized. Then, the rising and falling positions of the histogram are specified, and the upper side position A and the lower side position B of the product tag are specified. FIG. 13 schematically shows this process. The rising is a place where the black-to-white direction increases sharply (more than a predetermined ratio) in the histogram, and the falling is a place where the white-to-black direction decreases sharply in the histogram .

  Then, out of the image information of the product tag arrangement area which has been formalized, a section between the upper side position A and the lower side position B is cut out, and a histogram is generated by integrating the lightness information of the product tag arrangement area image information which is regularized in the vertical direction. Do. Then, the rising and falling positions of the histogram are identified, and the rising and the falling within a predetermined distance range to the right are paired, and the product tag area is specified as the left side position and the right side position. Then, with regard to the rise where the pair could not be formed, on the right side of the fall where the pair could not be formed, and as the product tag area within a predetermined distance on the left side, the product tag area is specified. FIG. 14 schematically shows this process.

  Further, FIG. 15 shows a state in which the product tag area is specified from the image information of the regularized product tag arrangement area of FIG. 11 by the first method. A rectangular area constituted by each of the upper side position A, the lower side position B, the left side position (rising) U, and the right side position (falling) D is an area of the identified product tag.

  The second method is so-called template matching. That is, the product tag area is specified by registering image information of a product tag in advance as a template, and matching the template with the image information of the product tag arrangement area that has been formalized.

  The product tags include tax-excluded prices, tax-included prices, product identification information (such as product names), manufacturer names, ratings, and the like. Therefore, if the image information of the product tag that is the template includes the product identification information such as the product name and the specific figures of the price and characters, including that part will be the judgment target of the image matching process, so that part will be mosaicked It is preferable to neutralize or exclude from the judgment object by deleting or the like. Neutralization means not giving high or low points for any input, and exclusion means excluding that part from matching in the image matching process.

  The product tag internal information specification processing unit 27 executes processing for specifying the information described in the product tag area specified by the product tag specification processing unit 26 by OCR recognition or the like. When performing OCR recognition, it can carry out about all or one part of the field specified as a goods tag field. If the process is performed on all the areas specified as the product tag area, there is a high possibility that false recognition may occur due to noise or the like, so it is preferable to limit the target area for OCR recognition. In this case, processing corresponding to the first method and the second method used by the product tag identification processing unit 26 is executed.

  When the first method is used in the product tag identification processing unit 26, first, binarization processing is performed in the identified product tag area. Then, labeling processing is performed on the binarized image information. The labeling process is a process of converting consecutive pixels into one island (group) by executing a process of allocating the same number (identification information) to pixels in which white or black is continuous in binary image information. is there. Then, a rectangular area (box) including islands detected by the labeling processing is generated, and the height, width, and baseline of the box are obtained. When generating a box, it is preferable, but not limited to, to generate a rectangle surrounded by the smallest vertical and horizontal line segments that encloses the areas labeled with the same number. When generating a box, an island that does not satisfy the height and width, which is a predetermined threshold value, is not generated as a noise as a box, and is removed as it is from the processing target. For example, an island whose height is too small may have horizontal ruled lines or dust on an image, and an island whose width is too wide may be a logo or the like, which are removed as noise.

  The characters used in product tags are generally in bold, such as gothic. Therefore, even if the image information has some blurring, the character group forming one character string can be detected as an island whose height is aligned with the baseline.

  Then, the in-product tag information identification processing unit 27 merges adjacent boxes having a predetermined similarity to form a block. That is, a block is formed by merging consecutive boxes whose baseline and height coincide within a predetermined range and whose height and width are within a certain threshold. At this time, small boxes and the like between the blocks to be merged are also collectively configured as one block. As a result, a cloud point, a cloud point, a hyphen, etc. will also be taken into one block. A block is an area targeted for OCR recognition. Then, among the blocks, the block with the highest height is estimated as a price area (excluding tax price area), and OCR recognition is performed. In addition, OCR recognition is similarly performed on other blocks. By executing the above-described processing, it is possible to execute character recognition processing more accurately than performing OCR recognition corresponding to a document freely laid out over a plurality of lines in the product tag area. FIGS. 16 to 18 schematically show this process. FIG. 16 shows binarized image information, and FIG. 17 shows a state in which a box (an area indicated by a broken line) is generated. FIG. 18 is a diagram showing a state in which a block (an area indicated by a broken line) is generated.

  As described above, the in-product tag information specification processing unit 27 can perform character recognition on the information described in the product tag.

  When the second method is used in the product tag identification processing unit 26, product identification information such as tax excluding price, tax included price, manufacturer name, product name, and rating are described in advance in the image information of the product tag as a template The position and size (height and width) of each character frame are set. Therefore, the image information of the corresponding part is cut out from the product tag area specified by the product tag specification processing unit 26, and the OCR recognition processing is executed. At this time, by defining product identification information such as price, manufacturer name, product name, etc., and character type (for example, numbers, Roman characters, symbols, character strings, etc.) used by rating as constraint conditions, Accuracy can be improved.

  Further, the in-product tag information identification processing unit 27 executes a process of confirming the integrity of the read information. As consistency check processing, it is preferable to perform two types of check processing of consistency by dictionary collation and logical check.

  The check process of consistency by dictionary collation is executed as follows, for example. The image recognition system 1 stores a product dictionary (not shown) that associates product identification information such as the product name of a product that may be displayed on the display shelf with code information (for example, JAN code) corresponding thereto. )). Then, the edit distance between the character string other than the character string read from the area indicating the price recognized by the in-product tag information specification processing unit 27 and the product identification information such as all product names registered in the product dictionary Find the distance). Then, if the minimum editing distance among the calculated editing distances is one, product identification information such as the product name of the editing distance is determined as the final candidate. Then, an allowable edit distance is determined in advance with respect to the length of the character string of the product identification information such as the product name which is the final candidate, and if within the allowable edit distance, the product identification information such as the product name is identified. Do. If the allowable edit distance is exceeded, the read character string is considered as undetermined. Also, if there are multiple minimum edit distances, the read character string is considered as undetermined.

  Editing distance is a kind of distance indicating how different two character strings are. Specifically, one character is converted to the other character string by inserting, deleting or replacing one character. The minimum number of steps required to do this. FIG. 19 shows an example of the product name with the minimum edit distance among the product names stored in the product dictionary and the product name stored in the product dictionary as a result of the OCR recognition and the product name described in the product tag. Further, FIG. 20 shows a table of correspondence relationships of edit distances which may be determined for each number of characters of the final candidate. Although the case of processing using the editing distance will be described in this specification, it may be a distance function obtained by performing processing for shortening the replacement distance in calculating the editing distance. This distance function is a function that calculates the distance indicating how different the two character strings are, and includes the above-mentioned editing distance.

  Then, after the product identification information such as the product name with the minimum editing distance is determined, a rating dictionary (shown in FIG. 6) is separately provided for the character string excluding the corresponding part of the product identification information such as the product name from the OCR recognition result. The edit distance to each rating stored in (a) and the edit distance to each maker name stored in a maker name dictionary (not shown) are respectively determined. For example, in the example of "350ml" in Fig. 19, "350ml" in the rated dictionary corresponds to "350ml" at the editing distance 0, and it is identified that it is the character string of the rated part. Similarly, the edit distance is determined for the maker name, and the character string at the minimum edit distance is identified as the maker name. Then, remove each character string of rated part and maker name part from the OCR recognition result and find the character string with the shortest editing distance in the product name dictionary for the remaining character string removed, is it an acceptable editing distance? It is determined whether or not. If it is within the allowable edit distance, the product identification information such as the product name, the manufacturer name, and the rating are determined. By performing such processing, it is possible to appropriately determine even when the product identification information such as the product name includes the rating and the manufacturer name. The rating dictionary is a dictionary that stores ratings (such as capacity) of products that may be displayed on the display shelf. The maker name dictionary is a dictionary that stores maker names of products that may be displayed on the display shelf.

  When displaying a recognition result for the user to display for selection, it is preferable that the confirmed character string and the unconfirmed character string be displayed by a method which can identify each. For example, the color of the confirmed character string and the unconfirmed character string may be separated, or the confirmed character string and / or the unconfirmed character string may be attached with information indicating fixed or unfixed. As for the undecided character string, when there are a plurality of candidates for product identification information such as a product name having the minimum editing distance, each candidate may be displayed.

  The logical check (determination process of the logical consistency of the recognized information) in the product tag internal information specification processing unit 27 can be performed as follows. For example, if two prices are read as the price area, one is the tax-excluded price and one is the tax-included price, but if the tax-exclusive price is higher than the tax-included price, they are reversed and recognized Let In addition, if the value obtained by multiplying the consumption tax rate by the tax-excluded price does not match the tax-included price, it is assumed that either or both have an erroneous recognition. Furthermore, when the product identification information such as the product name is recognized, it is determined whether or not it falls within the normal price range of the product or the product category. Further, it may be determined whether the correspondence relationship of the product identification information such as the product name, the maker name, and the rating match.

  As described above, by checking the consistency of the information in the product tag, it is possible to determine the information included in the product tag shown in the photographed image information. The information determined in this way can be output, for example, in tabular form.

  Next, an example of the processing process of the image recognition system 1 of the present invention will be described using the flowcharts of FIG. 3 to FIG.

  The photographed image information in which the display shelf of the store is photographed is inputted from the photographed image information input terminal 4 and the photographed image information input acceptance processing unit 20 of the management terminal 2 receives the input (S100). 21 and 22 show an example of the photographed image information. In addition, the input of the photographing date and time, the shop identification information, and the image information identification information of the photographed image information is received. Then, the photographed image information input acceptance processing unit 20 associates the photographed image information, the photographing date and time, the store identification information, and the image information identification information of the photographed image information, which have received the input, and stores them in the photographed image information storage unit 21.

  When the management terminal 2 receives a predetermined operation input, the orthographic image information regularization processing unit extracts the photographed image information stored in the photographed image information storage unit 21 and is a shelf which is a vertex for performing the keystone correction processing The input of four points of the position (the position of the display shelf) is received, and the keystone correction process is executed (S110). FIGS. 7 and 8 show an example of the photographed image information (right-handed image information) for which the keystone correction processing has been executed in this manner.

  Then, the position specifying processing unit 23 specifies the product tag arrangement area by receiving a predetermined operation input in the management terminal 2 with respect to the directly-placed image information (S120). That is, the input of the product tag arrangement area in the orthographic image information is accepted. FIG. 9 and FIG. 10 are diagrams showing a state in which the product tag placement area is specified. Then, the product tag placement area cutout processing unit 24 cuts out the image information of the product tag placement area specified in S120 (S130), and the product tag placement area regularization processing unit 25 executes the keystone correction processing to obtain the product. The orthographic process is performed on the image information of the tag placement area (S140).

  As a regularization process for the image information of the product tag arrangement area, first, edge detection is performed on the image information of the product tag arrangement area. Then, among the detected edges, an outline having a predetermined length or more and within an angle of a predetermined range from the vertical is specified (S200). Then, among the identified contour lines, the contour lines located at the left and right are identified (S210). An example of the outline specified in this way is shown in FIG. Then, the keystone correction process is performed on the image information of the product tag placement area so that the identified outlines (L1 and L2 in FIG. 12A and L3 and L4 in FIG. 12B respectively) become vertical lines. (S220). By executing such a process, the image information of the product tag arrangement area can be orthographically obtained, and the image information of the regularized product tag arrangement area shown in FIG. 11 can be obtained.

  When the formalization process of the image information of the product tag arrangement area is completed in S140, the product tag identification processing unit 26 determines the individual product tags from the image information of the product tag arrangement area by the first method or the second method. An area is identified (S150).

  In the case of the first method, a histogram is generated by integrating the lightness information of the image information in the horizontal direction among the image information of the regularized product tag placement area (S300), and the rising and falling positions of the histogram are Identify. Then, the rising edge of the identified histogram is specified as the upper side position A of the product tag, and the falling position of the histogram is specified as the lower side position B of the product tag (S310).

  Next, a histogram obtained by vertically extracting the lightness information of the image information of the product tag placement area cut out between the upper side position A and the lower side position B out of the image information of the product tag placement area turned out to be regularized Are generated (S320).

  The rising position U and the falling position D are specified in the generated histogram, and the rising position U (left side position) and the falling position D (right side position) within a predetermined distance range to the right thereof are specified as a pair. It specifies as a field (S330).

  For the rising position U where the pair could not be formed on the right side, for the falling position D where the pair could not be formed on the left side, if there is no product tag area within a predetermined distance, specification as a product tag area (S340).

  FIG. 15 shows a state in which the respective product tag areas are specified by the above processing.

  Further, when using the second method, the product tag identification processing unit 26 performs image matching processing between the image information of the product tag of the template registered in advance and the image information of the emplaced product tag arrangement area. The product tag area is specified by execution.

  As described above, when the product tag identification processing unit 26 identifies the product tag area, the in-product tag information identification processing unit 27 identifies information in the product tag (S160).

  The first method in the product tag internal information specification processing unit 27 binarizes the specified product tag area to obtain binarized image information (S400). Then, a box is specified by the labeling process on the binarized image information (S410). In addition, the island which does not satisfy height and width used as a predetermined threshold does not generate a box as noise, but removes it from processing object as it is.

  Then, by determining the height, width, and baseline of the generated box, and identifying adjacent boxes in which the baseline and height coincide within a predetermined range and the height and width are within a constant threshold range, The identified boxes are merged to form a block (S420). Then, among the blocks, the block with the highest height is estimated as a price area (excluding tax price area), and OCR recognition is performed (S430). As a result of the OCR recognition, if the price information can not be obtained (if the character string of the numerical value can not be recognized) (S440), the block with the next highest height is estimated as the price area (tax excluding price area). Recognize.

  In addition, in the OCR recognition of S430, the reading accuracy of the price can be improved by adding the character recognition constituting the price display such as "number" and "comma" used for the price as a constraint condition.

  When price information is acquired from the block estimated as the price area (tax excluded price area) by OCR recognition (S440), the block other than the price information acquired by estimating as the price area (tax excluded price area) in S430 is selected. The identification is performed (S450), and the OCR recognition is performed on each identified block (S460). It is preferable to perform two types of OCR recognition processing for this OCR recognition. That is, ordinary standard OCR recognition processing and OCR recognition processing in which character recognition constituting price display is added as a constraint condition.

  The blocks identified in S450 include a block of a price area (tax-included price area), a block of an area of information such as product identification information such as a manufacturer name and a product name, and a rating. Then, two types of OCR recognition processing are executed for each block. For a block of an area of information such as product identification information such as manufacturer name and product name, rating, etc., a character string is recognized in the standard OCR recognition processing, and many characters include an error in the restricted OCR recognition processing. It becomes a column. In this case, it is determined whether or not the recognition results of the two recognition processes differ by a predetermined value or more. If they differ, the character string executed in the standard OCR recognition process is regarded as the recognition result, and the price area (tax included price area) It is determined that the block is not a block of On the other hand, for blocks in the price area (tax-included price area), a standard OCR recognition process recognizes a character string of price information, and a character string of price information is recognized even in an OCR recognition process to which constraints are attached. In this case, it is determined whether or not the recognition results of the two recognition processes differ by a predetermined value or more, and if not, they are determined to be blocks in the price area (tax-included price area). A string is used as the price information of the recognition result.

  As described above, the in-product tag information specification processing unit 27 can perform character recognition on the information described in the product tag.

  In the case where the second method is used in the product tag identification processing unit 26, the product tag as a template is pre-taxed, tax-included price, manufacturer's name, product identification information such as product name, etc. The position and size of the character frame to be written are set. Therefore, the image information of the corresponding part may be cut out from the product tag area specified by the product tag specification processing unit 26 and the OCR recognition processing may be executed.

  Then, the in-product tag information specification processing unit 27 executes a dictionary matching process with the specified product name and the like (S470). That is, the edit distance between the read character string and the product identification information such as each product name in the product dictionary is determined, the product identification information such as the product name with the minimum edit distance is specified, and it is within a predetermined distance determined in advance. If it is, it identifies as goods identification information, such as a goods name. Then, the product name portion of the shortest editing distance is removed from the read character string, and for the remaining partial character strings, the editing distance to each rating in the rated dictionary is determined, and it is determined whether the minimum editing distance is within the predetermined distance. If it is within the predetermined distance, the part is identified as a rated character string. Similarly, the product name part and the rated part of the minimum editing distance are removed from the read character string, and for the remaining partial strings, the editing distance with each maker name in the maker name dictionary is determined, and the minimum editing distance is specified. It is determined whether it is within the distance, and if it is within the predetermined distance, the part is identified as a character string of the maker name.

  Further, the in-product tag information identification processing unit 27 executes processing of logical check on the character string in which the character is recognized (S480). That is, it is determined whether or not the character string in which the character is recognized logically contradicts.

  As a result of the logical check, if there is no contradiction, identify the product identification information such as tax-excluded price, tax-included price, and product name, manufacturer and rating for each recognized character string, identify them, shooting date and time, store identification Information and image information identification information of captured image information are stored and output in a predetermined storage area in association with each other. For example, output in tabular format.

  In the first embodiment, the case where the information described in the product tag is read out from the photographed image information has been described, but it may be further correlated with the image recognition of the product displayed on the display shelf. That is, the product identification information such as the product name of the product to be displayed on the display shelf may be specified also from the information described in the product tag, and they may be collated.

  In this case, the image recognition system 1 of the first embodiment constitutes part of the functions of the image recognition system 1 of the second embodiment. The image recognition system 1 in the second embodiment includes a photographed image information input acceptance processing unit 20, a photographed image information storage unit 21, a photographed image information storage unit 21, a photographed image information regularization processing unit 22, and a position specifying process. Unit 23, product tag recognition processing unit 28, display product recognition processing unit 29, sample image information storage unit 30, product identification information storage unit 31, consistency determination processing unit 32, recognition result verification processing unit 33 And. An example of a system configuration | structure of the image recognition system 1 in Example 2 is shown in FIG. Further, FIG. 24 shows an example of the configuration of the product tag recognition processing unit 28, and FIG. 25 shows an example of the configuration of the display product recognition processing unit 29.

  The management terminal 2 in the image recognition system 1 can transmit and receive information to and from the captured image information input terminal 4 via a network.

  The photographed image information input acceptance processing unit 20, the photographed image information storage unit 21, and the photographed image information regularization processing unit 22 are the same as in the first embodiment.

  In addition to the function of the position identification processing unit 23 in the first embodiment, the position identification processing unit 23 identifies a tray area (shelf area). That is, of the display racks shown in the photographed image information and the right-placed image information, there are a shelf area where products are displayed and a product tag arrangement area where product tags for products displayed there are attached . Therefore, the product tag placement area and the shelf area are specified from the orthographic image information. The operator of the management terminal 2 may manually designate the product tag placement area and the shelf area as the specification of the product tag placement area and the tray area, and the position specification processing unit 23 may receive it. The product tag placement area and the shelf area may be automatically specified for the second and subsequent times based on the information on the product tag placement area and the shelf area which have been manually input.

  The product tag recognition processing unit 28 executes a product tag recognition process. That is, each process of the product tag placement area cutout processing unit 24, the product tag placement area regularization processing unit 25, the product tag identification processing unit 26, and the product tag internal information identification processing unit 27 in the image recognition system 1 of the first embodiment. Run. The processing in each processing unit is the same as that of the image recognition system 1 of the first embodiment.

  The display product recognition processing unit 29 executes a process of recognizing a product displayed on the shelf in the display shelf reflected in the photographed image information. The display product recognition processing unit 29 has a shelf area extraction processing unit 291, a face identification processing unit 292, a product identification information identification processing unit 293, and a shelf image matching processing unit 294.

  The tray area extraction processing unit 291 cuts out the image information of the area of the tray specified by the position specification processing unit 23 as rack area image information. The tray area extraction processing unit 291 may actually cut out as image information, or may virtually cut out without actually cutting out as image information. In addition, when there are multiple shelf steps in the display shelf, each is cut out as shelf step area image information.

  The face identification processing unit 292 identifies, for each tray in the tray area in the face-to-face image information, the face on which the product is placed (the area in which the product is placed). The face identification processing unit 292 is divided into an identification process of the first face and an identification process of the second and subsequent faces.

  The first face identification process in the face identification processing unit 292 is an area in which a product is placed within a range (preferably a rectangular area) formed by the coordinates of the trays identified by the position identification processing unit 23 Identify the face). Specifically, the division is performed based on restrictions such as specifying narrow and narrow shadows that occur between products and products, specifying repetitive images of images, specifying steps on the upper side of a package, and identical product widths. Identify the area of the face by specifying the position, etc. As the identification process of the face, an arbitrary method can be adopted depending on the category of the product or the form of the product, and the method is not limited to the above. In addition, correction input by the person in charge may be accepted for the face that is automatically identified. Furthermore, the input of the position of the face may be accepted from the person in charge. The coordinates of the area constituting the identified face are the photographing date and time information, the store information, the image information identification information of the photographed image information, the image identification information of the normal image information, and the face at the coordinates of the face area in the orthographic image information. It manages in association with face identification information for identification. The coordinates indicating the area of the face are the coordinates of the vertices required to specify the rectangular area, and may be, for example, the coordinates of four points, or two points of upper right and lower left, upper left and lower right.

  The second and subsequent face identification processes in the face identification processing unit 292 are performed this time on the coordinates of the area of the face identified in the previous (N-1 th) orthographic image information for the same shelf row of the same display shelf The coordinates of the area of the face specified in the (N-th) orthographic image information are used.

  The coordinates of the area of the face are relative coordinates based on a predetermined place (for example, the upper left vertex C1 of the display shelf) in the image information such as the display shelf in the orthographic image information, as in the coordinate of the position of the tray.

  The commodity identification information identification processing unit 293 identifies commodity identification information of the commodity displayed on the face for each shelf of the display shelf. As the product identification information, in addition to the product name, there is a JAN code assigned to the product, but it is not limited thereto. Any information that can identify the product may be used.

  The product identification information identification processing unit 293 executes the following processing. That is, the product identification information of the product displayed on the face is specified by matching the image information of the face with the sample image information of the product stored in the sample image information storage unit 30 described later for each face. Specifically, first, the similarity between the image information of the area constituted by the coordinates of the face to be processed and the sample image information stored in the sample image information storage unit 30 is determined, and the similarity is highest The product identification information corresponding to the sample image information is specified as the product identification information of the product displayed on the face constituted by the coordinates.

  Here, in order to determine the similarity between the face image information and the sample image information, the following process is performed. First, in the processing before the product identification information identification processing in the product identification information identification processing unit 293, the direction of the image information of the area of the face in the tray of the orthographic image information and the sample image information is the same (rollover or And the size of each image information is approximately the same (when the size of the image information is different in a predetermined range or more, each of them is determined before the similarity determination. The size of the image information is adjusted so as to be within a predetermined range). Then, the similarity between the image information of the face and the sample image information is the image feature amount of the sample image information (for example, the local feature amount) based on the image feature amount (for example, the local feature amount) of the image information of the face Each feature point based on) is extracted. Then, a pair having the highest similarity is detected between the feature point of the image information of the face and the feature point of the sample image information, and the difference in the coordinates of the corresponding point is determined. Then, the average value of the differences is obtained. The average value of the differences indicates the overall average movement amount of the image information of the face and the sample image information. Then, the coordinate difference of all feature point pairs is compared with the average coordinate difference, and the pair with a large degree of deviation is excluded. Then, the similarity is ranked by the number of corresponding points remaining.

  The similarity between the face image information and the sample image information can be calculated by the above method. In addition, in order to improve the accuracy, EMD (Earth Moves Distance) between color histograms may be obtained as a measure of similarity. This makes it possible to compare relatively strong similarity to environmental changes such as brightness information of captured image information, and to specify with high accuracy. The determination of the similarity is not limited to the above. The identified product identification information is associated with the shooting date and time information, the store information, the image information identification information of the captured image information, the image identification information of the normal image information, the face identification information for identifying the face, and the product identification information storage unit Store in 31

  The product identification information specified as described above is associated with the shooting date and time information, the store information, the image information identification information of the photographed image information, the image identification information of the justification image information, and the face identification information for identifying the face. It is stored in the product identification information storage unit 31.

  The tray image matching processing unit 294 sets the image information of the region of the tray in the previous (N−1) -th emplacement image information and the image information of the region of the tray in the current (N th) orthographic image information If the similarity is high, the product identification information of each face in the tray is determined to be identical. As described above, this similarity determination process is performed on the image feature amount of the image information of the area of the tray in the previous (N-1st) orthographic image information and the current (Nth) orthographic image information. The determination of the similarity based on the image information of the area of the tray may be used, or the EMD of color histograms may be used. Moreover, it does not limit to them. Then, instead of the identification processing for each face unit in the product identification information identification processing unit 293, the product identification information identification processing unit 293 determines the product identification information of each face at that tray in the N-th justification image information. The item identification information storage unit 31 stores the item identification information as the same as the item identification information of each face in the first same tray. As a result, it is possible to omit processing for trays with few changes, such as trays with little product movement or trays that are managed with extremely short cycles. The processing by the tray image matching unit 294 may not be provided.

  The sample image information storage unit 30 stores sample image information for identifying which item the item of each face on the shelf of the display shelf is included in the regular image information. The sample image information is image information obtained by photographing a product that is likely to be displayed on the display shelf from a plurality of angles such as up, down, left, right, and diagonally. FIG. 28 shows an example of sample image information stored in the sample image information storage unit 30. Although FIG. 28 shows a case where canned beer is photographed from various angles as sample image information, the present invention is not limited to canned beer. The sample image information storage unit 30 associates and stores sample image information and product identification information.

  In the sample image information storage unit 30, information necessary for calculation of similarity extracted from sample image information together with sample image information or in place of sample image information, for example, an image feature amount and its position The information on the pair may be stored. It is assumed that the sample image information also includes information required to calculate the similarity. In this case, the product identification information identification processing unit 293 does not have to calculate the image feature amount for each sample image information when performing matching processing between the image information of the area of the face and the sample image information. Time can be shortened.

  The commodity identification information storage unit 31 stores commodity identification information of the commodity displayed on each face of the shelf of the display shelf. For example, in association with product identification information, shooting date and time information, store information, image information identification information of captured image information, image identification information of normal image information, face identification information for identifying a face, and product identification It is stored in the information storage unit 31.

  The consistency determination processing unit 32 performs each shelf or each shelf for the recognition result of the information displayed on the product tag by the product tag recognition processing unit 28 and the recognition result of the product (product identification information) by the display product recognition processing unit 29. Determine the integrity of items that are likely to be included in the column. For example, in the recognition result of the information displayed on the product tag or the recognition result of the product, the product is recognized as a product rated at 350 ml, but 500 ml of the product is displayed on the display shelf or tray of the product If it is determined, change the rating of the same brand name to "500 ml". It is preferable that items to be placed on the display rack or the shelf are set in advance and correspond to the photographed image information stored in the photographed image information storage unit 21.

  The recognition result collation processing unit 33 matches the product identification information of the product for each face recognized in the display product recognition processing unit 29 with the information (product identification information) of the product recognized in the product tag recognition processing unit 28, and the recognition result Check whether the matches.

  Specifically, first, as a result of the recognition processing by the display product recognition processing unit 29, it is specified as a group the sections in which the faces with high similarity are lined up, and how many groups are present in one tray. Also, identify the left and right positions of the trays of each group. Then, the information on the recognition result of the product tag by the product tag recognition processing unit 28 in which the left and right positions coincide with each group is associated with each group.

  If the group by the face is associated with the product tag, the following processing is performed. First, as a result of recognition of a product tag by the product tag recognition processing unit 28, the read product name (product identification information) is converted into a candidate product list with a likelihood. In this case, conversion is performed by associating the probability of being the product A1 with p1 and the probability of being the product A2 with the probability such as p2. If the manufacturer's name or rating information is obtained from the recognition result of the product tag, the likelihood of the product of the manufacturer and the product having that rating is set high. When price information is obtained from the recognition result of the product tag, the likelihood of the product having the price range as the selling price is increased. Furthermore, when the two prices of tax-excluded price and tax-included price are read, and the ratio between them is in agreement with the logical check, such as just matching the presence or absence of the consumption tax, the likelihood is further increased, Set high. In addition, at this time, if the genre of the product displayed on the shelf of the display shelf is known, the likelihood of the product belonging to the genre is set to be high or low.

  And about the information of the goods recognized as a result of recognition by the display goods recognition process part 29, the candidate goods list with a likelihood is similarly provided based on the grade of image similarity. In this case, the probability of being the product B1 is converted in such a manner that the probability of being the product Pb1 and the product B2 is made to correspond to the probability such as Pb2.

  Then, the list of candidate products such as products A1 and A2 according to the recognition result of the product tag is compared with the list of candidate products such as products B1 and B2 according to the recognition result of the display product recognition processing unit 29. Classification (case 1), only in group A (case 2), or only in group B (case 3), and for case 1, the likelihood of products in groups A and B respectively Combined, with a higher likelihood than either. In addition, for Case 2 and Case 3, a combination function is applied that reflects the agreement between the accuracy of the product tag recognition result by the product tag recognition processing unit 28 and the accuracy of the product information recognition result by the display product recognition processing unit 29. By doing this, the final candidate product list with likelihood is generated.

  As described above, by generating the final candidate product list with likelihood, it is possible to rank and specify candidate products, so for example, the product that is the candidate for the top (first) candidate is a product The determination may be made, or a product which is a candidate from the first place to a predetermined order may be displayed, and a selection input of a visual judgment result may be received.

  Then, for the determined product, the likelihood of the read price is calculated again. That is, the ratio of tax-excluded price to tax-included price, whether it is within the selling price range of the product, or consistency or divergence with the frequent price of the product is determined, and the likelihood of the price is determined. Then, if the likelihood is higher than a predetermined threshold value, the price may be automatically determined, and if lower, the result may be reflected on the display to receive an input by selection.

  Next, an example of the processing process of the image recognition system 1 according to the second embodiment will be described with reference to the flowchart of FIG. The same processes as in the first embodiment will not be described.

  The photographed image information (FIG. 21, FIG. 22) obtained by photographing the display shelf of the store is inputted from the photographed image information input terminal 4, and the photographed image information input acceptance processing unit 20 of the management terminal 2 receives the input (S100) . In addition, the input of the photographing date and time, the shop identification information, and the image information identification information of the photographed image information is received. Then, the photographed image information input acceptance processing unit 20 associates the photographed image information, the photographing date and time, the store identification information, and the image information identification information of the photographed image information, which have received the input, and stores them in the photographed image information storage unit 21.

  When a predetermined operation input is received in the management terminal 2, the photographed image information regularization processing unit 22 extracts the photographed image information stored in the photographed image information storage unit 21 and is a shelf which is a vertex for performing the keystone correction processing The input of four points of the position (the position of the display shelf) is received, and the keystone correction process is executed (S110). FIGS. 7 and 8 show an example of photographed image information (right-handed image information) for which the keystone correction processing has been performed in this manner.

  Then, the position specifying processing unit 23 specifies the rack area and the product tag arrangement area by receiving a predetermined operation input in the management terminal 2 with respect to the emplacement image information (S120). That is, the input of the shelf area and the product tag arrangement area in the orthographic image information is received. FIG. 29 and FIG. 30 are diagrams showing a state in which the tray area and the commodity tag arrangement area are specified.

  As described above, when the shelf area and the product tag arrangement area are specified, the display product recognition processing unit 29 recognizes the display product in the shelf area and the product tag recognition process in the product tag arrangement area Each unit 28 executes. Note that the recognition processing of the display commodity in the display commodity recognition processing unit 29 and the commodity tag recognition processing by the commodity tag recognition processing unit 28 may be performed in parallel or at different timings.

  The commodity tag recognition processing (S130 to S160) in the commodity tag recognition processing unit 28 is the same as that of the first embodiment. That is, the product tag placement area cutout processing unit 24 in the product tag recognition processing unit 28 cuts out the image information of the product tag placement area specified in S120 (S130), and the product tag placement area regularization processing unit 25 performs trapezoidal correction processing. Is executed to execute the regularization process (S140).

  When the product tag specification processing unit 28 in the product tag recognition processing unit 28 identifies individual product tag regions from the image information of the product tag arrangement region that has been made regularized, when the product tag recognition processing unit 28 completes the regularization process of the product tag arrangement region To do (S150).

  When the product tag identification processing unit 26 identifies the product tag area, the in-product tag information identification processing unit 27 identifies the information in the product tag (S160). By this specification, it is possible to character-recognize information described in the product tag, for example, information such as tax-excluded price, tax-included price, product name (product identification information), rating and the like.

  Next, recognition processing of a display commodity by the display commodity recognition processing unit 29 will be described.

  The tray area extraction processing unit 291 cuts out the image information of the tray area from the fixed image information based on the area of the tray for which the input is received in S120 (S170). Then, the process of specifying the face is executed for each tray in the tray area image information (S180). Specifically, for a tray in a tray area, a narrow and narrow shadow that occurs between goods and goods is specified within the range of a rectangular area formed by four coordinates, and a repetitive pattern of an image is specified. The face is specified by specifying a step on the upper side of the package, specifying a dividing position based on constraints such as the same product width, and the like. The identified face is attached with face identification information for identifying the face. Then, the coordinates of each identified face are stored in association with the shooting date, store identification information, image information identification information of the captured image information, image information identification information of the direct image information, and face identification information for identifying the face. Let The coordinates of the face may be two points that can specify a rectangular area without storing four points.

  As described above, when each face of each tray in the tray position area image information of the emplacement image information is specified, the product identification information identification processing unit 293 stores the sample image stored in the sample image information storage unit 30 for each face. Information and matching processing are executed, and the product identification information of the product displayed on the face is specified (S190). That is, from the image information in the rectangular area of the face of a certain tray (the face identification information of the face of this area is X) and each sample image information stored in the sample image information storage unit 30, each image feature amount The similarity is determined by calculating and calculating the feature point pairs. Then, the sample image information having the highest similarity is specified, and if the similarity at that time is equal to or more than a predetermined threshold value, the product identification information corresponding to the sample image information is based on the sample image information storage unit 30 Identify. Then, the specified product identification information is used as the product identification information of the product displayed on the face of the face identification information X. Then, the product identification information identification processing unit 293 associates the specified product identification information with the shooting date and time, the store identification information, the image information identification information of the captured image information, the image information identification information of the direct image information, and the face identification information. It stores in the commodity identification information storage unit 31 (S200).

  In addition, it is not always possible to specify the product identification information of all faces. Therefore, for faces that can not be identified, the input of product identification information is accepted, and the product identification information for which the input is accepted is the date and time of shooting, store identification information, image information identification information of captured image information, and image information of frontal image information The item identification information and the face identification information are stored in the item identification information storage unit 31 in association with each other. Similarly, the input may be accepted for the correction process of the specified product identification information.

  By performing the process as described above, it is possible to specify the product identification information of the product displayed on the shelf of the display shelf which is reflected in the photographed image information.

  Thus, as a result of recognition of the product tag by the product tag recognition processing unit 28, as a result of recognition of the display product by the display product recognition processing unit 29, there is a possibility that the consistency determination processing unit 32 is included in each shelf and each shelf It is determined whether the product is high or not (S210). For example, in the recognition result of the product tag or the recognition result of the product, although the product is recognized as a product whose rating is 350 ml, it is determined that 500 ml of the product is displayed on the display shelf or tray of the product In the case, change the rating of the same brand name to "500 ml".

  Further, the recognition result collation processing unit 33 collates the product identification information of the product for each face recognized in the display product recognition processing unit 29 with the information (product identification information) of the product recognized in the product tag recognition processing unit 28, It is collated whether the recognition results match (S220).

  That is, as a result of the recognition processing by the display product recognition processing unit 29, the recognition result matching processing unit 33 specifies, as a group, sections where faces with high similarity are arranged, how many groups are present in one tray. Also, identify the left and right positions of the trays of each group. Then, the information on the recognition result of the product tag by the product tag recognition processing unit 28 in which the left and right positions coincide with each other is associated with the group.

  As described above, product identification information (product name) corresponding to the face or group to be stored in the product identification information storage unit 31 after association of a group with a product tag by a face and recognition of a product tag by the product tag recognition processing unit 28 The results are compared, the recognition results are identified if they match, and the read price is determined.

  By the above processing, it is possible to compare and collate the result of the recognition of the commodity displayed on the display shelf by the image recognition processing with the result of the recognition of the character by the commodity tag.

  In the first and second embodiments described above, the keystone correction process is performed by specifying four points. However, it is burdensome to specify and input the reference vertex every time. Therefore, it may be configured to automatically specify a vertex which is a reference of the keystone correction processing. The process in this case will be described.

  The photographed image information regularization processing unit 22 in this case is divided into a first keystone correction process and a second and subsequent keystone correction process. In addition to the first case, the first case also includes the case where the process is performed manually at an arbitrary timing in order to correct a shift in automatically specifying a vertex. The second and subsequent times are other than the first time.

  As in the first embodiment, the first keystone correction processing in the photographed image information regularization processing unit 22 receives an input of designation of four vertices of a rectangular area of the display shelf. The four apexes of the rectangular area of the display rack may be the four apexes of the rack position of the display rack, or the four apexes of the racks or the four apexes of the area to which the product tag is attached. Also, it may be four vertices of a group of two or three trays. Here, the four vertexes whose specification has been accepted are stored in association with the shooting date and time information, the store information, and the image information identification information of the shot image information. Then, based on the coordinates of the four vertexes whose specification has been received, the photographed image information regularization processing unit 22 executes a keystone correction process on the photographed image information, and uses it as orthographic image information.

  It is desirable that the photographed image information be photographed at the same angle in the same area at regular intervals. However, it is not possible to shoot completely the same area at the same angle. Therefore, the photographed image information regularization processing unit 22 executes the second and subsequent keystone correction processing as follows.

  First, the photographed image information regularization processing unit 22 sets the vertex coordinates of the (N-1) th photographed image information obtained by photographing the same (approximately the same) area corresponding to the Nth photographed image information in the previous processing. Identify from stored information. The vertex coordinates of the (N-1) th shot image information obtained by shooting the same (almost the same) area corresponding to the N-th shot image information are the store identification information, the image identification information, the shooting date and time information, etc. Identify based on. Then, for the N−1th captured image information, a rectangular area of a predetermined size including the apex coordinates of the specified four apexes, for example, a square of about 1⁄5 of the width of the tray is selected as the feature amount extraction area 220 Set as. An example of a state in which the feature amount collecting area 220 is set for the (N−1) -th shot image information is shown in FIG. The feature amount extraction area 220 may be a rectangular area including vertex coordinates. On the other hand, if the backgrounds of the display shelves are matched with each other, the backgrounds may be largely deviated only by the slight shift of the photographing position. Therefore, it is preferable to set the feature amount collecting area 220 at a position including the inside of the display shelf as much as possible. That is, it is preferable that the vertex coordinates be located in the feature amount extraction area 220 in the outer direction of the display shelf than the center point of the feature amount extraction area 220.

  For example, four points of vertex coordinates are located at upper left, upper right, lower left, and lower right. Then, when the rectangular area of the feature extraction area 220 is divided into four areas each of which is divided into two at the vertical and horizontal centers, in the feature extraction area 220 including the upper left apex coordinates, the apex coordinates are the upper left area of the rectangular area. The feature amount collecting area 220 is set to be located at Similarly, in the feature amount extraction area 220 including the upper right apex coordinates, the feature amount extraction area 220 is set so that the apex coordinates are located in the upper right area of the rectangular area, and the feature amount extraction including the lower left apex coordinates In the area 220, the feature amount extraction area 220 is set so that the vertex coordinates are located in the lower left area of the rectangular area, and in the feature amount extraction area 220 including the vertex coordinates of the lower right, the vertex coordinates are rectangular areas. The feature value collecting area 220 is set to be located in the lower right area. As a result, the vertex coordinates are located in the feature amount extraction area 220 in the outer direction of the display shelf than the center point of the feature amount extraction area 220.

  Next, the photographed image information regularization processing unit 22 includes, in the Nth photographed image information, the feature amount collection area 220 set in the N-1th photographed image information, and the N−1th photographed image information A feature amount sampling area 221 having a size equal to or larger than the feature quantity sampling area 220 of FIG. The feature value collection area 221 set in the N-th captured image information does not exceed 1/2 the size of the short side. Furthermore, if it goes outside the captured image information, the area is trimmed. An example of a state in which the feature amount collecting area 221 is set for the Nth captured image information is shown in FIG.

  Then, the photographed image information regularization processing unit 22 extracts local feature amounts in each of the feature amount extraction regions 220 set for the (N−1) -th captured image information, and extracts the feature points and their coordinates according to the local feature amounts. Remember the set. In addition, local feature quantities are sampled in each feature quantity sampling area 221 set for N-th captured image information, and feature points and their sets of coordinates according to the local feature quantities are stored.

  The photographed image information regularization processing unit 22 corresponds to the local feature amount of the feature point in the feature amount sampling region 220 of the N-1th shot image information and the feature amount sampling region 220 of the N-1th shot image information The local feature amounts of the feature points in the feature amount collection region 221 of the N-th captured image information located at the same position are compared. Then, the feature point of each local feature amount of the N-th shot image information that is closest to each local feature amount of each feature point of the (N-1) -th shot image information is specified. Then, the feature points of the closest local feature amounts are paired, and the coordinates of the feature points according to the local feature amounts forming a pair are associated. At this time, pairs in which the closeness (similarity) between local feature quantities is less than a predetermined threshold value are excluded. In this way, it is possible to specify a pair of feature points of local feature amounts in the (N−1) -th feature amount extraction region 220 and feature points of nearest local feature amounts in the N-th feature amount extraction region 221. The relationship of the pair of the feature point of the local feature amount of the (N-1) -th feature amount extraction region 220 and the feature point of the local feature amount of the N-th feature amount extraction region 221 is shown in FIG. In FIG. 33, point groups of feature points by local feature amounts in the (N-1) -th feature amount extraction region 220 are A, and point groups of feature points by local feature amounts in the N-th feature amount extraction region 221 are B, N- The vertex used for the first trapezoidal correction process is indicated by C.

  Coordinates of point group A of feature points according to local feature amounts in N−1th feature amount extraction area 220 and point group B of feature points according to local feature amounts in Nth feature amount extraction area 221 corresponding to point group A The function F (affine transformation) for projecting the point group A onto the point group B is obtained based on the coordinates of The function F has methods such as repeating sampling estimation, using RANCAC of OpenCV which is a kind of robust estimation, but is not limited thereto. Note that pairs with large deviations from the relation line in the projection relationship are excluded from the processing targets.

  After obtaining the function F in the photographed image information regularization processing unit 22, the photographed image information normalization processing unit 22 determines the coordinates of the vertex C used in the N−1th keystone correction process based on the function F. It projects onto the Nth captured image information and specifies it as the coordinates of the vertex D for the Nth keystone correction processing. FIG. 34 schematically shows this.

  The above processing is performed on each of the feature amount extraction areas 220 and 221 to specify four vertexes of the shelf position for the keystone correction processing in the N-th captured image information. Then, the photographed image information regularization processing unit 22 executes trapezoidal correction processing on the Nth photographed image information based on the specified four vertices, normalizes it, and generates and stores orthographic image information. At this time, the photographed image information regularization processing unit 22 associates the photographed date information, the store information, the image information identification information of the photographed image information, and the image identification information of the regular image information in association with the fixed image information. I will make them remember. The coordinates of the vertex corresponding to the specified N-th captured image information are stored in association with the captured date / time information, the store information, and the image information identification information of the captured image information.

  The vertex identification process used in the keystone correction process in the captured image information regularization processing unit 22 is not limited to the case where the product is identified from the image information obtained by imaging the display shelf as in the present invention, and the same imaging object The present invention can also be applied to the image recognition system 1 that normalizes a plurality of pieces of image information obtained by photographing and generates the same. As a result, the plurality of pieces of image information obtained by photographing the same object to be photographed can be orthographically converted, and the image to be imaged of the object to be photographed can be generated.

  In addition, in order to specify the four vertices of the shelf position for the keystone correction processing in the N-th captured image information, the feature points by the local feature amount in the feature amount extraction region 220 in the N-1th captured image information are described above. Function F is obtained using point group A of the image point and point group B of feature points according to local feature amounts in the feature amount extraction region 221 in the N-th captured image information, and used in the N-1th trapezoid correction process The process of projecting the coordinates of the vertex C onto the N-th captured image information by the function F and specifying the coordinates as the coordinates of the vertex D for the N-th keystone correction process has been described. However, in such processing, it is sufficient to find the coordinates (position) of corresponding points of similar image information in the (N-1) -th shot image information and the N-th shot image information. Instead, any feature quantity may be used as long as it is a type of feature that identifies a location in image information. For example, there are sharp points in the image information and points of highlight. In the present specification, a feature that specifies a portion in image information, such as a local feature, is referred to as an image feature (position-specific image feature). In the description of the present specification, a case where local feature quantities are used as image feature quantities as described above will be described.

  Next, a process of specifying a vertex for performing the keystone correction process according to the third embodiment will be described. In this case, the apex for performing the keystone correction process has already been identified in the photographed image information obtained by photographing an arbitrary display shelf, and after a predetermined period (for example, one week) has elapsed, similar areas are displayed for the same display shelf. A case will be described in which photographed image information photographed at a similar angle is performed.

  The photographed image information in which the display shelf of the store is photographed is input from the photographed image information input terminal 4, and the photographed image information input acceptance processing unit 20 of the management terminal 2 receives the input. FIG. 35 shows an example of the photographed image information. In addition, the input of the photographing date and time, the shop identification information, and the image information identification information of the photographed image information is received. Then, the photographed image information input acceptance processing unit 20 associates the photographed image information, the photographing date and time, the store identification information, and the image information identification information, which have received the input, and stores them in the photographed image information storage unit 21.

  When a predetermined operation input is received in the management terminal 2, the photographed image information regularization processing unit 22 extracts the photographed image information stored in the photographed image information storage unit 21, and executes shelf correction processing to execute the keystone correction process. A process of specifying the vertex D (D1 to D4) is executed.

  The vertex coordinates (e.g., vertex coordinates C1 to C4) of the previous (N-1) -th shot image information (FIG. 36) obtained by shooting the same or substantially the same region corresponding to the (th) -th shot image information (FIG. 35) To identify The vertex coordinates of the previous captured image information may be specified based on the shooting date, store identification information, image identification information of the captured image information, and the like.

  The photographed image information regularization processing unit 22 extracts the (N-1) th photographed image information from the photographed image information storage unit 21 and a rectangle of a predetermined size including one vertex for each of the vertices C1 to C4. The region is set as the feature amount extraction region 220 in the (N-1) -th shot image information. FIG. 36 shows a state in which the feature amount collecting area 220 is set in the (N-1) th shot image information.

  Further, the photographed image information regularization processing unit 22 extracts the Nth photographed image information (FIG. 35) from the photographed image information storage unit 21, and the features of the range wider than the (N−1) th feature amount collecting region 220 The amount collection area 221 is set to N-th captured image information. FIG. 37 shows a state in which the feature amount collecting area 221 is set in the N-th shot image information. Each feature amount sampling area 221 in the N-th shot image information includes one N-1th feature amount sampling area 220. In FIG. 37, the inclusion relationship is indicated by showing the (N−1) -th feature amount sampling region 220 in the N-th feature amount sampling region 221.

  Then, the photographed image information regularization processing unit 22 extracts local feature amounts in each feature amount collection region 220 set for the N−1th captured image information, and sets a set of feature points and coordinates according to the local feature amounts. Remember. In addition, local feature quantities are sampled in each feature quantity sampling area 221 set for N-th captured image information, and feature points and sets of coordinates according to the local feature quantities are stored.

  The photographed image information regularization processing unit 22 calculates the feature amount sampling area 221 of the N-th shot image information closest to the local feature quantities of the feature points in the feature quantity sampling area 220 of the (N-1) -th shot image information. The feature points of the local feature amount at are identified, and the respective coordinates are associated with each other as the feature point of the local feature amount to be a pair. FIG. 33 shows the relationship of the pair of the (N−1) th N−1th feature amount sampling region 220 and the Nth time feature amount sampling region 221.

  Then, the point group of feature points by the local feature amount in the N-1th feature amount extraction region 220 is A, the point group of the feature point by the local feature amount in the Nth extraction amount feature region 221 is B, N-1 Let C (C1 to C4) be a vertex used in the keystone correction processing of the captured image information regularization processing unit 22 set the point group A to the point group B based on the point group A and the coordinates of the point group B. Find a function F (affine transformation) to project onto.

  Then, the photographed image information regularization processing unit 22 projects the coordinates of the vertex C (C1 to C4) used in the (N-1) th trapezoidal correction processing based on the obtained function F, and the Nth trapezoidal correction processing As the coordinates of the vertex D (D1 to D4) for.

  By performing the above-described process on each of the feature value extraction areas 220 and 221, it is possible to automatically specify four vertices D (D1 to D4) for the Nth keystone correction process. The coordinates of the specified N-th vertex D (D1 to D4) are stored in association with the shooting date and time, the store identification information, and the image information identification information of the captured image information. FIG. 38 shows the identified vertices D1 to D4.

  As described above, when the vertex D (D1 to D4) of the shelf position for the keystone correction process for the N-th captured image information is specified, the captured image information regularization processing unit 22 determines the vertex D (D1 to D4). The keystone correction process is performed on the N-th time captured image information on the basis of.

  By executing the above processing, for the second and subsequent keystone correction processing, it becomes possible to automatically identify the corresponding vertex without specifying four vertices used in the keystone correction processing. Can reduce the burden on

  Further, a modified example of the face identification process in the face identification processing unit 292 of the second embodiment will be described. In this embodiment, the process of specifying the face may be performed automatically as the process of specifying the second and subsequent faces, with the process of specifying the face of the second embodiment as the first process. The process in this case will be described.

  In addition to the first case, the first case also includes the case where the process of the second embodiment is performed at an arbitrary timing in order to correct a deviation in automatic specification. The second and subsequent times are other than the first time.

  The face identification processing unit 292 executes the same process as the process of the second embodiment as an initial face identification process. The second and subsequent face identification processes in the face identification processing unit 292 are the coordinates of the area of the face identified in the previous (N−1) -th emplacement image information for the same shelf of the same display shelf. The coordinates are extracted as the coordinates of the area of the face specified by the present (N-th) orthographic image information.

  The coordinates of the area of the face are relative coordinates based on a predetermined location (for example, the upper left vertex C1 of the display shelf) in the display shelf, like the coordinates of the position of the trays.

  Further, as a modification of the second embodiment, the process of the second embodiment is performed as the process of specifying the product identification information of the product displayed on the face for each shelf of the display shelf in the product identification information specification processing unit 293. The following processing is executed as the identification processing of the product identification information of the above and the identification processing of the commodity identification information of the second and subsequent times.

  The commodity identification information identification processing unit 293 first identifies the face identification information of the face to be processed in the identification processing of the commodity identification information of the face in the N-th direct image information. Let X be the identified face identification information. Then, the image information of the area of the face identification information X in the N-th orthographic image information and the image information of the area in the (N-1) -th orthographic information at the position corresponding to the face identification information X are compared. With regard to the determination of similarity, it is preferable to obtain the EMD of color histograms and use it as a measure of similarity, but it is not limited thereto. If the similarity is equal to or more than a certain threshold value, the product identification information corresponding to the face of the area in the N-1th orthographic image information is extracted from the product identification information storage unit 31, and the Nth orthographic image information Item identification information of the face identification information X in As a result, it is possible to specify the product identification information of the face identification information X in the N-th orthographic image information to be processed. If the similarity is less than a certain threshold, the image information of the area of the face identification information X in the N-th orthographic image information and the sample image information stored in the sample image information storage unit 30 as in the first case. The product identification information having the similarity higher than a predetermined threshold value and the highest similarity is specified as the product identification information of the face of the face identification information X in the N-th orthographic image information.

  In addition, in comparison with the image information of the area of the face in the (N−1) -th orthographic image information, not only the comparison with the position of the corresponding face but also a face in a predetermined range may be included as a comparison target. For example, when comparing with the image information of the area of face identification information X in the N-th orthographic image information, the comparison target includes the area of the face identification information X in the N-1th orthographic image information and the area It is also possible to include the area of the face within a predetermined range, for example, the face located at a position separated by one or more in the left-right direction, and the area of the face located in the upper and lower trays. Furthermore, in addition to the area of the face identification information X in the (N-1) -th time orthographic image information, areas of a plurality of adjacent faces are included as in the face identification information X-2, X-1, X, X + 1, X + 2. May be

  In this case, the image information of the area of the face identification information X in the N-th time orthographic image information, and the image information of each of the area of the range of the face to be compared in the (N-1) -th time orthographic image information To identify the face identification information of the (N-1) -th orthographic image information with the highest similarity. The similarity may be conditional on the similarity being equal to or greater than a certain threshold. Then, the product identification information corresponding to the face identification information of the specified N-1 first emplacement image information is extracted from the product identification information storage unit 31, and the product identification information of the face identification information X in the N th emposiment image information Do. FIG. 27 schematically shows this process. FIG. 27 (a) is the previous (N-1st) orthographic image information, and FIG. 27 (b) is the current (Nth) orthographic image information. Then, compare the image information of the area of each face of tray 1 of the N-th orthographical image information with the image information of the area of each face of tray 1 of the N-1st orthographic image information. Product identification information of the face of tray 1 of the N-1st right emplacement image information with the highest similarity and the product identification information of the face of tray 1 of the n th right emplaced image information Indicates to identify as. FIG. 27 shows the case where the comparison is made with the two right and left faces in addition to the (N-1) th face of the normal position image information at the position corresponding to the Nth direct position image information face. . In addition, comparison may be performed with the image information of the face position of not only the same tray but also the upper and lower trays. For example, in the case of FIG. 27, when specifying the product identification information of the face position of the center of the tray 2 of the N-th orthographic image information, the center of the tray 2 of the N-1-th orthographic image information and its left and right Not only comparison with the image information of the area of two faces each, but also the image information of the area of the center of the tray 1 of the N-1st right emplacement image information and the area of the two faces to the left and right thereof, N-1 Image information and similarity may be compared between the center of the tray 3 of the emplacement image information and the area of the face of each of the left and right two faces.

  When the product identification information can not be specified because the similarity does not satisfy the threshold value or the like as a result of comparison with the image information of the face of the N-1th orthographic image information, the face in the Nth orthographic information The image information of the area of the identification information X and the sample image information stored in the sample image information storage unit 30 are compared, and the product identification information having the similarity equal to or greater than a predetermined threshold and having the highest similarity is N The identification information is specified as the product identification information of the face of the face identification information X in the first emplacement image information. The determination processing of the similarity in this case can be performed in the same manner as the identification processing of the first item identification information.

  The product identification information specified as described above is associated with the shooting date and time information, the store information, the image information identification information of the photographed image information, the image identification information of the justification image information, and the face identification information for identifying the face. It is stored in the product identification information storage unit 31.

  In the first to fifth embodiments of the image recognition system 1 of the present invention, the accuracy is improved by executing a normalization process and the like as preprocessing for image information to be processed by each processing unit. In the above-described embodiments, the case has been described. However, it is not necessary that the orthographic process is necessarily performed on the image information to be processed, and in such a case, the image to be processed in each processing unit is an image when the orthographic process is not performed. It is information. For example, position specification processing unit 23, product tag placement area cutout processing unit 24, product tag placement area regularization processing unit 25, product tag specification processing unit 26, product tag internal information specification processing unit 27, product tag recognition processing unit 28. In the display product recognition processing unit 29, the tray area extraction processing unit 291, the face identification processing unit 292, the product identification information identification processing unit 293, and the tray image matching processing unit 294, the formalization processing is not executed. Image information may be processed. In this case, even if the formalization process is not performed, the process may be performed on the image information in which the display shelf on which the product is displayed is shown. Note that the image information in which the display rack on which the product is displayed includes both the image information subjected to the regularization process and the image information not subjected to the regularization process.

  By using the image recognition system 1 of the present invention, it is possible to accurately identify the goods displayed on the display shelf.

1: Image recognition system 2: Management terminal 4: Photographed image information input terminal 20: Photographed image information input acceptance processing unit 21: Photographed image information storage unit 22: Photographed image information regularization processing unit 23: Position specification processing unit 24: Merchandise tag placement area cutout processing unit 25: Merchandise tag placement area regularization processing unit 26: Merchandise tag specification processing unit 27: Merchandise tag internal information identification processing unit 28: Merchandise tag recognition processing unit 29: Display merchandise recognition processing unit 30 : Sample image information storage unit 31: Product identification information storage unit 32: Consistency determination processing unit 33: Recognition result comparison processing unit 220: Feature quantity collection area 221 in the N-1st image information: Feature in the Nth image information Amount collection area 291: tray stage area cutout processor 292: face identification processor 293: product identification information identification processor 294: tray image matching processor 70: arithmetic unit 71: storage unit 7 : Display device 73: input device 74: communication device

In the first invention, a product tag identification processing unit for specifying a product tag area in image information in which a display shelf for displaying a product is shown, and a product tag specifying processing unit performing OCR recognition processing in the specified product tag area Product tag internal information identification processing unit for identifying the processed information, the product tag identification processing unit identifies the product tag area using the lightness information of the image information, and the product tag internal information The specification processing unit specifies the box by binarizing the specified product tag area and performing labeling processing, and specifies the boxes satisfying the predetermined condition as blocks among the specified boxes, and the specified block the running OCR recognition process, the two price information identified by the OCR recognition process by executing the logical consistency of the determination process with respect to, tax To determine the rating and the price including tax, is an image recognition system.

According to a second aspect of the present invention, there is provided a product tag identification processing unit for specifying a product tag area in image information in which a display shelf for displaying a product is shown, and performing OCR recognition processing on a part or all of the specified product tag areas. The product tag identification processing unit identifies the information described in the product tag in the product tag , and the product tag identification processing unit starts up a histogram obtained by integrating the lightness information of the image information in the horizontal direction, By specifying the falling edge, the upper side position and the lower side position of the product tag are specified, and by specifying the rising edge and falling edge of the histogram obtained by integrating the lightness information of the image information in the vertical direction, within a predetermined distance from the rising edge. This is an image recognition system that specifies the left side position and the right side position of a product tag with a certain falling as a pair .

Similar to the first invention, by using the image recognition system of the present invention, the information written on the product tag can be recognized with high accuracy, so that the displayed product can be specified with high accuracy. it can.
In order to accurately recognize the information described in the product tag, it is necessary to specify the product tag area as accurately as possible. Generally, since the ground color of the product tag is white, when the display shelf is photographed, the part of the product tag is bright. Focusing on this point, as in the present invention, the product tag area can be specified with high accuracy by integrating the lightness information of the image information in the vertical direction and the horizontal direction.

According to a third aspect of the present invention, there is provided a product tag identification processing unit for specifying a product tag area in image information in which display shelves for displaying products are displayed, and performing OCR recognition processing on a part or all of the specified product tag areas. The product tag identification processing unit identifies the information described in the product tag in the product tag, and the product tag identification processing unit is configured to start up a histogram obtained by integrating the lightness information of the image information in the vertical direction. Among them, if there is no tag area within a predetermined distance on the right side of the rising that failed to form a pair, it is identified as a product tag area, and among the fallings of the histogram integrated vertically in the lightness information of the image information, identifying a product tag area if there is no tag area within distance predetermined to the left about the fall could not be formed a pair, image The recognition system.

Similar to the first invention, by using the image recognition system of the present invention, the information written on the product tag can be recognized with high accuracy, so that the displayed product can be specified with high accuracy. it can.
The product tag may be attached partially overlapping. In that case, in the histogram based on the integration of the lightness information, the rise and fall occur continuously. Therefore, by executing the process of the present invention, it is possible to specify a product tag area for a product tag that is attached in an overlapping manner.

The fourth invention is described in a product tag by performing product recognition processing with a product tag identification processing unit for specifying a product tag area in image information in which a display shelf for displaying a product is shown and an identified product tag area. Product tag internal information specification processing unit for specifying the processed information, the product tag internal information specification processing unit executes the binarization processing in the specified product tag area, and performs the binarization By performing labeling processing in the processed image information, consecutive pixels are grouped to specify a box, and adjacent boxes having a predetermined similarity among the specified boxes are merged and specified as a block. The image recognition system is configured to estimate a block having the highest height among the identified blocks as a price area and execute an OCR recognition process .

Similar to the first invention, by using the image recognition system according to these inventions, the information written on the product tag can be recognized with high accuracy, so the products displayed corresponding to it can be specified with high accuracy. Can.
In order to recognize the information described in the product tag, OCR recognition processing may be executed for all of the product tag areas, but in that case a large amount of noise etc. will be detected and the accuracy will deteriorate. Connect. Moreover, processing time also increases. Therefore, it is preferable to execute the OCR recognition processing only on a part of the product tag area. The recognition accuracy can be improved by executing the OCR recognition processing on the block specifying the processing of the present invention. Also, since the price is most important to the purchaser of the goods, the highest block is often the price area. Therefore, the price can be recognized with high accuracy by estimating this block as a price area and executing the OCR recognition processing.

In the fifth invention, a product tag identification processing unit for specifying a product tag area in image information in which a display shelf for displaying a product is shown, and a product tag specifying processing unit performing OCR recognition processing in the specified product tag area The product tag internal information identification processing unit identifies the processed information, and the product tag internal information identification processing unit recognizes the price information recognized as the tax-excluded price by the OCR recognition processing and the tax-included price It is an image recognition system which specifies the tax excluded price and the tax-included price by executing judgment processing of consistency of the size of the price and / or the tax rate for the recognized price information .

Similar to the first invention, by using the image recognition system according to these inventions, the information written on the product tag can be recognized with high accuracy, so the products displayed corresponding to it can be specified with high accuracy. Can.
In general, the price excluding tax and the price including tax are indicated on the product tag. Therefore, the accuracy can be further improved by determining whether the logical consistency of the price recognized as the tax-excluded price and the tax-included price respectively is established.

A sixth aspect of the invention relates to a product tag identification processing unit for specifying a product tag area in image information in which a display shelf for displaying a product is shown, and performing OCR recognition processing on a part or all of the specified product tag areas. And the in-product tag information specification processing unit for specifying information described in the product tag in the product tag, and the in-product tag information specification processing unit performs logic on the two price information specified by the OCR recognition process . It is an image recognition system that specifies tax-excluded price and tax-included price by executing the judgment process of the target consistency.

Similar to the first invention, by using the image recognition system according to these inventions, the information written on the product tag can be recognized with high accuracy, so the products displayed corresponding to it can be specified with high accuracy. Can.

A ninth invention is a product tag recognition processing unit for specifying product identification information in a product tag area in image information in which a display shelf for displaying a product is displayed , and product identification information of a product displayed in the image information. The recognition result of comparing the displayed product recognition processing unit to be specified, the product identification information in the specified product tag area, and the product identification information on the specified displayed product, and collating whether the recognition result matches And the recognition result collating processing unit, by the recognition processing in the display product recognition processing unit, makes a group in which faces with high similarity are lined up as a group, and each group in each tray And an image recognition system that associates the recognition result of the product tag by the product tag recognition processing unit .

According to the present invention, it is possible to collate the result of recognition of the product appearing on the display shelf with the recognition result of the information written on the product tag. This makes it possible to identify the displayed product with high accuracy.
Further, by using the present invention, the recognition result of the displayed product can be associated with the recognition result of the product tag.

In the above-described invention, the image recognition system further includes a photographed image information regularization processing unit that executes keystone correction processing on the image information obtained by photographing the display shelf, and the photographed image information corrector置化processing unit includes a first image information, based on the correspondence between the second image information, the vertices used in the keystone correction process in the first image information, projected to the second image information The present invention can be configured as an image recognition system that executes keystone correction processing of the second image information based on the projected vertices.

Photographed image information, it is also often not be the image information taken from the front. When executing the processing from the captured image information itself, its accuracy is low. Therefore, it is preferable to perform keystone correction processing to correct the image. Therefore, although the person in charge must specify the vertex that is the basis of the keystone correction processing, it is burdensome. By using the present invention, it is possible to automatically specify a vertex which is a reference of the keystone correction processing, and it is possible to reduce the workload of the person in charge.

In the above-described invention, the image recognition system further includes a position specification processing unit for specifying a product tag arrangement area in the image information on which the trapezoidal correction processing is performed in the photographed image information regularization processing unit; And a product tag placement area regularization processing unit that executes keystone correction processing in the product tag placement area.

The image recognition system of the first invention can be realized by reading and executing the image recognition program of the present invention on a computer. That is, the product tag identification processing unit identifies the product tag area in the image information in which the display shelf for displaying the product is displayed, and the computer displays the product tag by performing the OCR recognition processing in the specified product tag area. An image recognition program to be functioned as a product tag internal information specification processing unit for specifying information, wherein the product tag specification processing unit specifies a product tag area using lightness information of the image information, and The information identification processing unit binarizes the identified product tag area and performs labeling processing to identify a box, and among the identified boxes, identifies boxes satisfying a predetermined condition as blocks, and identifies the blocks. executing the OCR recognition process for a block, the two price information identified by the OCR recognition process, logical By executing the determining process of compatibility, specifying the tax price and price including tax, an image recognition program.

The image recognition system of the second invention can be realized by causing a computer to read and execute the image recognition program of the present invention. That is, the product tag identification processing unit that specifies the area of the product tag in the image information in which the display shelf displaying the product is displayed, and the computer performs OCR recognition processing on part or all of the specified product tag area. An image recognition program that causes the product tag identification processing unit to function as a product tag internal information identification processing unit that identifies information represented in a product tag by performing the product tag identification processing unit, the brightness information of the image information is accumulated in the horizontal direction By specifying the rising and falling of the histogram, the upper side position and the lower side position of the product tag are specified, and the rising and falling of the histogram obtained by integrating the lightness information of the image information in the vertical direction are specified. left position of the product tag falling as a pair is within a predetermined distance from, it identifies the right position, an image recognition program That.
The image recognition system of the third invention can be realized by reading and executing the image recognition program of the present invention on a computer. That is, the product tag identification processing unit that specifies the area of the product tag in the image information in which the display shelf displaying the product is displayed, and the computer performs OCR recognition processing on part or all of the specified product tag area. An image recognition program that causes the product tag identification processing unit to function as a product tag internal information specification processing unit that specifies information described in a product tag by performing the product tag identification processing unit, the brightness information of the image information is integrated in the vertical direction Among the rising edges of the histograms, if there is no tag region within a predetermined distance on the right side of the rising edge that can not form a pair, it is identified as a product tag region, and the histogram of the brightness information of the image information is integrated in the vertical direction. Of the falling edges, the falling edge that could not form a pair is displayed on the left side within a predetermined distance. Identifying a product tag area Without area, an image recognition program.

The image recognition system of the fourth invention can be realized by reading and executing the image recognition program of the present invention on a computer. That is, the product tag identification processing unit that specifies the area of the product tag in the image information in which the display shelf displaying the product is displayed, and the product tag by performing the OCR recognition processing in the area of the specified product tag. An image recognition program that causes the product identification information processing unit to function as a product tag information identification processing unit for identifying the information described in the item information processing unit, and the product tag in-data identification processing unit executes binarization processing in the identified product tag area By performing labeling processing on the binarized image information, successive pixels are grouped to specify a box, and adjacent boxes having a predetermined similarity among the specified boxes are merged. Block, and the block with the highest height among the above identified blocks is estimated as the price area and the OCR recognition is performed. Processing is executed, an image recognition program.
The image recognition system of the fifth invention can be realized by reading and executing the image recognition program of the present invention on a computer. That is, the product tag identification processing unit that specifies the area of the product tag in the image information in which the display shelf displaying the product is displayed, and the product tag by performing the OCR recognition processing in the area of the specified product tag. An image recognition program that causes the product identification information processing unit to function as the in-product tag information identification processing unit for identifying the information described in the item information, wherein the in-product tag information identification processing unit recognizes price information recognized as a tax excluded price by the OCR recognition processing. And, it is an image recognition program which specifies the tax excluded price and the tax-included price by executing judgment processing of the size of the price and / or the consistency of the tax rate on the price information recognized as the tax-included price .

The image recognition system of the sixth invention can be realized by reading and executing the image recognition program of the present invention on a computer. That is, the product tag identification processing unit that specifies the area of the product tag in the image information in which the display shelf displaying the product is displayed, and the computer performs OCR recognition processing on part or all of the specified product tag area. an image recognition program to function as a product tag information specifying unit, for identifying the title information to the product tag by performing the product tag information specifying processing unit, two identified by the OCR recognition process It is an image recognition program that identifies tax-excluded prices and tax-included prices by executing logical consistency determination processing on price information.

The image recognition system of the ninth invention can be realized by reading and executing the image recognition program of the present invention on a computer. That is, a product tag recognition processing unit that specifies product identification information in a region of a product tag in image information in which a display shelf displaying a product is displayed on a computer, product identification information of a displayed product in the image information A display product recognition processing unit for specifying the product identification information for comparing the product identification information in the area of the specified product tag with the product identification information of the specified displayed product, and checking whether the recognition results match An image recognition program to be functioned as a result collating processing unit , wherein the recognition result collating processing unit groups sections in which faces with high similarity are lined up as a group by recognition processing in the display product recognition processing unit; It is an image recognition program which matches the recognition result of the goods tag by the said goods tag recognition process part to each group in a row | line .

Claims (19)

A product tag specification processing unit for specifying a product tag area in image information in which a display shelf for displaying a product is shown;
And an in-product tag information specification processing unit for specifying information described in the product tag by performing an OCR recognition process in the specified product tag area.
The product tag identification processing unit
Identify the product tag area using the lightness information of the image information,
The product tag internal information identification processing unit
Specify a box by binarizing the specified product tag area and performing labeling processing,
Of the identified boxes, identify the boxes satisfying the predetermined condition as blocks,
Execute the OCR recognition process on the identified block;
Specify the tax-excluded price and the tax-included price by executing logical consistency judgment processing on the two price information identified by the OCR recognition,
An image recognition system characterized by A product tag specification processing unit for specifying a product tag area in image information in which a display shelf for displaying a product is shown;
And an in-product tag information specification processing unit that specifies information described in the product tag by performing an OCR recognition process on a part or all of the specified product tag area.
The product tag identification processing unit
Identify the product tag area using the lightness information of the image information,
An image recognition system characterized by The product tag identification processing unit
By integrating the lightness information of the image information in the horizontal direction, the upper side position and lower side position of the product tag are specified,
Identify the left side position and the right side position of the product tag by vertically integrating the lightness information of the image information,
The image recognition system according to claim 1 or 2, characterized in that: The product tag identification processing unit
Identify the upper side position and lower side position of the product tag by specifying the rising and falling of the histogram that integrates the lightness information of the image information in the horizontal direction,
By specifying the rising and falling of a histogram obtained by integrating the lightness information of the image information in the vertical direction, the left side position and the right side position of the product tag are specified by pairing the falling within a predetermined distance from the rising.
The image recognition system according to any one of claims 1 to 3, characterized in that: The product tag identification processing unit
Of the rising edges of the histogram obtained by integrating the lightness information of the image information in the vertical direction, if there is no tag region within a predetermined distance on the right of the rising edge that can not form a pair, it is specified as a product tag region;
Among the falling edges of the histogram obtained by integrating the lightness information of the image information in the vertical direction, if there is no tag region within a predetermined distance on the left side of the falling edge where the pair could not be formed,
The image recognition system according to any one of claims 1 to 4, characterized in that: A product tag specification processing unit for specifying a product tag area in image information in which a display shelf for displaying a product is shown;
And an in-product tag information specification processing unit for specifying information described in the product tag by performing an OCR recognition process in the specified product tag area.
The product tag internal information identification processing unit
Specify a box by binarizing the specified product tag area and performing labeling processing,
Of the identified boxes, identify the boxes satisfying the predetermined condition as blocks,
Execute the OCR recognition process on the identified block,
An image recognition system characterized by A product tag specification processing unit for specifying a product tag area in image information in which a display shelf for displaying a product is shown;
And an in-product tag information specification processing unit that specifies information described in the product tag by performing an OCR recognition process on a part or all of the specified product tag area.
The product tag internal information identification processing unit
Specify the tax-excluded price and the tax-included price by executing logical consistency judgment processing on the two price information identified by the OCR recognition,
An image recognition system characterized by The product tag internal information identification processing unit
Execute binarization processing in the specified product tag area,
By performing labeling processing on the binarized image information, consecutive pixels are grouped to specify a box,
Of the specified boxes, adjacent boxes having a predetermined similarity are merged and specified as a block,
Of the identified blocks, the block with the highest height is estimated to be the price area and the OCR recognition process is performed,
The image recognition system according to any one of claims 1 to 7, characterized in that: The product tag internal information identification processing unit
With regard to the price information recognized as tax excluded price and the price information recognized as tax-included price by the OCR recognition, the tax-excluded price and tax-included price are executed by executing judgment processing of consistency of the price and / or the tax rate. Identify the
The image recognition system according to any one of claims 1 to 8, characterized in that. A product tag recognition processing unit that specifies product identification information in a product tag area in image information in which display shelves for displaying products are displayed;
A display product recognition processing unit that specifies product identification information of a displayed product in the image information;
A recognition result collation processing unit that compares the product identification information in the specified product tag area with the product identification information of the displayed displayed product, and checks whether the recognition results match each other;
An image recognition system characterized by having. The recognition result collation processing unit
By the recognition processing in the display product recognition processing unit, divisions in which faces with high similarity are arranged are made into a group, and the recognition result of the product tag by the product tag recognition processing unit is associated with each group in each tray
The image recognition system according to claim 10, characterized in that: The image recognition system further comprises
And a photographed image information regularization processing unit that executes keystone correction processing on photographed image information obtained by photographing the display shelf.
The photographed image information regularization processing unit
Projecting a vertex used in the keystone correction process in the first captured image information onto the second captured image information based on the correspondence between the first captured image information and the second captured image information,
Performing a keystone correction process of the second captured image information based on the projected vertex;
The image recognition system according to any one of claims 1 to 11, characterized in that: The image recognition system further comprises
A position specification processing unit for specifying a product tag placement area in the photographed image information for which the trapezoidal correction processing has been performed in the photographed image information regularization processing unit;
And a product tag placement area regularization processing unit that executes keystone correction processing in the specified product tag placement area.
The product tag identification processing unit
Execute processing on the image information after the keystone correction processing has been performed,
The image recognition system according to any one of claims 1 to 12, characterized in that: The commodity tag placement area regularization processing unit
Edge detection is performed in the product tag arrangement area, and a straight line group satisfying a predetermined condition of the angle in the vertical direction is detected;
Identify at least two straight lines out of the detected straight line group,
Execute keystone correction processing using the two specified straight lines,
The image recognition system according to claim 13, characterized in that. Computer,
Merchandise tag specification processing unit for specifying a merchandise tag area in image information in which a display shelf for displaying merchandise is shown,
It is an image recognition program to be functioned as an in-product tag information specification processing unit that specifies information described in a product tag by performing an OCR recognition process in the specified product tag area,
The product tag identification processing unit
Identify the product tag area using the lightness information of the image information,
The product tag internal information identification processing unit
Specify a box by binarizing the specified product tag area and performing labeling processing,
Of the identified boxes, identify the boxes satisfying the predetermined condition as blocks,
Execute the OCR recognition process on the identified block;
Specify the tax-excluded price and the tax-included price by executing logical consistency judgment processing on the two price information identified by the OCR recognition,
An image recognition program characterized by Computer,
Merchandise tag specification processing unit that specifies the area of the merchandise tag in the image information in which the display shelf displaying the merchandise is shown,
It is an image recognition program to be functioned as an in-product tag information specification processing unit that specifies information described in a product tag by performing an OCR recognition process on a part or all of the specified product tag area,
The product tag identification processing unit
Identify the area of the product tag using the lightness information of the image information,
An image recognition program characterized by Computer,
Merchandise tag specification processing unit that specifies the area of the merchandise tag in the image information in which the display shelf displaying the merchandise is shown,
It is an image recognition program which is made to function as an in-product tag information specification processing unit that specifies information described in a product tag by performing OCR recognition processing in the area of the specified product tag,
The product tag internal information identification processing unit
Specifying a box by binarizing the area of the specified product tag and performing a labeling process,
Of the identified boxes, identify the boxes satisfying the predetermined condition as blocks,
Execute the OCR recognition process on the identified block,
An image recognition program characterized by Computer,
Merchandise tag specification processing unit that specifies the area of the merchandise tag in the image information in which the display shelf displaying the merchandise is shown,
It is an image recognition program to be functioned as an in-product tag information specification processing unit that specifies information described in a product tag by performing an OCR recognition process on a part or all of the specified product tag area,
The product tag internal information identification processing unit
Specify the tax-excluded price and the tax-included price by executing logical consistency judgment processing on the two price information identified by the OCR recognition,
An image recognition program characterized by Computer,
Merchandise tag recognition processing unit that specifies merchandise identification information in the area of the merchandise tag in the image information in which the display shelf displaying the merchandise is shown,
A display product recognition processing unit that specifies product identification information of a displayed product in the image information,
A recognition result collation processing unit that compares product identification information in the area of the specified product tag with product identification information of the specified displayed product, and checks whether the recognition results match
An image recognition program characterized by functioning as