JP2018142293A - Commodity discrimination device, commodity discrimination program, and commodity discrimination method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a commodity discrimination device which streamlines a process to discriminate commodities on a commodity display shelf.SOLUTION: Focusing on the fact that commodities belonging to identical categories are often arranged together on commodity display shelves in general retail shops and every commodity category has a representative commodity which is always displayed on each commodity display shelf, a commodity discrimination device: checks an image of the commodity display shelf in an original state with the representative commodity in each commodity category as a sole search object; estimates the category of the commodities on the commodity display shelf on the basis of the discriminated representative commodity; and discriminates the commodities by executing a search process to check the image of the commodity display shelf with the commodities in the estimated category as the sole search objects. For the other commodities which cannot be discriminated, the commodity discrimination device: checks the image of the commodity display shelf with the search object expanded to the other commodity category; and discriminates the commodities corresponding to all the commodities in the image of the commodity display shelf.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a product discriminating apparatus, a product discriminating program, and a product discriminating method for discriminating products displayed on a merchandise display shelf.

  Thousands to tens of thousands of products are displayed in general retail stores such as convenience stores, drug stores, and supermarkets. It is required to check the display status of the product from time to time.

  In response to these issues, the product label with product identification information that identifies each product displayed on the product display shelf is used to identify the product on the product display shelf and quickly grasp the display status. A method of attaching to a shelf (see Patent Document 1), a mark that can identify a product placed at the place on the product placement surface of the product display shelf, and whether or not the mark appears in an image of the product display shelf Therefore, a method for grasping the display status of a product (see Patent Document 2) and the like are disclosed.

  In order to carry out these methods, retail stores must put product labels and product identification marks on product display shelves, but the means for grasping the product display status without requiring such work on the retail store side. As an invention, a product display shelf is photographed, and an image of a product included in the photographed image is compared with image information of a registered product to determine a product on the product display shelf (for example, (See Patent Documents 3 and 4).

Japanese Patent Laid-Open No. 2006-194833 JP 2016-42307 A JP2015-210651A JP 2009-187482 A

  Understanding the display status of products on product display shelves is not limited to retailers that operate retail stores that require appropriate management of products to be displayed or inventory, and where their products are located on retail store shelves. It is also a challenge for manufacturers who want to get information on what products are being sold and what kind of competing products are being sold alongside them. For this reason, these manufacturers have assigned a person in charge of visiting a retail store called a rounder to grasp the display status of their products on the retail store shelves.

  When the rounder grasps the state of the merchandise display shelf, it depends on a method using merchandise labels or merchandise identification marks that require work on the retail store side as shown in Patent Literature 1 and Patent Literature 2. Therefore, it is easy to adopt a method for discriminating products by taking images of product display shelves as shown in Patent Document 3 and Patent Document 4, and recently various systems using image recognition technology are provided. It has come to be.

  However, the problem with such product recognition systems using image recognition technology is that there are a huge number of products handled in general retail stores such as convenience stores, drug stores, and supermarkets. The data processing that matches the product image identified from the image and the product image registered in advance for each product becomes very heavy, and the accuracy of matching tends to decrease when the number of objects to be matched becomes large is there.

  For example, if it is a standard store of a convenience store, it is about 3,000 SKU per store (SKU is the smallest unit for managing merchandise, about 10,000 SKU per year for the entire convenience store chain) If it is a store, about 15,000 SKU per store (about 50,000 SKU per year for the entire drug store chain), if it is a standard store of a general supermarket, about 50,000 SKU per store (annual for the entire general supermarket chain) If it is estimated that about 80,000 SKU) products are handled, and all of these items are to be collated, the burden of collation data processing and a decrease in matching accuracy tend to be serious problems. There is a need for a mechanism that makes processing for discriminating products more efficient while targeting a huge variety of products.

  The present invention has been made in order to cope with such a problem, and a product discrimination device, a product discrimination program, and a product discrimination method for improving the efficiency of processing for discriminating products displayed on a product display shelf. Is intended to provide.

  The present invention that solves such a problem is a product discriminating device that discriminates products displayed on a product display shelf, and identifies a product image included in the image from an image of the product display shelf Information relating to the product image identified as the representative product by comparing the product image identified by the identifying unit and the product image identifying unit with the product image of the representative product designated as the product that is necessarily displayed for each product category From one or two or more product categories to which at least one representative product belongs, product category determination means for preferentially determining the product category to be searched, and the product image identified by the product image identification means, Products that identify the product corresponding to each product image by comparing with the product image of the product belonging to the product category determined by the product category determination means And another unit, a product discriminating apparatus comprising: a. Each product image is compared with a product image of a product belonging to one or two or more other product categories, excluding the product category determined by the product category determination unit. It is possible to provide a second product discrimination means for discriminating a product corresponding to the item.

  In the present invention, focusing on the fact that merchandise belonging to the same merchandise category is often arranged on the merchandise display shelves of ordinary retail stores, and there are representative merchandise that are always displayed for each merchandise category, The product images are collated from the images obtained by photographing the display shelves for only the representative products for each product category as search targets, and the product categories arranged on the product display shelves are inferred from the determined representative products. Next, a configuration is made in which a product is discriminated by executing a search process for matching product images only for products in the product category. For products that could not be discriminated, the search object is expanded to other product categories, and product images are collated, and products corresponding to all product images are discriminated.

  By configuring in this way, it is possible to greatly narrow down the products that are subject to processing for comparing product images, and to significantly reduce the burden on data processing for verification. By focusing on the representative product, it is possible to appropriately narrow down the search target according to the actual state of the product display shelf, and it can be expected that the matching accuracy will be improved.

  The present invention further comprises second product category determining means for determining one or more product categories related to the product category determined by the product category determining means as a product category to be searched, The second product discriminating unit collates the product images that cannot be discriminated by the product discriminating unit with the product images of the products belonging to the product category determined by the second product category determining unit, and determines each product image. It can also be characterized by determining corresponding products.

  Furthermore, the present invention provides the product category determination means, wherein one or more of the product categories to which the representative product for which the selected product image is identified belong not preferentially determined as a search target product category. A third product category determining unit that determines another product category as a product category to be searched; and a product image in which a product cannot be determined by the product determining unit and the second product determining unit. Third product discrimination means for discriminating a product corresponding to each product image by collating with a product image of a product belonging to the product category determined by the category determination means may be a feature.

  With the above configuration, for product image matching that cannot be identified by the product category determined from the representative product, the product category related to the determined product category or representative products other than the previously determined product category By determining other product categories to be determined with priority, product discrimination by efficient product image matching is realized.

  The present invention can also be specified as a product discrimination program that operates in the product discrimination device according to the present invention.

  The product identification program according to the present invention is a product image identification step for identifying a product image included in the image from an image obtained by photographing the product display shelf in a product determination device that determines a product displayed on the product display shelf. The product image identified in the product image identification step is collated with the product image of the representative product designated as the product that is necessarily displayed for each product category, and at least one of the information about the product image determined as the representative product is obtained. A product category determining step for preferentially determining one or more product categories to which the representative product belongs to a product category to be searched, and the product image identified in the product image identifying step, the product category determining step Compare the product image of the product that belongs to the product category determined in step 1, and determine the product corresponding to each product image Is a product discrimination program characterized by executing a goods determination step.

  Moreover, the product discrimination program according to the present invention can be specified as a product discrimination program corresponding to each configuration of the product discrimination device described above.

  The present invention can also be specified as a product discrimination method executed by the product discrimination device according to the present invention.

  The product discrimination method according to the present invention includes a product image identification step in which a product discrimination device that discriminates a product displayed on a product display shelf identifies a product image included in the image from an image obtained by photographing the product display shelf; The product discriminating apparatus collates the product image identified in the product image identification step with the product image of the representative product designated as the product that is necessarily displayed for each product category, and the product identified as the representative product. A product category determining step for preferentially determining one or two or more product categories to which at least one representative product belongs from information relating to an image as a product category to be searched, and the product determining device includes the product image identifying step. The product image identified in step 1 is compared with the product image of the product belonging to the product category determined in the product category determination step. And product determination step of determining a product corresponding to the product image of a discriminating method of the product being characterized by having a.

  The product discrimination method according to the present invention can also be specified as a product discrimination method corresponding to each configuration of the product discrimination device described above.

  According to the present invention, the process of photographing the product display shelf and discriminating the displayed product from the photographed product image is remarkably improved without reducing the matching accuracy. In particular, the present invention contributes to the efficiency and sophistication of the work of confirming the display status of products in retail stores by a manufacturer's rounder who delivers products to retail stores with product display shelves. A positive impact on product sales strategies can be expected.

It is a figure which shows the outline | summary of embodiment of this invention. It is a block diagram which shows an example of a structure of the goods discrimination | determination apparatus which concerns on this invention. It is a flowchart which shows an example of the processing flow of the goods discrimination | determination program which concerns on this invention. It is a figure which shows an example of the goods master in this invention. It is a figure which shows the example which discriminate | determines representative goods from the image which image | photographed the goods display shelf in this invention. It is a figure which shows an example of the category master in this invention.

  Embodiments for carrying out the present invention will be described below in detail with reference to the drawings. In the following description, an example in which an image of a merchandise display shelf photographed with a mobile terminal such as a smartphone is transmitted to a server via the Internet to discriminate the merchandise will be described. The example shown below is an embodiment of the present invention. However, the present invention is not limited to such an embodiment.

  FIG. 1 shows an overview of an embodiment of the present invention. In FIG. 1, the product identification device according to the present invention is included in a product information management server, and a user such as a rounder images a product display shelf installed in a retail store using a mobile terminal such as a smartphone. Is transmitted to the product information management server, the product information management server determines each product included in the image of the product display shelf, stores information on the display status of the product in the product display shelf of the retail store in the database, Managed as product display information. Information regarding the product displayed on the product display shelf may be returned to the mobile terminal that has transmitted the image of the product display shelf.

  Here, in the present invention, when each product included in the product display shelf image is determined by the product information management server, all products whose product images are registered in the database of the product information management server are targeted. Rather than executing product image verification, product images are verified by narrowing down the target to only the products designated as representative products for each product category from the image of the product display shelf. In the present invention, focusing on the fact that merchandise belonging to the same product category is often arranged on the merchandise display shelves of retail stores, and that there are representative products that are always displayed for each product category, narrow down to representative products. It is usually possible to identify one of the products by searching with, and if the search target is narrowed down to the product category of the identified product, matching accuracy is high and efficient product image matching is realized. This is because it is considered possible.

  When representative products are identified, product categories that are likely to be displayed on the product display shelf are determined from the product categories of the identified representative products, and only the determined product categories are narrowed down to product images. The process which discriminate | determines goods by collation of is performed. For the product image of the product that could not be discriminated, the search target is expanded to the product category or the like that is strongly related to the previously determined product category, and the product is further discriminated by matching the product image.

  FIG. 1 shows an example of an embodiment in which the product information management server that can be connected via the Internet to the portable terminal that captures the product display shelf image is shown in FIG. However, such a process may be executed by a mobile terminal such as a smartphone without communicating with the product information management server. In this case, the mobile terminal possessed by the rounder or the like is included in the product discrimination device according to the present invention. That will be true. Also, the type of the mobile terminal is not particularly limited, and using a tablet computer or a personal computer, it is also possible to execute processing for identifying a product by comparing product images with these mobile terminals.

  FIG. 2 shows an example of the configuration of the product discrimination device according to the present invention. In FIG. 2, a part of the functions provided in the product information management server 10 constitutes the product discrimination device according to the present invention.

  The product information management server 10 having a function corresponding to the product discrimination device according to the present invention is a computer that includes an auxiliary storage device such as a CPU, a main memory, and an HDD, and is connected to a network and functions as a server. A program stored in the auxiliary storage device is read into the main memory, and a predetermined function is realized by executing arithmetic processing by the CPU.

  The physical configuration of the computer constituting the product information management server 10 is not particularly limited, and the same computer is provided with other functions for determining the product included in the product display shelf image in the present invention. It may be a thing. Each function necessary for the present invention may be physically realized by one computer or may be realized by using a plurality of computers.

  The image receiving unit 11, the product determining unit 12, the representative product searching unit 121, the category determining unit 122, the product searching unit 123, and the product display information transmitting unit 13 of the product information management server 10 are all functionally specified. Thus, a program corresponding to the function of each unit stored in an auxiliary storage device such as an HDD is read into the main memory, and a function corresponding to each unit is realized by executing arithmetic processing by the CPU.

  The product information storage unit 14 (product master 141, category master 142, etc.) and product display information storage unit 15 of the product information management server 10 are each assigned a predetermined storage area of an auxiliary storage device such as an HDD. . These storage areas are not required to be physically provided in one computer, but may be provided in a plurality of computers including a computer constituting a database server.

  As the portable terminal 20, a portable network terminal such as a smartphone or a tablet computer is used. The image capturing unit 21, the image transmitting unit 22, and the product display information receiving unit 23 are all functionally specified, and call a corresponding program from an auxiliary storage device such as an SSD in the main memory of the portable terminal 20. Thus, the functions corresponding to the respective units are realized by executing arithmetic processing by the CPU.

  The administrator terminal 30 is a computer capable of inputting / outputting information to / from the product information storage unit 14 and the product display information storage unit 15 of the product information management server 10, and the configuration thereof is not particularly limited. Etc. may be used. The administrator terminal 30 may be connected to the product information management server 10 via the Internet.

  Based on the above configuration, the processing flow of the product discrimination program according to the present invention will be described along the flowchart of FIG.

  A manufacturer's rounder or the like who owns a mobile terminal 20 that visits a retail store activates the image capturing unit 21 of the mobile terminal 20 to capture a product display shelf in the retail store where the displayed product is to be identified. . The photographed product display shelf image (image file) is transmitted from the image transmission unit 22 to the product information management server 10 via the Internet.

  In the merchandise information management server 10, the image receiving unit 11 is activated to receive the image of the merchandise display shelf transmitted from the portable terminal 20 (S1). Image recognition of the received product display shelf image is executed to identify each product image included in the product display shelf image (S2). The technology employed in this image recognition is not particularly limited. For example, the part corresponding to the product displayed on the product display shelf is recognized from the shape and size of the contour, the color pattern, etc. And it is sufficient.

  Subsequently, the product determination unit 12 is activated, and each product image identified as a product in the product display shelf image is converted into the SKU (minimum unit for managing the product) of the product stored in the product information storage unit 14. ) The product image registered every time is searched to find out which product each identified product image corresponds to. In the present invention, product information is registered in the product information storage unit 14 here. Rather than performing the matching process on the product images of all the products that are currently being sold, the representative product search unit 121 performs matching on only the product images of the products set as the representative products for each product category, Search processing is executed (S3).

  FIG. 4 shows an example of the product master 141 stored in the product information storage unit 14, but the product image used for matching is shown in the record provided for each SKU of the product identified by the product code. Information on the product category to which the product belongs, and a product that is recognized as a representative product handled in many retail stores in each product category is a flag indicating that it is a representative product (in this example, “1” in the “representative product” field is set as a flag indicating that the product is a representative product). The product for which the flag of the representative product is set is not limited to one product (SKU) for each product category, and a plurality of products (SKU) may be selected as the representative product.

  Next, in the category determination unit 122, the product category that is presumed to be most frequently displayed on the photographed product display shelf is determined from the information regarding the product image determined as the representative product in S3 (S4). . Note that the product category determined here may be a plurality of product categories.

  Referring to the example shown in FIG. 5, in FIG. 5, a search is performed by the representative product search unit 121, and a part of each product image identified as a product in the product display shelf image is a representative. Although an example of being identified as a product is shown, six product images are identified as representative products, and when the total product category of each identified representative product is tabulated, there are three representative products in category A, It can be confirmed that two representative products of the B category and one representative product of the C category are identified.

  The present invention focuses on the fact that merchandise belonging to the same merchandise category is often arranged on merchandise display shelves of ordinary retail stores, and that representative merchandise is always displayed for each merchandise category. The representative product included in the image of the display shelf is identified, the product category to which the representative product belongs is inferred as the product category of the product that is likely to be displayed on the product display shelf, and the product image is verified. The search target is narrowed down. In the example of FIG. 5, since the number of representative products of category A is the largest, it is assumed that a large number of products of category A are displayed on this product display shelf, and the remaining products are narrowed down to products of category A. It can be determined that it is efficient to discriminate the image. A plurality of product categories may be determined as search target product categories, such as the top two A category and B category.

  As described above, when the product category for searching for the product image other than the product determined as the representative product is determined, the product image other than the product determined as the representative product is determined by the product search unit 123. Only the product images of the products belonging to the product category are collated and the search process for discriminating the products is executed (S5).

  If the product is determined for all the product images identified from the images on the product display shelf by the search process of S5 (Yes in S6), the product display information transmission unit 13 is activated and displayed on the product display shelf. The merchandise display information in which each merchandise is identified is transmitted to the portable terminal 20 (S7), and the transmitted merchandise display information is stored in the merchandise display information storage unit 15 (S8). The merchandise display information transmitted to the portable terminal 20 is received by the merchandise display information receiving unit 23 and output on the screen of the portable terminal 20 in a predetermined format.

  Since the product display information stored in the product display information storage unit 15 can be confirmed from the administrator terminal 30, a sales person in charge of the manufacturer can consider the sales strategy of the product using this information. . In order to clarify which retail store's merchandise display shelf is information, for example, when transmitting an image of the merchandise display shelf from the mobile terminal 20, the position information specified by the GPS function of the mobile terminal 20 is also added. The store code of the retail store that is transmitted and specified from the location information may be included in the product display information.

  If the product is not determined for all product images identified from the image on the product display shelf by the search processing in S5 (No in S6), the process returns to the category determination unit 122 to determine the next product category to be searched ( S4).

  The method for determining the product category to be searched next is not particularly limited, but for example, the category master 142 of the product information storage unit 14 is shown in FIG. Store a category master that associates product categories that are strongly related to each product category as shown as related categories. Here, the related categories that are associated with the product categories that were initially searched are listed below. The product category to be searched may be determined.

  For example, for products belonging to the product category “clothing detergent”, products belonging to the product category such as “clothing softener” and “clothing bleach” are often displayed side by side on the same product display shelf. Such product categories may be associated as related categories and set in the category master 142.

  Even if the product search unit 123 performs the search process again for the product category determined as described above (S5), the product is not determined for all product images identified from the images on the product display shelf. (S6 is No), it returns to the category determination part 122, and the goods category searched next is determined (S4).

  The method for determining the product category to be searched next is not particularly limited, but when there are a plurality of related categories, the remaining related categories may be searched in order. In addition, when determining the product category from the representative product, if multiple representative products belonging to different product categories are identified in the image of the product display shelf, other than the product category previously determined as the search target Other product categories (In the example of FIG. 5, the category A with the largest number of products is determined as the first search target, then the category B with the second largest number of products, and then the category C. ) May be determined as a search target.

DESCRIPTION OF SYMBOLS 10 Product information management server 11 Image receiving part 12 Product discrimination | determination part 121 Representative product search part 122 Category determination part 123 Product search part 13 Product display information transmission part 14 Product information storage part 141 Product master 142 Category master 15 Product display information storage part 20 Mobile terminal 21 Image photographing unit 22 Image transmitting unit 23 Product display information receiving unit 30 Administrator terminal

Claims (3)

A product discrimination device for discriminating products displayed on a product display shelf,
Product image identification means for identifying a product image included in the image from an image of the product display shelf;
The product image identified by the product image identification means is compared with the product image of the representative product designated as the product that is necessarily displayed for each product category, and at least one of the information on the product image determined as the representative product is used. Product category determining means for preferentially determining one or more product categories to which the representative product belongs to a product category to be searched for;
A product image identified by the product image identifying unit, a product image of a product belonging to the product category determined by the product category determining unit, and a product determination unit for determining a product corresponding to each product image;
A product discriminating apparatus comprising: In the product discriminating device for discriminating the products displayed on the product display shelf,
A product image identification step for identifying a product image included in the image from an image of a product display shelf;
The product image identified in the product image identification step is collated with the product image of the representative product designated as the product that is necessarily displayed for each product category, and at least one of the information about the product image determined as the representative product is obtained. A product category determination step for preferentially determining one or more product categories to which the representative product belongs as a product category to be searched;
A product determination step of comparing the product image identified in the product image identification step with a product image of a product belonging to the product category determined in the product category determination step, and determining a product corresponding to each product image; ,
A product discrimination program characterized by causing A product image identifying step of identifying a product image included in the image from an image obtained by photographing a product display shelf, wherein the product determination device that determines the product displayed on the product display shelf;
The product image identified by the product image identification step is compared with the product image of the representative product designated as the product that is necessarily displayed for each product category, and the product image identified as the representative product. A product category determining step for preferentially determining one or two or more product categories to which at least one representative product belongs from the information regarding the product category to be preferentially searched;
The product discriminating device collates the product image identified in the product image identification step with the product image of the product belonging to the product category determined in the product category determination step, and selects a product corresponding to each product image. A product discrimination step to discriminate;
A method for discriminating a product characterized by comprising: