JP2016095765A - Information processing device, sales data processing device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make a registration status of products easily comprehensible when concurrently performing a product registration by code reading and a product registration by object processing.SOLUTION: First processing means is configured to display product information as read sales registration information on the basis of a code symbol detected from an imaging image, and second processing means is configured to: detect an imaging object from the imaging image together with a start of the first processing means; select a product candidate similar to the imaging object; display a selection screen having the selected product candidate lined; and display the product information on the product as sales registration information when receiving a selection operation of the product subject to a product registration from the selection screen. Halt control means is configured to, when any one of the first processing means and the second processing means displays the sales registration information first, halt processing by the other processing means. The selection screen end means is configured to, when the first processing means displays the sales registration information first, and the halt control means causes the second processing means to be halted, causes the display of the selection screen by the second processing means to be ended.SELECTED DRAWING: Figure 6

Description

  Embodiments described herein relate generally to an information processing apparatus, a sales data processing apparatus, and a program.

  2. Description of the Related Art Conventionally, there is a technique related to general object recognition (object recognition) that identifies a type of a target article by extracting a feature quantity of a target article from a captured image of an image sensor and comparing it with a feature quantity for collation prepared in advance. is there. In the object recognition processing, a technique is known in which an information processing device such as a POS (Point Of Sales) terminal or a scanner device displays a selection screen in which product candidates are arranged in descending order of similarity. When the operator selects a target product from among the product candidates displayed on the selection screen, the information processing apparatus ends display of the selection screen and registers the selected product for sales.

  As a conventional technique, a technique for registering product sales by simply switching between a code reading process for reading a code symbol of a product code from a captured image and an object recognition process is disclosed. In recent years, information processing apparatuses that perform code reading processing and object recognition processing in parallel have been developed.

  When the code reading process and the object recognition process are performed in parallel, a product code with a small amount of information processing is often read first. When the product code is read first, it is not necessary for the operator to perform a product candidate selection operation, so it is desirable to end the display of the selection screen. However, the conventional information processing apparatus has still displayed the selection screen even after the product code has already been read, and it is difficult to understand that the product has already been registered.

  If the display of the selection screen remains, the operator misunderstands that the product registration has not yet been completed, and the operator may continue to hold the product over the imaging unit and read the registered product again. Therefore, if the selection screen is not terminated at an appropriate timing, there is a problem that an operation mistake that registers the same product twice easily occurs.

  The problem to be solved by the present invention is an information processing apparatus and a sales data processing apparatus capable of making it easy to understand the registration status of a product when a code reading process and an object recognition process are performed in parallel on a captured image. And to provide a program.

  The information processing apparatus according to the embodiment includes first processing means, second processing means, stop control means, and selection screen end means. The first processing means captures a captured image of the imaging means, detects a code symbol from the captured image, reads product information based on the detected code symbol, and displays the product information as sales registration information. Do. The second processing unit starts capturing the captured image of the imaging unit simultaneously with the start of the first process, detects the imaging target from the captured image, and checks the feature of the imaging target and the product image for comparison When a product candidate similar to the object to be imaged is selected by comparing with the characteristics of the imaged object, a selection screen in which the selected product candidates are arranged is displayed, and a selection operation of a product to be registered for sales is received from the selection screen, the product The second process of displaying the product information as sales registration information is performed. When one of the first processing means and the second processing means first displays the sales registration information, the stop control means stops the processing by the other processing means. The selection screen ending means displays the selection screen by the second processing means when the first processing means first displays the sales registration information and the stop control means stops the second processing means. Terminate.

FIG. 1 is a perspective view illustrating an example of a checkout system according to the embodiment. FIG. 2 is a block diagram illustrating a hardware configuration of the POS terminal and the commodity reading apparatus. FIG. 3 is a conceptual diagram illustrating the data structure of a PLU file. FIG. 4 is a block diagram showing a functional configuration of the POS terminal. FIG. 5 is a diagram illustrating a screen configuration example of the object recognition screen. FIG. 6 is a flowchart showing an example of the procedure of product registration processing by image recognition processing. FIG. 7 is a flowchart illustrating a procedure example of the code symbol recognition process. FIG. 8 is a flowchart illustrating an exemplary procedure of object recognition processing. FIG. 9 is a diagram illustrating a screen configuration example of the selection screen. FIG. 10 is a flowchart illustrating a procedure example of the stop control process.

  Hereinafter, the information processing apparatus and the program according to the present embodiment will be described with reference to the drawings, taking a checkout system as an example. The checkout system is a checkout system (POS system) including a POS terminal that performs registration and settlement of commodities related to one transaction. This embodiment is an application example to a checkout system introduced in a store such as a supermarket.

  FIG. 1 is a perspective view illustrating an example of a checkout system 1 according to the embodiment. As shown in FIG. 1, the checkout system 1 includes a product reading device 101 that reads information about a product, and a POS terminal 11 that performs registration and settlement of the product related to one transaction. Below, the example which applies the POS terminal 11 as an information processing apparatus concerning this embodiment is demonstrated.

  The POS terminal 11 is placed on the upper surface of the drawer 21 on the checkout table 41. The drawer 21 is controlled by the POS terminal 11 for the opening operation. On the upper surface of the POS terminal 11, a keyboard 22 that is pressed by an operator (a store clerk) is arranged. A store clerk display 23 for displaying information toward the operator is provided behind the keyboard 22 as viewed from the operator who operates the keyboard 22. The clerk display 23 displays information on the display surface 23a. A touch panel 26 is laminated on the display surface 23a. A customer display 24 that displays information toward the customer is erected on the back side of the store clerk display 23 so as to be rotatable. The customer display 24 displays information on its display surface 24a. The customer display 24 shown in FIG. 1 has the display surface 24a facing the front side in FIG. 1, but the customer display 24 is rotated so that the display surface 24a faces the back side in FIG. Thus, the customer display 24 displays information for the customer.

  A horizontally long table-shaped counter table 151 is arranged so as to form an L shape with the checkout table 41 on which the POS terminal 11 is placed. A load receiving surface 152 is formed on the upper surface of the counter table 151. A shopping basket 153 for storing products is placed on the cargo receiving surface 152. The shopping cart 153 may be divided into a first shopping cart 153a brought in by a customer and a second shopping cart 153b placed at a position sandwiching the commodity reading device 101 from the first shopping cart 153a. it can. The shopping basket 153 is not limited to a so-called basket shape, and may be a tray or the like. The shopping basket 153 (second shopping basket 153b) is not limited to a so-called basket shape, and may be a box shape, a bag shape, or the like.

  On the receiving surface 152 of the counter stand 151, the commodity reading apparatus 101 connected to the POS terminal 11 so as to be able to transmit and receive data is installed. The commodity reading apparatus 101 includes a thin rectangular housing 102. A reading window 103 is disposed in front of the housing 102. A display / operation unit 104 is attached to the upper portion of the housing 102. The display / operation unit 104 is provided with a store clerk display 106 which is a display unit having a touch panel 105 laminated on the surface thereof. A keyboard 107 is arranged on the right side of the store clerk display 106. A card reading groove 108 of a card reader (not shown) is provided on the right side of the keyboard 107. A customer display 109 that provides information to the customer is installed on the back left side of the display / operation unit 104 as viewed from the operator.

  Such a product reading apparatus 101 includes a product reading unit 110 (see FIG. 2). The product reading unit 110 arranges an imaging unit 164 (see FIG. 2) on the back side of the reading window 103.

  In the first shopping basket 153a brought in by the customer, products related to one transaction are stored. The product in the first shopping basket 153a is moved to the second shopping basket 153b by an operator who operates the product reading apparatus 101. In this movement process, the product is directed to the reading window 103 of the product reading apparatus 101. At this time, the imaging unit 164 (see FIG. 2) arranged in the reading window 103 images the product.

  The product reading apparatus 101 notifies the POS terminal 11 of the product code of the product to be registered for sales. Specifically, the product reading device 101 is for specifying which product registered in the PLU file F1 (see FIG. 3) described later corresponds to the product included in the image captured by the imaging unit 164. The screen is displayed on the display / operation unit 104. Then, the commodity reading apparatus 101 notifies the POS terminal 11 of the commodity code of the designated commodity. In the POS terminal 11, based on the product code notified from the product reading device 101, information related to sales registration such as product classification, product name, and unit price of the product corresponding to the product code is stored in a sales file (not shown). Record and register sales.

  FIG. 2 is a block diagram illustrating a hardware configuration of the POS terminal 11 and the commodity reading apparatus 101. The POS terminal 11 includes a microcomputer 60 as an information processing unit that executes information processing. The microcomputer 60 is configured by connecting a ROM (Read Only Memory) 62 and a RAM (Random Access Memory) 63 to a CPU (Central Processing Unit) 61 that executes various arithmetic processes and controls each unit.

  The RAM 63 is used as a work memory for object recognition processing and a work memory for code symbol recognition processing. Details of each process will be described later.

  The drawer 61, keyboard 22, store clerk display 23, touch panel 26, and customer display 24 are all connected to the CPU 61 of the POS terminal 11 via various input / output circuits (all not shown). The These are controlled by the CPU 61.

  The keyboard 22 includes a numeric key 22d, a temporary fastening key 22e, and a fastening key 22f on which numbers such as “1”, “2”, “3”... Including. The temporary closing key 22e (for example, a subtotal key) is a key for calculating a subtotal amount, a tax amount, and the like of the input product. The closing key 22f (for example, a current key) is a key for declaring the end of one transaction and performing a settlement process. When the closing key 22f is operated, the POS terminal 11 accepts the deposit amount input from the numeric keypad 22d and calculates change. Each display of the POS terminal 11 (the store clerk display 23 and the customer display 24) and each display of the product reading device 101 (the store clerk display 106 and the customer display 109) are the total payment amount, change Display information such as the amount.

  An HDD 64 (Hard Disk Drive) is connected to the CPU 61 of the POS terminal 11. The HDD 64 stores programs and various files. All or a part of the programs and various files stored in the HDD 64 are copied to the RAM 63 and executed by the CPU 61 when the POS terminal 11 is activated. An example of a program stored in the HDD 64 is a product sales data processing program PR1. An example of a file stored in the HDD 64 is a PLU file F1 distributed and stored from the store computer SC. The HDD 64 only needs to be a storage device, and may be a storage device configured with another storage medium such as an SSD (Solid State Drive).

  The PLU file F1 is a product file that stores information related to product sales registration for each product displayed and sold in a store. In the following description, the PLU file F1 is used as a dictionary, but the dictionary may be a file different from the PLU file F1. The dictionary stores collation data (features) for identifying products extracted from captured image data for a plurality of products. When the dictionary is a file different from the PLU file F1, the collation data (feature amount) stored in the dictionary and the information (identification information) of the PLU file F1 are linked. The feature amount is a parameterization of appearance features such as the standard shape, surface color, pattern, and unevenness of the product.

  FIG. 3 is a conceptual diagram illustrating the data configuration of the PLU file F1. As shown in FIG. 3, the PLU file F1 is a file that stores, for each product, a product code, information about the product, an illustration image, and a feature amount as product information of the product. The product code is identification information that can identify the product. The information regarding the product is information such as a product category to which the product belongs, a product name, a product type, and a unit price. The illustration image is an image showing the product. The feature amount is information such as a hue read from the captured product image and a surface unevenness state. Note that the feature amount is data for collation used for determination of similarity described later. The PLU file F1 is configured to be readable by the product reading apparatus 101 via a connection interface 65 described later.

  Returning to FIG. 2, the communication interface 25 for executing data communication with the store computer SC is connected to the CPU 61 of the POS terminal 11 via an input / output circuit (not shown). The store computer SC is installed in a store backyard or the like. A PLU file F1 distributed to the POS terminal 11 is stored in the HDD (not shown) of the store computer SC.

  Further, the CPU 61 of the POS terminal 11 is connected to a connection interface 65 that enables data transmission / reception with the commodity reading apparatus 101. The product reading apparatus 101 is connected to the connection interface 65. A printer 66 that prints on receipts and the like is connected to the CPU 61 of the POS terminal 11. The POS terminal 11 prints the transaction content of one transaction on the receipt under the control of the CPU 61.

  The commodity reading apparatus 101 also includes a microcomputer 160. The microcomputer 160 is configured by connecting a ROM 162 and a RAM 163 to a CPU 161 via a bus. The ROM 162 stores a program executed by the CPU 161. An imaging unit 164 and an audio output unit 165 are connected to the CPU 161 via various input / output circuits (all not shown). The operations of the imaging unit 164 and the audio output unit 165 are controlled by the CPU 161. The display / operation unit 104 is connected to the product reading unit 110 and the POS terminal 11 via the connection interface 176. The operation of the display / operation unit 104 is controlled by the CPU 161 of the product reading unit 110 and the CPU 61 of the POS terminal 11.

  The imaging unit 164 includes an image sensor (not shown) such as a color CCD image sensor or a color CMOS image sensor, and a light source (not shown) such as an LED, and from the inside of the reading window 103 under the control of the CPU 161. Take an image. For example, the imaging unit 164 captures an image of 30 fps (Flame Per Second). The imaging unit 164 sequentially stores frame images (captured images) captured at a predetermined frame rate in the RAM 163.

  The audio output unit 165 is an audio circuit and a speaker for generating a preset warning sound or the like. The audio output unit 165 performs warning sound or audio notification under the control of the CPU 161.

  Further, a connection interface 175 that connects to the connection interface 65 of the POS terminal 11 and enables data transmission / reception with the POS terminal 11 is connected to the CPU 161. In addition, the CPU 161 transmits and receives data to and from the display / operation unit 104 via the connection interface 175.

  Next, the functional configuration of the CPU 61 realized by the CPU 61 executing the program (product sales data processing program PR1) will be described.

  FIG. 4 is a block diagram showing a functional configuration of the POS terminal 11. By executing the program PR1, the CPU 61 executes the parallel control unit 610, the object recognition engine 620, the code symbol recognition engine 630, the stop control unit 640, the input reception unit 613, the registration unit 614, and the notification unit 615. , Function as a display control unit 616.

  The parallel control unit 610 outputs an imaging on signal to the imaging unit 164 and causes the imaging unit 164 to start an imaging operation. The parallel control unit 610 also starts the object recognition engine 620, the code symbol recognition engine 630, and the stop control process in parallel, and controls the processes in parallel.

  The stop control unit 640 monitors the processing status of the object recognition engine 620 and the code symbol recognition engine 630, and when one of the recognition engines recognizes the product and temporarily registers the sales product, the stop control unit 640 performs the processing of the other recognition engine. Stop. This prevents both engines from registering the same product twice.

  The object recognition engine 620 performs object recognition processing described later. The registration unit 614 uses the recognition result to register (provisional registration) the product information of the sales product as sales registration information. Further, the display control unit 616 displays the product information of the temporarily registered sales product in the registration details display area 83 (both see FIG. 5) of the registration screen 71. As described above, the object recognition engine 620, the registration unit 614, and the display control unit 616 function as the second processing unit of the present embodiment.

  The provisional registration is a registration process (for example, step S8 in FIG. 6) in which sales data is displayed after settlement by displaying the product of the sales product in the registration details display area 83 (both see FIG. 5) of the registration screen 71. Is different in its processing stage.

  The code symbol recognition engine 630 performs code symbol recognition processing described later. The registration unit 614 uses the recognition result to register (provisional registration) the product information of the sales product as sales registration information. Further, the display control unit 616 displays the product information of the temporarily registered sales product on the registration screen 71 (see FIG. 5). Accordingly, the code symbol recognition engine 630, the registration unit 614, and the display control unit 616 function as the first processing unit of the present embodiment.

  Here, more detailed functional configurations of the object recognition engine 620 and the code symbol recognition engine 630 will be described together with configuration examples of the respective units.

(Object recognition engine)
First, the functional configuration of each unit included in the object recognition engine 620 will be described. As illustrated in FIG. 4, the object recognition engine 620 includes an image capturing unit 621, a product detection unit 622, a similarity calculation unit 623, a candidate selection unit 624, and a candidate presentation unit 625.

  The image capturing unit 621 sequentially captures the frame images stored in the RAM 163 of the commodity reading apparatus 101 by the image capturing unit 164 onto the work memory for object recognition processing in the RAM 63 of the POS terminal 11. The frame images are captured by the image capturing unit 621 in the order stored in the RAM 163.

  The product detection unit 622 detects all or part of products included in the frame image captured by the image capture unit 621 using a pattern matching technique or the like. Specifically, a contour line or the like is extracted from an image obtained by binarizing the captured frame image. Next, based on the difference between the contour line extracted from the previous frame image and the contour line extracted from the current frame image, a product directed to the reading window 103 for sales registration is detected.

  The previous frame image is a background frame image captured by the image capturing unit 621 before the object recognition engine 620 is activated, for example. Further, as a previous frame image, a background frame image captured by the image capturing unit 621 immediately after execution of the previous object recognition process may be used. Furthermore, the user may acquire the previous frame image as a background at an arbitrary timing.

  As another method for detecting a product, the presence or absence of a skin color region is detected from a captured frame image. Next, when a skin color area is detected, that is, when a clerk's hand is detected, it is assumed that the clerk's hand is grasped by detecting the above-described contour line in the vicinity of the skin color area. Try to extract the outline of the product. At this time, when the contour indicating the shape of the hand and the contour of the other object are detected in the vicinity of the contour of the hand, the product is detected from the contour of the object.

  The similarity calculation unit 623 extracts the surface state such as the color of the product and the unevenness of the surface as a feature amount from all or part of the frame image of the product imaged by the imaging unit 164. The similarity calculation unit 623 compares the feature amount of each product registered in the PLU file F1 (hereinafter referred to as a registered product) with the feature amount of the product, thereby registering the product and the PLU file F1. The similarity with the product is calculated. Here, the degree of similarity is the degree to which the images of all or some of the products are similar when the product image of each product stored in the PLU file F1 is 100% = “similarity: 1.0”. It indicates whether or not Note that, for example, the similarity may be calculated by changing the weight between the color tone and the surface roughness.

Recognizing an object contained in an image in this way is called generic object recognition. Regarding such general object recognition, various recognition techniques are described in the following documents.
Keiji Yanai, “Current Status and Future of General Object Recognition”, IPSJ Journal, Vol. 48, no. SIG16 [Search August 10, 2010], Internet <URL: http://mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>

Further, techniques for performing general object recognition by dividing an image into regions for each object are described in the following documents.
Jamie Shotton et al., “Semantic Texton Forests for Image Categorization and Segmentation”, [searched August 10, 2010], Internet <URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1. 1.145.3036 & rep = rep1 & type = pdf>

  It should be noted that the method of calculating the degree of similarity between the captured product image and the registered product product image registered in the PLU file F1 is not particularly limited. For example, the similarity between the captured product image and each registered product registered in the PLU file F1 may be calculated as an absolute evaluation or may be calculated as a relative evaluation.

  Based on the similarity calculated by the similarity calculation unit 623, the candidate selection unit 624 uses a registered product that satisfies the predetermined threshold as a product candidate (hereinafter referred to as a product candidate) captured by the imaging unit 164. elect.

  The candidate presentation unit 625 displays information on the product candidates recognized by the candidate selection unit 624 on the display 106 for clerk. More specifically, the candidate presentation unit 625 reads the illustration images and product names of registered products that satisfy the conditions from the PLU file F1, arranges them in descending order of similarity calculated by the similarity calculation unit 623, and displays the display 106 for the store clerk. To display.

  In addition, the candidate presentation unit 625 may cause the audio output unit 165 to output a sound indicating that there are a plurality of candidates in accordance with the timing at which information related to the product candidates is displayed on the display for the store clerk 106. Further, the candidate presentation unit 625 may cause the audio output unit 165 to output information that can easily determine the distinction between vegetables and fruits.

(Code symbol recognition engine)
Next, functions of each unit included in the code symbol recognition engine 630 will be described. As shown in FIG. 4, the code symbol recognition engine 630 includes an image capturing unit 631, a code symbol detection unit 632, and a code reading unit 633.

  The image capturing unit 631 captures the frame image stored in the RAM 163 of the commodity reading apparatus 101 by the imaging unit 164 into the work memory for code symbol recognition processing in the RAM 63 of the POS terminal 11.

  The code symbol detection unit 632 detects a code symbol such as a barcode or a two-dimensional code from the frame image captured by the image capture unit 631.

  The code reading unit 633 recognizes the product by reading the product code from the code symbol detected by the code symbol detection unit 632.

  The input receiving unit 613 receives various input operations corresponding to the display on the store clerk display device 106 via the touch panel 105 or the keyboard 107 of the commodity reading apparatus 101. In addition, the input receiving unit 613 receives various input operations corresponding to the display on the store clerk display 23 via the touch panel 26 or the keyboard 22 of the POS terminal 11. For example, the input receiving unit 613 receives an input for selecting a product corresponding to the product held over the imaging unit 164 from the product candidates (icons 91a to 91c in FIG. 9) presented by the candidate presenting unit 625.

  The registration unit 614 performs sales registration (provisional registration) on the product candidate recognized by the object recognition engine 620 as the sale target product for the product for which the input receiving unit 613 has received the selection operation. That is, the registration unit 614 reads the product information from the PLU file F1 for the product for which the input receiving unit 613 has received the selection operation, and temporarily registers it together with the number of sales received by the input receiving unit 613.

  The registration unit 614 registers (provisionally registers) the product recognized by the code symbol recognition engine 630 as a sale target product. That is, the registration unit 614 reads the product classification, product name, unit price, and the like corresponding to the product code read by the code reading unit 633 from the PLU file F1, and temporarily registers them together with the number of sales received by the input receiving unit 613.

  In addition, the registration unit 614 generates sales data including the temporarily registered product information and the amount at the time of settlement when the end of one transaction is declared by the selection operation of the lock key, etc. (Registered in the figure).

  When the registration unit 614 registers a product, the notification unit 615 performs notification with a buzzer sound and display output. For example, the notification unit 615 displays a screen for notifying registration of a product on the store clerk display 106 of the product reading apparatus 101 or the store clerk display 106 of the POS terminal 11.

  In addition, when the product is provisionally registered by the code symbol recognition engine 630, the notification unit 615 may notify that the product has been temporarily registered by the code symbol. When a product is temporarily registered by the object recognition process, the operator needs to select a product candidate from the selection screen 85 (see FIG. 9). On the other hand, the code symbol recognition process automatically performs temporary registration without an operator operation. Therefore, when a product is provisionally registered by the code symbol recognition process, the operator is prompted to stop holding the product over the imaging unit 164. Thereby, it can be avoided that the operator keeps holding the product even after provisional registration and registers the same product again.

  The display control unit 616 controls an image to be displayed on the store clerk display 106 or the like. For example, the display control unit 616 displays a registration screen 71 for merchandise registration (see FIG. 5) on the clerk display 23 of the POS terminal 11 and the clerk display 106 of the merchandise reading apparatus 101.

  FIG. 5 is a diagram illustrating a screen configuration example of the registration screen 71. In the registration screen 71, a product can be registered using image recognition processing (that is, object recognition processing or code recognition processing). From the registration screen 71, it is also possible to register a product by receiving an input of a product code, or register a product by receiving a selection from a product list.

  An image recognition tag 81 is provided on the registration screen 71. When the image recognition tag 81 is selected, the CPU 61 starts a product registration process by the image recognition process. The product registration process by the image recognition process is a product registration process using the object recognition engine 620 and the code symbol recognition engine 630. When the image recognition tag 81 is selected and operated, the display control unit 616 assigns the display area of the object recognition screen G1 to one display area in the registration screen 71 and displays it.

  The object recognition screen G1 mainly has a captured image display area G12 and a guidance display area G13. The captured image display area G12 is an area for displaying an image captured by the imaging unit 164. By looking at the image displayed in the captured image display area G12, the operator can visually check how the product is held. The guidance display area G13 is an area for displaying a guidance illustration or a guidance message for object recognition.

  A product information display area 82 is provided at the bottom of the registration screen 71. The display control unit 616 displays, in the product information display area 82, product information that has been read most recently from the PLU file F1 through image recognition processing or a product list. As the product information, a product name, a unit price, and the like are displayed. The number of sales received by the input receiving unit 613 through the operation of the numeric keypad 22d or the like is also displayed in the product information display area 82.

  In addition, a registration details display area 83 is provided at the left end of the registration screen 71. The display control unit 616 displays the product information read from the PLU file F1 as described above in the registration details display area 83 in order of registration.

  Next, an operation example of the product registration process by the image recognition process performed by the POS terminal 1 will be described with reference to FIG. FIG. 6 is a flowchart showing an example of the procedure of product registration processing by image recognition processing.

  When the input receiving unit 613 receives a selection operation of the image recognition tag 81 on the registration screen 71, the parallel control unit 610 outputs an imaging on signal to the imaging unit 164 and causes the imaging unit 164 to start an imaging process (step S1). The parallel control unit 610 starts each process of the code symbol recognition process (step S2), the object recognition process (step S3), and the stop control process (step S4) in parallel. The procedure in each process will be described later with reference to FIGS.

  Next, the registration unit 614 determines whether or not the closing key 22f (see FIG. 2) has been selected (step S5). If the closing key 22f is not selected (step S5: No), the parallel processing of step S2, step S3, and step S4 is continued. When the closing key 22f is selected and a settlement instruction is received (step S5: Yes), the parallel control unit 610 outputs an imaging off signal to the imaging unit 164 and causes the imaging unit 164 to stop the imaging process (step S6). . Then, the registration unit 614 performs a settlement process (step S7). That is, the registration unit 614 calculates change from the total amount of the transaction and the deposit received from the customer from the numeric keypad 22d of the keyboard 22. The display control unit 616 displays the calculated change amount on the clerk display units 23 and 106 and the customer display units 24 and 109, respectively. Note that the display output destination may be output to some of the display units instead of all of these display units. The store clerk hands the change displayed on the display to the customer and finishes the settlement of this transaction.

  The registration unit 614 stores, in the sales file, product information of sales products for one transaction, total amount, total points, deposit amount, change amount, information indicating a cashier, transaction date / time, cash register number, receipt number, etc. Registration (main registration) is performed (step S8), and the transaction registration process is terminated.

  Next, an operation example of the code symbol recognition process performed in step S2 will be described with reference to FIG. FIG. 7 is a flowchart illustrating a procedure example of the code symbol recognition process. The code symbol recognition process here includes a process in which the registration unit 614 temporarily registers a product recognized by the code symbol recognition engine 630 and a process in which the display control unit 616 displays the product on the registration screen 71.

  First, the image capturing unit 631 starts capturing a frame image (step S21). That is, the image capturing unit 631 captures the frame image stored in the RAM 163 of the commodity reading apparatus 101 by the image capturing unit 164 into the code recognition work memory in the RAM 63 of the POS terminal 11 (step S21).

  The code symbol detection unit 632 sequentially performs code symbol detection processing on each of the frame images taken into the code recognition work memory. While the code symbol is not detected (step S22: No), the process returns to step S22 to continue the detection process. When the code symbol is detected (step S22: Yes), the image capturing unit 631 finishes capturing the frame image (step S23). The code reading unit 633 reads a product code from the detected code symbol (step S24). When the check digit does not become an appropriate value or the like, if the read code is incomplete and the code needs to be read again (No at Step S25), the process returns to Step S22. Then, the code symbol is detected again and the product code is read again (steps S22 to S25).

  On the other hand, if the code read in step S24 is appropriate (step S25: Yes), the registration unit 614 temporarily reads the product information corresponding to the product code from the PLU file F1 as the sales registration information. Register (step S26). The display control unit 616 displays the product information read in step S26, the points received from the keyboard 107, and the subtotal calculated based on these in the product information display area 82 and the registration details display area 83 of the registration screen 71. (Step S27).

  Next, an operation example of the object recognition process performed in step S3 will be described with reference to FIG. FIG. 8 is a flowchart illustrating an exemplary procedure of object recognition processing. The object recognition process here includes a process in which the registration unit 614 temporarily registers a product recognized by the object recognition engine 620 and a process in which the display control unit 616 displays the product on the registration screen 71.

  First, the image capturing unit 621 starts capturing a frame image (step S31). In other words, the image capturing unit 621 captures the frame image stored in the RAM 163 of the commodity reading apparatus 101 by the image capturing unit 164 into the object recognition work memory in the RAM 63 of the POS terminal 11 (step S31).

  The product detection unit 622 performs a process of detecting an area in which the product is captured with respect to the frame image captured in the object recognition work memory. Note that the product detection unit 622 may perform a product detection process on the processed image by integrating a plurality of frame images or performing image processing such as averaging or smoothing. When an object (that is, an imaging region of a product) is detected (step S32: Yes), the image capturing unit 621 ends the frame image capturing process (step S33). While no object is detected (step S32: No), the process returns to step S32. The product detection unit 622 continues the product detection process on the frame images sequentially captured by the image capturing unit 621 or the image newly generated by the image processing as described above until an object is detected.

  Next, the similarity calculation unit 623 calculates the feature amount of the product from the detected image area of the product. Then, the similarity calculation unit 623 calculates the similarity with the product stored in the PLU file F1 based on the calculated feature amount and the PLU file F1 (step S34). Next, the candidate selection unit 624 selects a product whose similarity is equal to or higher than a predetermined threshold as a product candidate (step S35). If no product candidate is selected (step S36: No), the process returns to step S31 and image capture is started again. When a product candidate is selected (step S36: Yes), the display control unit 616 displays a selection screen 85 as shown in FIG. 9 in the display area of the object recognition screen G1 (see FIG. 5) (step S37).

  FIG. 9 is a diagram illustrating a screen configuration example of the selection screen 85. The selection screen 85 is a screen for selecting a product to be registered from among product candidates. The display control unit 616 displays icons 91a to 91c including product candidate illustrations and product names in order of decreasing similarity. The selection screen 85 displays a guidance message 92 that prompts the operator to select a product to be registered from the product candidate icons 91a to 91c. In addition, the selection screen 85 is provided with an icon 93 for starting the process of re-selecting the product candidates by performing the object recognition process again. The operator may select the icon 93 when the displayed product candidate icons 91a to 91c do not include a product to be registered.

  Returning to FIG. 8, when the input receiving unit 613 receives a selection operation of any one of the product candidate icons 91 a to 91 c (step S <b> 38: Yes), the registration unit 614 stores the product name and the product corresponding to the product candidate. Product information such as price is read from the PLU file F1 and temporarily registered as sales registration information (step S39). If the selection operation of the icons 91a to 91c is not accepted (step S38: No), the process returns to step S31 and image capture is started again.

  Subsequently, the display control unit 616 displays the product information read by the registration unit 614, the number of products received by the input reception unit 613 from the keyboard 107, the subtotal amount calculated by the product information display area of the registration screen 71, and the like. 82 and the registration details display area 83 (both see FIG. 5) (step S40). Thereafter, the process proceeds to step S5 in FIG.

  Next, an operation example of the stop control process performed in step S4 will be described with reference to FIG. FIG. 10 is a flowchart illustrating a procedure example of the stop control process.

  The stop control unit 640 monitors the processing status of the code symbol recognition process (see FIG. 7) and the object recognition process (see FIG. 8) (step S41). That is, the stop control unit 640 monitors which step shown in FIG. 7 is currently being processed. Further, the stop control unit 640 monitors which step shown in FIG. 8 is being processed at the present time. Until either one of the object recognition engine 620 and the code symbol recognition engine 630 temporarily registers a product (step S42: No), the stop control unit 640 continues the monitoring process (step S41).

  When either the object recognition engine 620 or the code symbol recognition engine 630 recognizes the product and the registration unit 614 temporarily registers the product information based on the recognition result (step S42: Yes), the stop control unit 640 temporarily registers. The recognition process of the person who has not performed is stopped. In this case, the stop control unit 640 stops the recognition process in accordance with the processing progress status at that time.

  First, a stop processing procedure when a product is provisionally registered in the code symbol recognition process (see FIG. 7) and the object recognition process (see FIG. 8) is stopped will be described.

  When the code symbol recognition engine 630 is stopped by reading the product code (step S43: Yes), the stop control unit 640 reads which step of the object recognition processing (see FIG. 8) the processing stage of the object recognition engine 620 is. (Step S44). If the product detection unit 622 has not detected an object (product) from the frame image (step S45: No), the stop control unit 640 causes the image capture unit 621 to stop capturing the image (step S46). Then, the stop control unit 640 clears the work memory for object recognition (step S47), and proceeds to step S5 in FIG.

  If it is after step S32: Yes in the object recognition process (see FIG. 8) and the product detection unit 622 has already detected the product (step S45: Yes), the process proceeds to step S48. Then, in the object recognition process (see FIG. 8), if the process has not progressed to step S35 and no product candidate has been selected (step S48: No), the process of step S49 is performed. In step S49, the stop control unit 640 stops the product candidate selection process (step S35 in FIG. 8) by the candidate selection unit 624. Thereafter, the stop control unit 640 clears the work memory for object recognition (step S47), and proceeds to step S5 in FIG.

  On the other hand, if a product candidate has already been selected (step S48: Yes), the stop control unit 640 ends the display process of the selection screen 85 (step S38 in FIG. 8) by the display control unit 616 (step S50). Thereafter, the stop control unit 640 clears the work memory for object recognition (step S47), and proceeds to step S5 in FIG.

  On the other hand, a stop process procedure in the case where a product is provisionally registered in the object recognition process (see FIG. 8) and the code symbol recognition process (see FIG. 7) is stopped will be described.

  When it is a stop by object recognition of the object recognition engine 620 (step S43: No-step S51: Yes), the stop control part 640 stops the code symbol recognition process shown in FIG. 7 (step S52). Then, the stop control unit 640 clears the work memory for code symbol recognition (step S53), and proceeds to step S5 in FIG.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  For example, in the above embodiment, the POS terminal 11 includes the PLU file F1. However, the present invention is not limited thereto, and the commodity reading apparatus 101 may include the PLU file F1. Alternatively, an external device accessible by the POS terminal 11 and the product reading device 101 may have a PLU file F1.

  In the above embodiment, the image capturing unit 621, the parallel control unit 610, the object recognition engine 620, the code symbol recognition engine 630, the input reception unit 613, the registration unit 614, the notification unit 615, and the display The POS terminal 11 is applied as an information processing apparatus having a function with the control unit 616. However, the image capturing unit 621, the parallel control unit 610, the object recognition engine 620, the code symbol recognition engine 630, the input receiving unit 613, the registration unit 614, the notification unit 615, and the display are not limited thereto. The product reading apparatus 101 may be applied as an information processing apparatus that has a function with the control unit 616 and outputs a calculation result or a determination result to the POS terminal 11. Moreover, although the said embodiment demonstrated the data for collation as a feature-value, the data for collation are good also as the imaged product image (reference | standard image).

  Further, the function described in the above embodiment may be provided in one sales data processing apparatus having the functions of the POS terminal 11 and the commodity reading apparatus 101. As an example, the various functions of this embodiment described above may be provided in a self-checkout device (self-POS terminal). In other words, the self-POS terminal (sales data processing apparatus) may include the imaging unit 164 and have a configuration capable of realizing the function of each unit described above with reference to FIG. 4 as a control unit. The functions of the control unit include the parallel control unit 610, the object recognition engine 620, the code symbol recognition engine 630, the stop control unit 640, the input reception unit 613, the registration unit 614, the notification unit 615, and the display control unit 616 described above. Just have it.

  In addition, if the product candidate selected by the object recognition engine 620 in the object identification process before stopping and the product candidate selected by the object recognition engine 620 in the object identification process after resuming processing are the same product, the notification unit 615 You may alert | report that the previous goods candidate and this goods candidate are the same goods. More preferably, the product candidate having the highest similarity (similarity) selected in the previous object identification process (that is, before the process is stopped) and the most similar selected in the current object identification process (after the process is resumed). When the product candidate with a high price is the same, it may be notified by color or mark that the previous product candidate and the current product candidate are the same.

  As a result, when the next image capture is resumed, the operator continues to hesitate to pick up the same product to the imaging unit 164, and if the same product is imaged again, the same product is re-registered. Easier to avoid.

  In the above description, an example has been described in which the stop control unit 640 stops the other recognition process at the timing when the product is provisionally registered in either the code symbol recognition process or the object recognition process. The timing for triggering is not limited to provisional registration. That is, it may be a timing that can be confirmed from the outside, such as temporary registration and display of temporary registration information that accompanies it, or may be a timing that depends on other processing situations that are difficult to confirm from the outside. In FIG. 7, for example, the time when the code reading in step S24 is completed, the time when it is determined that the code is appropriate (step S25: Yes), the time when the product information is read from the PLU file F1 (step S26), and the like. Alternatively, it may be used as a timing for triggering a process stop.

  The program executed by each device of the above-described embodiment or modification is provided by being incorporated in advance in a storage medium (ROM or storage unit) included in each device, but is not limited thereto. For example, an installable or executable file is recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk). You may comprise. Furthermore, the storage medium is not limited to a medium independent of a computer or an embedded system, but also includes a storage medium in which a program transmitted via a LAN, the Internet, or the like is downloaded and stored or temporarily stored.

  Further, the program executed by each device of the above-described embodiment or modification may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network, or the Internet It may be configured to be provided or distributed via a network.

  As described above, according to the present embodiment, when the product information read in the code symbol recognition process (first process) is displayed on the registration screen 71 as the product information of the sales product, the object recognition process (second process). The display of the selection screen 85 displayed at is ended. The operator can recognize that the product has been registered (provisionally registered) as a sales product by the code reading process when the selection screen 85 is displayed. Therefore, according to this embodiment, it becomes easy to understand the processing stage of the image recognition processing performed in parallel.

  Further, according to the present embodiment, the display of the selection screen 85 can be automatically terminated without an operator's operation. Therefore, according to the present embodiment, it is possible to improve work efficiency when performing product registration by code reading processing and product registration by object recognition processing in parallel.

DESCRIPTION OF SYMBOLS 1 Checkout system 11 POS terminal 101 Goods reading apparatus 164 Imaging part 610 Parallel control part 613 Input reception part 614 Registration part 615 Notification part (notification means, 2nd notification means)
616 Display control unit (selection screen end means)
620 Object recognition engine (first processing means)
621 Image capture unit 622 Product detection unit 623 Similarity calculation unit 624 Candidate selection unit 625 Candidate presentation unit 630 Code symbol recognition engine (second processing means)
631 Image capturing unit 632 Code symbol detection unit 633 Code reading unit F1 PLU file

JP 2013-178722 A

Claims (6)

First processing means for capturing a captured image of the imaging means, detecting a code symbol from the captured image, reading product information based on the detected code symbol, and performing a first process of displaying the product information as sales registration information; ,
At the same time as the start of the first processing, the captured image capturing process of the imaging unit is started, the imaging object is detected from the captured image, and the characteristics of the imaging object are compared with the characteristics of the product image for verification. A product candidate similar to the object to be imaged is selected, a selection screen in which the selected product candidates are arranged is displayed, and when a selection operation of the product to be registered for sales is received from the selection screen, the product information of the product is registered as sales. Second processing means for performing second processing to be displayed as information;
When one of the first processing means and the second processing means first displays the sales registration information, a stop control means for stopping processing by the other processing means,
Selection screen end means for ending display of the selection screen by the second processing means when the first processing means first displays the sales registration information and the stop control means stops the second processing means. When,
An information processing apparatus comprising:   The stop control means monitors the processing status of the first process or the second process, and when one of the first processing means and the second processing means displays the sales registration information of the product first. The information processing apparatus according to claim 1, wherein processing by the other processing unit is stopped according to a processing status of the other processing. When the first processing means first displays the sales registration information of the product and stops the processing of the second processing means, the stop control means,
If the second processing means has not detected the imaging object from the captured image, the capturing of the captured image by the second processing means is stopped,
If the second processing means has not selected the product candidate, the selection is stopped,
The information processing apparatus according to claim 2, wherein if the second processing unit has selected the product candidate and displays the selection screen, the display of the selection screen is terminated. The first processing unit and the second processing unit restart the first processing and the second processing, respectively, after the stop control unit stops the processing by the other processing unit,
The product candidate with the highest similarity selected by the second processing means in the second processing before the stop and the product candidate with the highest similarity selected by the second processing means in the second processing after the restart The information processing according to any one of claims 1 to 3, further comprising an informing means for informing that the previous product candidate and the current product candidate are the same product if they are the same product. apparatus. Imaging means;
First processing means for capturing a captured image of the imaging means, detecting a code symbol from the captured image, reading product information based on the detected code symbol, and displaying the product information as sales registration information When,
At the same time as the start of the first processing, the captured image capturing process of the imaging unit is started, the imaging object is detected from the captured image, and the characteristics of the imaging object are compared with the characteristics of the product image for verification. A product candidate similar to the object to be imaged is selected, a selection screen in which the selected product candidates are arranged is displayed, and when a selection operation of the product to be registered for sales is received from the selection screen, the product information of the product is registered as sales. Second processing means for performing second processing to be displayed as information;
When one of the first processing means and the second processing means first displays the sales registration information, a stop control means for stopping processing by the other processing means,
Selection screen end means for ending display of the selection screen by the second processing means when the first processing means first displays the sales registration information and the stop control means stops the second processing means. When,
Sales registration means for registering the product information of the sales goods in the storage means as a sales record;
Sales data processing device including. A computer in an information processing device;
First processing means for capturing a captured image of the imaging means, detecting a code symbol from the captured image, reading product information based on the detected code symbol, and performing a first process of displaying the product information as sales registration information; ,
At the same time as the start of the first processing, the captured image capturing process of the imaging unit is started, the imaging object is detected from the captured image, and the characteristics of the imaging object are compared with the characteristics of the product image for verification. A product candidate similar to the object to be imaged is selected, a selection screen in which the selected product candidates are arranged is displayed, and when a selection operation of the product to be registered for sales is received from the selection screen, the product information of the product is registered as sales. Second processing means for performing second processing to be displayed as information;
When one of the first processing means and the second processing means first displays the sales registration information, a stop control means for stopping processing by the other processing means,
Selection screen end means for ending display of the selection screen by the second processing means when the first processing means first displays the sales registration information and the stop control means stops the second processing means. When,
Program to function as.