JP7167437B2 - Article identification device - Google Patents

Description

The present invention relates to an article identification device .

2. Description of the Related Art There is a technique of performing image recognition on a captured image of a product and specifying the captured product. For example, Patent Literature 1 discloses a system that uses a recognition model based on deep learning to recognize a product in a captured image from pre-registered products. In the system disclosed in Patent Literature 1, an image of a product placed in a predetermined product recognition area is captured, for example, from above or from the side, and the product as a subject is specified based on the obtained captured image.

Japanese Patent No. 6209717

In a configuration for recognizing a product as a subject using a recognition model as in Patent Literature 1, there is a possibility that the product cannot be recognized with high accuracy depending on the imaging environment in which the product is imaged. For example, it is difficult to accurately identify the product as the object when the product is imaged under different imaging environments such as the background of the object, the imaging direction with respect to the object, and the brightness of the surroundings. This is because it is difficult to use images captured in all imaging environments when constructing a recognition model.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and an object of the present invention is to provide an article identifying apparatus capable of improving the recognition accuracy of a subject based on a captured image.

An article identification apparatus according to an aspect of the present invention includes an image acquisition unit that acquires an image, a candidate identification unit that identifies an article candidate existing in the image using a classifier from the acquired image, and the identified An output unit that outputs information related to an article candidate, an information acquisition unit that acquires correct information indicating whether or not the outputted article candidate is correct, and an image that is acquired by the acquired correct information and the image acquisition unit. a learning unit that re- learns the classifier using , specifying a plurality of product candidates having high calculated degrees of certainty, the output unit outputting an instruction to highlight and display the specified product candidates according to the respective degrees of certainty, and the information acquiring unit, It is characterized by acquiring correct answer information indicating an article that is the correct answer among the plurality of article candidates .

According to one aspect of the present invention, it is possible to improve the accuracy of recognizing the article of the subject based on the captured image.

1 is a schematic diagram showing a configuration example of an article identification system; FIG. It is a block diagram which shows the structural example of a server apparatus and a user terminal. 3 is a block diagram showing functions implemented by a control unit of the server device; FIG. FIG. 4 is an explanatory diagram of processing performed by an article identification system; 4 is a flow chart showing a procedure of processing performed by a user terminal and a server device; 4 is a flowchart showing part of the procedure of processing performed by a user terminal and a server device; FIG. 4 is a schematic diagram showing an example of a screen of a user terminal; 4 is a flowchart showing part of the procedure of processing performed by a user terminal and a server device; FIG. 4 is a schematic diagram showing an example of a screen of a user terminal; It is a schematic diagram which shows the structural example of user information DB. 4 is a flow chart showing a procedure of processing performed by a user terminal and a server device; 4 is a flow chart showing a procedure of processing performed by a user terminal and a server device; FIG. 4 is a schematic diagram showing an example of a screen of a user terminal; FIG. 4 is a schematic diagram showing an example of a screen of a user terminal; 4 is a flow chart showing a procedure of processing performed by a user terminal and a server device; 4 is a flow chart showing a procedure of processing performed by a user terminal and a server device; FIG. 4 is a schematic diagram showing an example of a screen of a user terminal; 4 is a schematic diagram showing a configuration example of a reception history DB; FIG. 4 is a flowchart showing the procedure of determination processing performed by the server device; 4 is a flowchart showing part of the procedure of processing performed by a user terminal and a server device;

(Embodiment 1)
FIG. 1 is a schematic diagram showing a configuration example of an article identification system. In this embodiment, an article identification system that identifies an article as a subject in a captured image based on a captured image obtained by capturing an article such as a product will be described. The article identification system 100 includes a server device (information processing device) 10 and a plurality of user terminals 20. The server device 10 and each user terminal 20 are connected for communication via a network N such as the Internet. The user terminal 20 captures an image of an article such as a product according to an operation by the user, and transmits the obtained captured image to the server device 10 . The server device 10 refers to a learning model that has been learned in advance for the captured image acquired from the user terminal 20, and identifies an article (product) included in the captured image. A plurality of server devices 10 may be provided, may be realized by a plurality of virtual machines provided in one device, or may be realized using a cloud server.

FIG. 2 is a block diagram showing a configuration example of the server device 10 and the user terminal 20. As shown in FIG. The server device 10 is a server computer, a personal computer, or the like. The server device 10 includes a control section 11, a storage section 12, a communication section 13, etc., and these sections are interconnected via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), or a GPU (Graphics Processing Unit). The control unit 11 appropriately executes a control program stored in the storage unit 12 to cause the server device 10 to perform various information processing, control processing, and the like that should be performed by the article identification device of the present disclosure.

The storage unit 12 includes a RAM (Random Access Memory), flash memory, hard disk, SSD (Solid State Drive), and the like. The storage unit 12 stores in advance control programs executed by the control unit 11 and various data necessary for executing the control programs. The storage unit 12 also temporarily stores data and the like generated when the control unit 11 executes the control program. The control programs stored in the storage unit 12 include an article identification program 12a, which is a program of the present disclosure. The storage unit 12 also stores an article classifier 12b of a CNN (Convolution Neural Network) model constructed by machine learning processing, for example. Based on the photographed image of the article, the article identifier 12b determines which of a plurality of pre-learned articles the article of the subject is from a plurality of feature amounts such as the shape and color of the appearance of the subject in the photographed image. is a neural network for identifying The server device 10 uses the article identifier 12b to identify the article of the subject in the captured image.

The communication unit 13 is an interface for connecting to the network N, and transmits and receives information to and from an external device via the network N.
The server apparatus 10 of this embodiment may include a display section, an input section, and the like in addition to the above-described configuration. The display unit displays various information according to instructions from the control unit 11 , and the input unit accepts operation inputs from the user and sends control signals corresponding to the operation contents to the control unit 11 . The server device 10 may acquire the article identification program 12 a and the article identifier 12 b from an external device via the communication section 13 and store them in the storage section 12 . Further, if the server device 10 includes a reading unit or the like for reading information stored in a portable storage medium, the article identification program 12a and the article identifier 12b may be read from the portable storage medium and stored in the storage section 12. .

The user terminal 20 is, for example, a smart phone, a tablet terminal, a personal computer, or the like. The user terminal 20 includes a control section 21, a storage section 22, a communication section 23, a display section 24, an input section 25, a camera 26, etc. These sections are interconnected via a bus. The control unit 21 includes one or more processors such as CPU, MPU or GPU. The control unit 21 appropriately executes control programs stored in the storage unit 22 to perform various information processing, control processing, and the like that should be performed by the user terminal 20 .

The storage unit 22 includes RAM, flash memory, hard disk, SSD, and the like. The storage unit 22 stores in advance control programs executed by the control unit 21 and various data necessary for executing the control programs. The storage unit 22 also temporarily stores data and the like generated when the control unit 21 executes the control program. The control programs stored in the storage unit 22 include an article inquiry application program (hereinafter referred to as an article inquiry application) 22a. The article inquiry application 22 a is acquired from an external device via the communication section 23 and stored in the storage section 22 , for example. Note that if the user terminal 20 has a reading unit or the like that reads information stored in a portable storage medium, the article inquiry application 22 a may be read from the portable storage medium and stored in the storage unit 22 .

The communication unit 23 is an interface for connecting to the network N, and transmits and receives information to and from an external device via the network N. The display unit 24 is, for example, a liquid crystal display or an organic EL display, and displays various information according to instructions from the control unit 21 . The input unit 25 is a touch sensor or push button arranged on the surface of the display unit 24 , receives operation input by the user, and sends a control signal corresponding to the operation content to the control unit 21 . The input unit 25, which is a touch sensor, constitutes a touch panel integrally with the display unit 24. FIG.

The camera 26 has a lens, an imaging element, etc., and acquires image data by photoelectrically converting light incident through the lens with the imaging element. The camera 26 takes an image according to an instruction from the control unit 21 and sequentially sends acquired image data (captured image) to the control unit 21 . It should be noted that the user terminal 20 may have a configuration in which the camera 26 is built in, or a configuration in which an external camera can be connected. In this case, the connection unit accepts input of image data acquired by the external camera and sequentially sends the input image data to the control unit 21 .
The user terminal 20 may be any information device that has a camera 26 , a display section 24 and a communication section 23 .

Next, the functions realized by the controller 11 executing the article identification program 12a in the server device 10 will be described. FIG. 3 is a block diagram showing functions realized by the control unit 11 of the server device 10, and FIG. When executing the item identification program 12a stored in the storage unit 12, the control unit 11 of the server device 10 implements the functions of the candidate identification unit 10a, the display information generation unit 10b, and the learning unit 10c. In this embodiment, each of these functions is implemented by the control unit 11 executing the article identification program 12a, but some of these functions may be implemented by dedicated hardware circuits.

In the article identification system 100 of the present embodiment, a user takes an image of an article such as a product using the user terminal 20 and transmits the obtained picked-up image to the server device 10 . A control unit (image acquisition unit) 11 of the server device 10 acquires a captured image from the user terminal 20 (external device) via the communication unit 13 . FIG. 4A shows an example of an image captured using the user terminal 20. FIG. The user terminal 20 captures an image of the appearance (package) of the article (product) as shown in FIG. 4A. The image shown in FIG. 4A is a captured image of orange juice in a PET bottle. For example, the captured image includes the hand of the user (photographer) holding the PET bottle and the background, but is omitted in FIG. 4A. .

The candidate identification unit 10a uses the article identifier 12b on the captured image acquired from the user terminal 20 to determine whether the article of the subject in the captured image is any product (article) that can be identified by the article identifier 12b. to identify It is assumed that the product classifier 12b of the present embodiment has already learned the image feature amount of a predetermined product including beverages and seasonings in PET bottles, for example. For example, the server device 10 acquires teacher data in which information indicating each product is labeled from a teacher image obtained by photographing pre-registered products (for example, a company's own product), and uses the teacher image to determine the product container. Machine learning processing is performed to learn feature amounts such as color and shape, and an article identifier 12b (learning model) for identifying each article is constructed. It is desirable to use images captured from a plurality of directions, such as an image captured from the front of the product and an image captured from the back, as the teacher image.

The article identifier 12b calculates a degree of certainty indicating the possibility that each pre-registered identifiable article is the subject in the captured image. The article identifier 12b calculates the degree of certainty using, for example, the softmax function, and the degree of certainty is indicated by a numerical value from 0 to 1. The candidate specifying unit 10a specifies products (hereinafter referred to as product candidates) that are highly likely to be the subject in the captured image based on the degree of certainty of each product calculated by the product identifier 12b. For example, the candidate identification unit 10a may extract a predetermined number (for example, three) of products in descending order of certainty and identify them as product candidates (article candidates) of the subject. Further, the candidate identification unit 10a may identify the product with the highest degree of certainty as the product candidate of the subject. Further, the candidate specifying unit 10a may specify products for which the certainty factor of each product calculated by the product identifier 12b is equal to or higher than a predetermined value as product candidates for the subject, and there is no product having a certainty factor equal to or higher than the predetermined value. In this case, it may be specified that there is no product candidate for the subject.

The display information generation unit 10b generates display information indicating the result of identification by the candidate identification unit 10a. Specifically, the display information generating unit 10b generates display information for displaying a sample image of each product candidate of the subject in the captured image specified by the candidate specifying unit 10a, as shown in FIG. 4B, for example. . The screen shown in FIG. 4B displays a captured image 24a captured using the user terminal 20, and a sample image 24b of each product specified as a subject product candidate by the candidate specifying unit 10a. The server device 10 transmits the display information generated by the display information generation unit 10b to the user terminal 20 from the communication unit (output unit) 13 as the result of identification by the candidate identification unit 10a. The control unit 21 of the user terminal 20 receives display information from the server device 10 via the communication unit 23, and displays a screen as shown in FIG. 4B based on the received display information.

Here, in the product identification system 100 of the present embodiment, the screen displayed on the user terminal 20 is configured so that a selection operation can be performed via the input unit 25 on the sample image 24b of the displayed product candidate. . The user of the user terminal 20 confirms the displayed sample image 24b, and if there is a sample image 24b of the product (item) captured by the user, performs a selection operation on this sample image 24b. When a selection operation is performed on the sample image 24b being displayed, the control unit 21 of the user terminal 20 accepts that the selected product candidate is the subject product (correct product). In the screen example shown in FIG. 4B, the user taps and selects the orange juice sample image 24b displayed on the left side, for example. Then, the control unit 21 receives correct information including information indicating the product candidate selected by the user via the input unit 25 and information indicating that the product candidate is the correct product. The user terminal 20 transmits the accepted correct answer information to the server device 10 .

The control unit (information acquisition unit) 11 of the server device 10 acquires the correct answer information transmitted by the user terminal 20 . The learning unit 10 c re-learns the article identifier 12 b based on the correct answer information received from the user terminal 20 and the captured image already received from the user terminal 20 . For example, the learning unit 10c uses the captured image received from the user terminal 20 as a teacher image, and generates teacher data by labeling the teacher image with the correct product information indicated by the correct answer information received from the user terminal 20. . Then, the learning unit 10c inputs the generated teacher data to the article identifier 12b. As a result, the product classifier 12b learns the feature quantities such as the color and shape of the product container from the teacher image and reconstructs the product container.

Next, article identification processing by each device in the article identification system 100 will be described. FIG. 5 is a flow chart showing a procedure of processing performed by the user terminal 20 and the server device 10. As shown in FIG. The following processing is executed by the control unit 21 according to the control program stored in the storage unit 22 of the user terminal 20 and executed by the control unit 11 according to the control program stored in the storage unit 12 of the server device 10 .

The user of the user terminal 20 instructs the activation of the article inquiry application 22a via the input unit 25. FIG. When the controller 21 of the user terminal 20 receives an activation instruction for the product inquiry application 22a via the input unit 25, it activates the product inquiry application 22a stored in the storage unit 22 (S1). When the product inquiry application 22a is activated, the control unit 21 activates the camera 26 (S2) and acquires a captured image with the camera 26 (S3). The control unit 21 acquires an image of the appearance (package) of the article (product) as shown in FIG. 4A, for example. The control unit 21 may acquire the captured image with the camera 26 when receiving an imaging instruction from the user via the input unit 25 . The activated camera 26 acquires an image at a predetermined timing, and the acquired image is sent to the display unit 24 and displayed. Therefore, the user can issue an imaging instruction at a desired timing while viewing the image displayed on the display unit 24 .

The control unit 21 transmits the acquired captured image to the server device 10 (S4). Note that the control unit 21 transmits the captured image together with the identification information of its own terminal 20 . The destination of the server device 10 is incorporated in, for example, the article inquiry application 22a. When transmitting the acquired captured image to the server device 10 , the control unit 21 may temporarily display the captured image on the display unit 24 and transmit it to the server device 10 after receiving a transmission instruction from the user. The control unit 11 of the server device 10 acquires the captured image transmitted by the user terminal 20 together with the identification information of the user terminal 20 (S5).

The control unit 11 uses the article identifier 12b for the captured image to identify which of the articles that can be identified by the article identifier 12b is the subject article in the captured image (S6). The control unit 11 identifies product candidates that are likely to be the subject in the captured image, and transmits the identification result to the user terminal 20 (S7). The control unit 11 generates display information for displaying a sample image of each product identified as a subject product candidate, for example, as shown in FIG. 4B, and transmits the display information to the user terminal 20 as a result of identification. The control unit 21 of the user terminal 20 acquires the identification result from the server device 10, and displays, for example, a display screen as shown in FIG. 4B on the display unit 24 (S8).

The control unit 21 receives a selection of a product candidate from the user via the input unit 25 on the display screen, and receives information indicating the selected product candidate and information indicating that the product candidate is the correct product. correct information including is received (S9). Then, the control unit 21 transmits the accepted correct answer information to the server device 10 (S10). The control unit 21 also transmits the correct answer information together with the identification information of the own terminal 20 here.

The control unit (information acquisition unit) 11 of the server device 10 acquires the correct answer information transmitted by the user terminal 20 together with the identification information of the user terminal 20 (S11). Then, the control unit 11 re-learns the article discriminator 12b based on the captured image acquired in step S5 and the correct answer information acquired in step S11 (S12). Note that the control unit 11 generates teacher data by labeling the captured image (teacher image) acquired in step S5 with the correct product information indicated by the correct answer information acquired in step S11, and stores it in the storage unit 12. A configuration may be employed in which the training data is stored, and after a certain amount of teacher data is accumulated, the article classifier 12b is re-learned.

In the present embodiment, the server device 10 identifies the subject article (product) based on the image captured by the user using the user terminal 20, and the user inputs whether or not the identified product is correct. . Thereby, the user can evaluate the product identification result by the server device 10 . In addition, the product classifier 12b can sequentially learn using the user's evaluation results. Furthermore, since images captured by the user using the user terminal 20 are used as training data, captured images of articles captured in different imaging environments such as the background of the subject, the imaging direction with respect to the subject, and the brightness of the surroundings are used as training data. can be used for Since the article discriminator 12b is re-learned using such teacher data, the accuracy of article identification by the article discriminator 12b can be improved.

(Embodiment 2)
An article identification system according to the second embodiment will be described. In this embodiment, an article identification system that provides information about the identified article after identifying an article as a subject based on a captured image of an article such as a product will be described. Each device of the article identification system of the present embodiment has the same configuration as each device of the first embodiment, so the description of the configuration is omitted.

An article identification process in the article identification system 100 of this embodiment will be described. FIG. 6 is a flow chart showing a part of the procedure of processing performed by the user terminal 20 and the server device 10, and FIG. The process shown in FIG. 6 is obtained by adding the processes of steps S21 and S22 after the process of step S12 in the process of the first embodiment shown in FIG. In FIG. 6, illustration of steps S1 to S10 in FIG. 5 is omitted.

In the article identification system 100 of this embodiment, the user terminal 20 and the server device 10 perform the processes of steps S1 to S12 in FIG. Thereafter, the control unit (article information output unit) 11 of the server device 10 outputs to the user terminal 20 product information (article information) corresponding to the product indicated by the correct answer information acquired in step S11 (S21). The product information corresponding to each product that can be identified by the product identifier 12b is stored in the storage unit 12 in advance, and the control unit 11 reads the product information corresponding to the product indicated by the correct answer information from the storage unit 12. Output to the user terminal 20 . For example, as shown in FIG. 7, the control unit 11 displays a sample image 24c of the product selected by the user as the correct answer, displays a link L1 to a website that provides product information on this product, and displays a link L1 to a website that sells this product. A display screen on which a link L2 to a sales site that sells products is set is generated and output as product information.

The control unit 21 of the user terminal 20 acquires the product information from the server device 10, displays, for example, a display screen as shown in FIG. 7 on the display unit 24 (S22), and ends the process. When one of the links L1 and L2 is selected via the input unit 25 on the screen shown in FIG. Get information or the web page of the sales site. In the screen shown in FIG. 7, the product information itself may be displayed instead of the link L1 for acquiring the product information, and the web page of the sales site may be displayed instead of the link L2 to the product sales site. may

In this embodiment, the same effects as those of the first embodiment described above can be obtained. In addition, in the present embodiment, the user can confirm information corresponding to the product imaged using the user terminal 20 on the user terminal 20 .

(Embodiment 3)
An article identification system according to Embodiment 3 will be described. In this embodiment, an article identification system that identifies the respiratory system of a subject based on an image obtained by imaging an inhaler for inhaling a drug for respiratory diseases such as asthma at home will be described. Each device of the article identification system of the present embodiment has the same configuration as each device of the first embodiment, so the description of the configuration is omitted. In the server device 10 of the present embodiment, it is assumed that the article identifier 12b has already learned the image feature amount of an inhaler for inhaling medicine for respiratory diseases. For example, the server device 10 acquires teacher data in which information indicating each inhaler is labeled from a pre-registered teacher image obtained by imaging an inhaler, and acquires the color and shape of the appearance of the inhaler from the teacher image. Machine learning processing is performed to learn feature amounts such as , etc., and an article identifier 12b that identifies each inhaler is constructed.

An article identification process in the article identification system 100 of this embodiment will be described. FIG. 8 is a flowchart showing a part of the procedure of processing performed by the user terminal 20 and the server device 10, and FIG. In the process shown in FIG. 8, in the process of the first embodiment shown in FIG. 5, the processes of steps S31 to S34 are performed instead of the processes of steps S7 to S8. In FIG. 8, illustration of steps S1 to S4 in FIG. 5 is omitted.

In the article identification system 100 of this embodiment, the user terminal 20 and the server device 10 perform the processes of steps S1 to S6 in FIG. Then, the control unit 11 of the server device 10 identifies a plurality of inhalers (hereinafter referred to as inhaler candidates) that are highly likely to be the subject in the captured image based on the identification result by the article identifier 12b (S31). . Here, for example, the confidence calculated for each inhaler that can be identified by the article identifier 12b is equal to or greater than a predetermined value, and a predetermined number (for example, two) of inhalers are identified as inhaler candidates in descending order of confidence. do.

The control unit 11 of the server device 10 of the present embodiment generates display information for displaying a sample image of each inhaler identified as a subject inhaler candidate (S32). For example, as shown in FIG. 9A, the control unit 11 captures an image 24a acquired by the user terminal 20 (camera 26) and a sample image 24b of each inhaler identified as an inhaler candidate by the server device 10. Generate a display screen to display. Here, in the present embodiment, the control unit 11 generates a display screen that displays the sample image 24b of each inhaler candidate in a manner corresponding to the degree of certainty calculated by the article identifier 12b. FIG. 9A shows an example in which the sample image 24b of each inhaler candidate is displayed larger as the confidence calculated by the article identifier 12b increases. In the example shown in FIG. 9A, the left sample image 24b has a higher confidence than the right sample image 24b, so the left sample image 24b is larger than the right sample image 24b. is displayed. Note that, as shown in FIG. 9B, the control unit 11 may generate a display screen in which the sample image 24b of each inhaler candidate is added with the degree of certainty calculated by the article identifier 12b. In FIG. 9B, confidence factors of 0.6 and 0.3 are displayed in association with each inhaler candidate.

The control unit 11 transmits the generated display information to the user terminal 20 (S33). The control unit 21 of the user terminal 20 acquires the display information from the server device 10 and displays the display screens shown in FIGS. 9A and 9B on the display unit 24 (S34). In the screens shown in FIGS. 9A and 9B as well, by performing a selection operation on the displayed sample image 24b via the input unit 25, the selected inhaler candidate is the subject's inhaler (the correct inhaler). It is configured to accept that Therefore, the control unit 21 of the user terminal 20 and the control unit 11 of the server device 10 perform the processes after step S9. It should be noted that in step S9, the control unit 21 of the user terminal 20 displays the selected inhaler candidate in the screens shown in FIGS. Correct information including information and information indicating that this inhaler candidate is the correct inhaler is accepted.

In this embodiment, the same effects as those of the first embodiment described above can be obtained. In addition, in this embodiment, the sample image 24b of each inhaler candidate can be displayed in a manner corresponding to the degree of certainty calculated by the article identifier 12b. can. As a result, when the user selects the correct inhaler from the displayed inhaler candidates, it is possible to more easily select an appropriate inhaler.

The configuration of the present embodiment can also be applied to the second embodiment, and even if it is applied to the second embodiment, the same effects as those of the second embodiment can be obtained. When applied to the second embodiment, for example, the storage unit 12 of the server device 10 stores inhaler information such as usage and handling methods of each inhaler. Then, the server device 10 outputs inhaler information such as how to use and handle the inhaler to the user terminal 20 according to the inhaler indicated by the correct answer information acquired from the user terminal 20 . Thereby, the user can check the information about the subject's inhaler imaged using the user terminal 20 on the user terminal 20 . Therefore, the user can reconfirm how to use and handle the inhaler at home, etc., and can avoid using it in the wrong way. The information about the inhaler may be screen information with a link to a predetermined website, as shown in FIG.

(Embodiment 4)
An article identification system according to Embodiment 4 will be described. In the present embodiment, a system for identifying a subject's respiratory system based on an image obtained by imaging an inhaler, which is an article identifying system used by users who have registered in advance, will be described. Each device of the article identification system of the present embodiment has the same configuration as each device of the first embodiment, so the description of the configuration is omitted. Note that the server device 10 of this embodiment stores in the storage unit 12 a user information database (hereinafter referred to as a user information DB) in which user information of users registered as users for using the article identification system 100 is registered. is doing.

FIG. 10 is a schematic diagram showing a configuration example of a user information DB. User IDs of registered users, passwords for user authentication, and drug information are associated and registered in the user information DB. In the server device 10, the control unit 11 updates the registration contents of the user information DB when receiving a change or addition of user information from the user terminal 20 via the communication unit 13, for example. Note that the drug information includes, for example, information on drugs prescribed to each user and the timing of taking the drug, and may be registered in the user information DB of the server device 10 by the attending physician who prescribed the drug. The drug information may also include inhaler information such as how to use and handle the inhaler used to take the drug. The user information DB may be stored in an external storage device or the like connected to the server device 10, or may be stored in another storage device connected to the server device 10 via the network N. FIG.

An article identification process in the article identification system 100 of this embodiment will be described. 11 and 12 are flowcharts showing procedures of processing performed by the user terminal 20 and the server apparatus 10, and FIG. 13 is a schematic diagram showing an example of a screen of the user terminal 20. FIG.
When the controller 21 of the user terminal 20 receives an activation instruction for the product inquiry application 22a via the input unit 25, it activates the product inquiry application 22a stored in the storage unit 22 (S41). When the article inquiry application 22a is activated, the control unit 21 activates the camera 26 (S42) and acquires a captured image (S43). The control unit 21 displays an input screen for inputting user information on the display unit 24 (S44). For example, as shown in FIG. 13A, the control unit 21 displays the captured image 24d acquired by the camera 26 in step S43, and displays an input screen having input fields for user ID and password.

The user inputs a user ID and password via the input unit 25 on the input screen. Thereby, the control unit 21 receives the user information such as the user ID and the password via the input unit 25 (S45). A send button is displayed on the input screen shown in FIG. 13A , and the control unit 21 of the user terminal 20 accepts a send instruction when the user operates the send button via the input unit 25 . When receiving the transmission instruction, the control unit 21 transmits the captured image acquired in step S43 and the user information received in step S45 to the server device 10 (S46). Note that the control unit 21 may transmit the captured image and the user information separately to the server device 10 . For example, the control unit 21 may first transmit the user information to the server device 10 , and when the user is authenticated by the server device 10 , the control unit 21 may transmit the captured image to the server device 10 . Further, once the user is authenticated by the server device 10 , the control section 21 may transmit only the captured image to the server device 10 .

The control unit 11 of the server device 10 acquires the captured image and user information transmitted by the user terminal 20 (S47). The control unit 11 uses the article identifier 12b for the captured image to identify which of the inhalers identifiable by the article identifier 12b is the subject (article) in the captured image (S48). ). Note that the control unit 11 may perform user authentication based on the user information acquired from the user terminal 20, and if the user has been authenticated, the process from step S48 onward may be performed. Specifically, the control unit 11 compares and collates the user ID and password acquired from the user terminal 20 with the registration information of the user information DB, and if they are registered in the user information DB, it may be determined that the authentication has been successful. .

When the user is authenticated, the control unit 11 considers the drug information included in the user information stored in the user information DB, and determines whether the subject in the captured image is an inhaler that can be identified by the article identifier 12b. You may specify which one. The subject in the captured image is likely to be an inhaler for taking medicine prescribed by the user who captured this image. Therefore, if there is any drug information included in the user information among the inhaler candidates identified by the article identifier 12b, there is a high possibility that the subject is the inhaler indicated by the drug information. Therefore, the control unit 11 extracts the inhaler included in the drug information of the user information from the inhaler candidates identified by the article identifier 12b, and identifies the extracted inhaler as the subject's inhaler candidate.

The control unit 11 transmits the identification result to the user terminal 20 (S49). Here, the control unit 11 generates and transmits display information for displaying a screen as shown in FIG. 4B. Here, in the screen shown in FIG. 4B, the product candidate sample image 24b is a sample image of an inhaler. The control unit 21 of the user terminal 20 acquires the identification result from the server device 10 and displays a display screen as shown in FIG. 4B on the display unit 24 (S50). When the control unit 21 receives a selection of an inhaler candidate from the user via the input unit 25 on the display screen, the control unit 21 receives information indicating the selected inhaler candidate and information indicating that the inhaler candidate is correct. The correct answer information including and is received (S51). Then, the control unit 21 transmits the accepted correct answer information to the server device 10 (S52).

The control unit 11 of the server device 10 acquires the correct information transmitted by the user terminal 20 (S53), and reproduces the article identifier 12b based on the captured image acquired in step S47 and the correct information acquired in step S53. Learning is performed (S54). After that, the control unit 11 of the server device 10 reads drug information registered in the user information DB for the inhaler indicated by the correct answer information acquired in step S53, and outputs it to the user terminal 20 (S55). The control unit 11 generates display information for displaying a screen such as that shown in FIG. 13B and outputs the display information to the user terminal 20 . On the screen shown in FIG. 13B, a sample image 24e of the inhaler selected by the user as the correct answer and information on the timing of taking the medicine using this inhaler are displayed. and a link L4 to a website providing drug information on drugs to be taken with this inhaler.

The control unit 21 of the user terminal 20 acquires the drug information from the server device 10, displays a display screen as shown in FIG. 13B on the display unit 24 (S56), and ends the process. Note that, on the screen shown in FIG. 13B, the method of using the inhaler itself may be displayed instead of the link L3, and the drug information itself regarding the drug may be displayed instead of the link L4.

In this embodiment, the same effects as those of the first embodiment described above can be obtained. Further, in the present embodiment, by performing user registration in advance, the user of the user terminal 20 is provided with not only information related to the subject (inhaler) imaged by the user using the user terminal 20 but also information corresponding to the user. be able to. Further, in this embodiment, the server device 10 accumulates the reception history (specification history) of the object (product, medicine) identified based on the captured image from the user terminal 20, and stores the accumulated reception history for each user. may be treated as the usage history or purchase history of the goods. In this case, for example, the server device 10 calculates the use frequency or purchase frequency of each user's goods from the accumulated reception history, and transmits an article advertisement message or the like to each user based on the calculated use frequency or purchase frequency. be able to. Therefore, it is possible to advertise for each user, and to efficiently advertise to users who have used or purchased the product. The configuration of this embodiment can also be applied to the second and third embodiments, and even when applied to the second and third embodiments, the same effects as those of the second and third embodiments can be obtained.

(Embodiment 5)
An article identification system according to Embodiment 5 will be described. Each device of the article identification system of the present embodiment has the same configuration as each device of the first embodiment, so the description of the configuration is omitted. FIG. 14 is a schematic diagram showing a screen example of the user terminal 20. As shown in FIG. In the present embodiment, when the server device 10 transmits to the user terminal 20 the identification result of the product candidate (inhaler candidate) by the product identifier 12b, the control unit 11 displays a screen as shown in FIG. generate and send display information for On the screen shown in FIG. 14, a sample image 24b of each inhaler candidate identified as an inhaler candidate is displayed in the same direction as the subject in the captured image 24a (imaging direction with respect to the subject). In the server device 10 of the present embodiment, the storage unit 12 stores, as the sample images 24b, images of articles (inhalers) identifiable by the article identifier 12b, which are captured from a plurality of directions.

In the article identification system of this embodiment, the user terminal 20 and the server device 10 perform the same processing as the processing shown in FIG. 5 described in the first embodiment. Note that in step S7 in FIG. 5, the control unit 11 of the server device 10 of the present embodiment, as shown in FIG. Display information for displaying in the same orientation as the subject in the captured image 24a is generated. For example, the control unit 11 reads from the storage unit 12 sample images 24b captured in the same orientation as the subject in the captured image 24a from among the plurality of sample images 24b of the inhaler candidates, and uses the read sample images 24b to display information. to generate In addition, the control unit 11 performs image processing on the sample image 24b of the inhaler candidate so that it faces in the same direction as the subject in the captured image 24a, and generates display information using the processed sample image 24b. good too. Also, as the sample image 24b, a 3D model image obtained by synthesizing images obtained by capturing the entire circumferential surface of each inhaler may be used. In this case, the control unit 11 generates a 2D image of the inhaler viewed from the same direction as the subject in the captured image 24a from the sample image 24b of the 3D model image, and uses the generated 2D image to display the display information. may be generated.

In the present embodiment, the sample image 24b of each inhaler identified as an inhaler candidate by the server device 10 is displayed in a state corresponding to the imaging state of the subject imaged by the user terminal 20 . Specifically, since the sample image 24b is displayed in the same orientation as the subject imaged by the user terminal 20, each inhaler candidate can be displayed more easily. As a result, when the user selects the correct inhaler from the displayed inhaler candidates, it is possible to more easily select an appropriate inhaler.

In this embodiment, when a 3D model image of each product (inhaler) is used as the sample image of each product (inhaler), for example, the user performs an operation such as a slide operation on the sample image 24b on the display screen. By doing so, the orientation of the sample image 24b being displayed can be changed. For example, when the user operates the sample image 24b on the display screen, the user terminal 20 transmits operation information related to the user's operation to the server device 10 . Then, the control unit 11 of the server device 10 rotates the sample image 24 b in the orientation corresponding to the received operation information and transmits the rotated sample image 24 b to the user terminal 20 . This allows the user terminal 20 to display the sample image 24b in which the product (inhaler) is viewed in a three-dimensional direction desired by the user. Therefore, when the user selects the correct answer from the displayed product candidates (inhaler candidates), it is easy to distinguish between the product candidates, and selection errors can be suppressed. Note that the control unit 21 of the user terminal 20 may perform processing to rotate the sample image 24b being displayed according to the user's operation.

(Embodiment 6)
An article identification system according to Embodiment 6 will be described. In the present embodiment, an article identification system will be described that, after identifying an article as a subject based on a captured image of an article such as a product, provides information on the article, a sales site, and an inquiry address for the identified article. Each device of the article identification system of the present embodiment has the same configuration as each device of the first embodiment, so the description of the configuration is omitted.

An article identification process in the article identification system 100 of this embodiment will be described. 15 and 16 are flowcharts showing the procedures of processing performed by the user terminal 20 and the server apparatus 10, and FIG. 17 is a schematic diagram showing an example of the screen of the user terminal 20. FIG. In the article identification system 100 of the present embodiment, when the controller 21 of the user terminal 20 receives an instruction to activate the article inquiry application 22a via the input section 25, the article inquiry application 22a stored in the storage section 22 is activated. Start up (S61). When the article inquiry application 22a is started, the control section 21 displays a selection screen as shown in FIG. 17 on the display section 24 (S62). The selection screen shown in FIG. 17 is a screen for selecting a process to be performed regarding an item specified by the item identification system 100, such as a process of checking how to use the item, a process of buying the same item, and a process of making an inquiry about the item. can be selected.

The control unit 21 receives the process selected by the user via the input unit 25 on the selection screen (S63). Then, the control unit 21 activates the camera 26 (S64), and acquires a captured image with the camera 26 (S65). The control unit 21 transmits the acquired captured image and processing information indicating the processing accepted in step S63 to the server device 10 (S66). The control unit 11 of the server device 10 acquires the captured image and the processing information transmitted by the user terminal 20 (S67). The control unit 11 performs article identification using the article identifier 12b on the acquired captured image (S68), and identifies an article that is highly likely to be the subject in the captured image. The control unit 11 transmits the identification result to the user terminal 20 (S69).

The control unit 21 of the user terminal 20 acquires the identification result from the server device 10, and displays, for example, a display screen as shown in FIG. 4 on the display unit 24 (S70). The control unit 21 receives correct information including information indicating the product candidate selected by the user via the input unit 25 on the display screen and information indicating that the product candidate is the correct product (S71), The received correct answer information is transmitted to the server device 10 (S72). The control unit 11 of the server device 10 acquires the correct information transmitted by the user terminal 20 (S73), and identifies the article based on the captured image acquired from the user terminal 20 in step S67 and the correct information acquired in step S73. Re-learning of the device 12b is performed (S74).

Further, the control unit 11 acquires product information corresponding to the process indicated by the processing information obtained from the user terminal 20 in step S67 for the product indicated by the correct answer information obtained in step S73 (S75). Product information corresponding to each process for each product is stored, for example, in the storage unit 12 , and the control unit 11 reads the product information from the storage unit 12 . The product information corresponding to each process of each product may be stored in an external storage device connected to the server device 10 or an external storage device capable of communicating with the server device 10 via the network N, or the like. In this case, the control unit 11 acquires product information from an external storage device. The control unit 11 outputs the acquired product information to the user terminal 20 (S76).

The control unit 21 of the user terminal 20 acquires the product information from the server device 10 and displays the display screen on the display unit 24 (S77). The display screen here displays information according to the process selected by the user via the selection screen shown in FIG. For example, when the process of checking how to use an article is selected via the selection screen shown in FIG. 17, the user terminal 20 obtains from the server device 10 and displays the display screen on which the link L1 in FIG. 7 is set. . 17, the user terminal 20 obtains from the server device 10 and displays the display screen on which the link L2 in FIG. 7 is set. . In addition, in each display screen, the product information itself including how to use the product may be displayed instead of the link L1, and the web page of the sales site may be displayed instead of the link L2. 17, the user terminal 20 makes an inquiry via the display screen displaying the telephone number of the inquiry destination or via the network N. A display screen or the like on which a link to a website is set is obtained from the server device 10 and displayed. Note that the user terminal 20 may make a call to a predetermined call center.

In this embodiment, the same effects as those of the first embodiment described above can be obtained. Further, in the present embodiment, when the user terminal 20 activates the item inquiry application 22a, the user selects the desired process for the imaged item (merchandise). It can be confirmed on the terminal 20 . The configuration of the present embodiment can also be applied to Embodiments 2 to 5, and even when applied to Embodiments 2 to 5, the same effects as those of Embodiments 2 to 5 can be obtained. In addition, when applied to the second to fifth embodiments, the user terminal 20 displays a selection screen as shown in FIG. may be configured to accept the selection of .

(Embodiment 7)
An article identification system according to Embodiment 7 will be described. In the present embodiment, whether or not to use teacher data obtained by labeling captured images (teacher images) received by the server device 10 from the user terminal 20 based on correct information for re-learning the article classifier 12b is determined by: An article identification system that switches according to correct information will be described. Each device of the article identification system of the present embodiment has the same configuration as each device of the first embodiment, so the description of the configuration is omitted.

The user terminal 20 and the server device 10 of this embodiment perform the same processing as that of the first embodiment shown in FIG. Note that after the process of step S11 in FIG. 5, the server device 10 of the present embodiment stores the captured image obtained from the user terminal 20 in step S5 and the correct answer information obtained from the user terminal 20 in step S11. The data is stored in a reception history database (hereinafter referred to as reception history DB) stored in the unit 12 . FIG. 18 is a schematic diagram showing a configuration example of a reception history DB. In the reception history DB, user identification information for identifying the user terminal 20 or the user who sent the captured image and the correct answer information, the received captured image and the correct answer information are registered in association with each other. The correct answer information includes a product ID indicating a product candidate selected as the correct product by the user on the user terminal 20, and the degree of certainty calculated by the product identifier 12b of the server device 10 for this product candidate. . Note that when the identification result transmitted from the server device 10 to the user terminal 20 includes the degree of certainty of each product candidate, the correct information transmitted from the user terminal 20 to the server device 10 includes the correct product (the selected product candidate) may be included.

The server device 10 of the present embodiment accumulates reception histories as shown in FIG. A determination process is performed to determine whether or not the item should be learned by the item classifier 12b. This determination process may be performed periodically, for example, every day or every week, or every time a predetermined amount of reception history is accumulated.

FIG. 19 is a flow chart showing the procedure of determination processing performed by the server device 10 . When the timing for performing the judgment process arrives, the control unit 11 of the server device 10 reads correct information corresponding to one piece of user identification information from the reception history DB shown in FIG. 18 (S81). In the read correct information, the control unit 11 counts the number of correct information whose certainty is less than a predetermined value (for example, less than 0.5) and the number of correct information whose certainty is equal to or higher than a predetermined value. A ratio of correct information less than a predetermined value is calculated (S82). The control unit 11 determines whether the calculated ratio is a predetermined ratio or more (for example, 60% or more) (S83). , is stored in the storage unit 12 as information of users or user terminals 20 not to be learned by the article identifier 12b (S84). If the percentage of correct information with a degree of certainty less than a predetermined value is greater than or equal to a predetermined percentage, it is possible that the user has maliciously selected a product candidate that is not the correct answer as being the correct product candidate. highly sexual. Therefore, information (captured images and correct information) from users who have a high rate of selecting product candidates with a low degree of certainty as being correct products is excluded from learning objects of the product classifier 12b.

When determining that the ratio calculated in step S82 is less than the predetermined ratio (S83: NO), the control unit 11 skips the process of step S84. The control unit 11 determines whether or not processing has been completed for all reception histories stored in the reception history DB (S85). The process returns to step S81. Then, the control unit 11 performs the processing of steps S81 to S84 for unprocessed user identification information. If it is determined that the process has been completed for all reception histories (S85: YES), the control unit 11 ends the process. Through such processing, the server device 10 can store in the storage unit 12 identification information of users or user terminals 20 that should not be learned by the article identifier 12b.

As described above, in the present embodiment, the server device 10 stores in the storage unit 12 user identification information of users or user terminals 20 that should not be learned by the article identifier 12b. Article identification processing in the article identification system 100 having such a configuration will be described. FIG. 20 is a flow chart showing part of the procedure of processing performed by the user terminal 20 and the server device 10 . The process shown in FIG. 20 is obtained by adding the process of step S91 between the process of step S11 and the process of step S12 in the process of the first embodiment shown in FIG. In FIG. 20, illustration of steps S1 to S10 in FIG. 5 is omitted.

In the article identification system 100 of this embodiment, the user terminal 20 and the server device 10 perform the processes of steps S1 to S11 in FIG. Then, the control unit 11 of the server device 10 determines whether the transmission source of the captured image acquired in step S5 and the correct information acquired in step S11 is the user or the user terminal 20 to be learned by the article identifier 12b. (S91). Specifically, the control unit 11 identifies the identification information of the user terminal 20 of the transmission source or the user of the user terminal 20, and identifies the user or user terminal whose identified identification information should not be learned by the article identifier 12b. 20 is stored in the storage unit 12 or not. If the specified identification information is not stored in the storage unit 12 as the identification information of the user or user terminal 20 that should not be the learning target, the transmission source here is the user or the user terminal 20 that should be the learning target. to decide.

When the control unit 11 determines that the transmission source is the user or the user terminal 20 to be studied (S91: YES), the process proceeds to step S12, and the captured image and the correct answer information acquired from the user terminal 20 Based on this, the article classifier 12b is re-learned (S12). If it is determined that the transmission source is not the user or the user terminal 20 that should be the learning target (S91: NO), the control unit 11 skips the processing of step S12 and ends the processing.

In this embodiment, there is a possibility that an incorrect product candidate was intentionally selected as the correct product candidate for the subject identified by the product identifier 12b of the server device 10 based on the image captured by the user terminal 20. can identify users with high In addition, such teacher data obtained by the user can be excluded from the learning target of the article identifier 12b, thereby avoiding deterioration in article identification accuracy by the article identifier 12b. The configuration of the present embodiment can also be applied to Embodiments 2 to 5, and even when applied to Embodiments 2 to 5, the same effects as those of Embodiments 2 to 5 can be obtained.

In each of the above-described embodiments, the server device 10 identifies which product is based on an image of a beverage product, and the server device 10 identifies which inhaler is based on an image of an inhaler. We have described the configuration that identifies the presence. The article identification system 100 of this embodiment can be applied to various article identification processes. For example, the types and classifications of objects identifiable by the article identifier 12b are determined, and training data labeled with information indicating each object is used for training images in which the objects are imaged for each type and classification. By performing machine learning processing in advance, an article discriminator 12b is constructed for each type and classification. Also, by appropriately installing an article inquiry application 22a corresponding to each article identifier 12b in the user terminal 20, communication is made with the server device 10 having the corresponding article identifier 12b via each article inquiry application 22a. Then, it becomes possible to perform the article specifying process. The types of objects can be various, such as beverages, seasonings, sweets, detergents, cosmetics, inhalers, medicines, and the like.

In each of the above-described embodiments, when re-learning the article identifier 12b of the server device 10, the article identifier 12b has already calculated a certainty factor of 0.8 or more for the article selected by the user as being correct. If so, the captured image and the teacher data based on the correct answer information at this time may be excluded from learning objects of the article classifier 12b. In this way, it may be configured such that whether or not to use an image captured by the user terminal 20 as teacher data is appropriately switched when the article classifier 12b is re-learned.

In each of the above-described embodiments, the server device 10 identifies the article candidate of the subject in the captured image acquired from the user terminal 20, transmits the identification result to the user terminal 20, and sends correct information indicating the correct article to the user. The configuration for acquiring from the terminal 20 has been described. In addition, for example, the server device 10 uses the article identifier 12b to capture images acquired from an imaging device such as a camera connected via a communication cable, or captured images captured by a built-in camera. It is also possible to use a configuration in which the article candidate of the subject is identified by Further, the server device 10 may display the identification result on the display unit provided in the device 10, and select the correct item from the displayed item candidates by using the input unit provided in the device 10. may be accepted through Even with such a configuration, the server device 10 can acquire the user's evaluation of the result of specifying the article using the article identifier 12b. In addition, since the article classifier 12b can be re-learned by using captured images obtained in various imaging environments as training data, it is possible to improve the accuracy of product identification by the article classifier 12b.

The embodiments disclosed this time are illustrative in all respects and should be considered not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the meaning described above, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

100 article identification system 10 server device (article identification device, information processing device)
11 control unit 12 storage unit 20 user terminal (external device)
21 control unit 22 storage unit 10a candidate identification unit 10c learning unit 12a article identification program 12b article identifier

Claims (7)

an image acquisition unit that acquires an image;
a candidate identification unit that identifies an article candidate present in the acquired image using a classifier;
an output unit that outputs information related to the identified article candidate;
an information acquisition unit that acquires correct information indicating whether or not the output product candidate is correct;
a learning unit that re-learns the classifier using the acquired correct information and the image acquired by the image acquisition unit,
The candidate specifying unit calculates a certainty factor indicating a possibility that each of the plurality of article candidates is the article in the image, and specifies a plurality of article candidates with a high calculated certainty factor,
The output unit outputs an instruction to highlight and display the identified item candidates according to the degrees of certainty,
The article identification device, wherein the information acquisition unit acquires correct information indicating an article that is a correct answer among the plurality of article candidates. The article identification device according to claim 1, further comprising an article information output section that outputs article information corresponding to a correct article based on the correct information acquired by the information acquisition section. The image acquisition unit acquires the image and user information,
3. The article identification device according to claim 2, wherein the article information output unit outputs the article information according to the acquired user information. The image acquisition unit acquires the image and user information,
4. The output unit extracts product candidates corresponding to the user information from the product candidates specified by the candidate specifying unit, and outputs information related to the extracted product candidates. The article identification device according to any one of . 5. The item according to any one of claims 1 to 4, wherein the output unit outputs an image of the item candidate identified by the candidate identification unit according to an imaging direction of the item in the image. Article identification device. The classifier is a classifier that has learned the feature amount of each of a plurality of articles from a plurality of article images,
The article identification device according to any one of claims 1 to 5, wherein the candidate identification unit identifies an article candidate in the image from the plurality of articles. The image acquisition unit acquires a captured image captured by an external device,
The article identification device according to any one of claims 1 to 6, wherein the candidate identification unit identifies an article candidate corresponding to the subject in the acquired captured image.

Images