WO2021176432A1 - Intelligent food image recognition and recommendation method - Google Patents
Intelligent food image recognition and recommendation method Download PDFInfo
- Publication number
- WO2021176432A1 WO2021176432A1 PCT/IB2021/052100 IB2021052100W WO2021176432A1 WO 2021176432 A1 WO2021176432 A1 WO 2021176432A1 IB 2021052100 W IB2021052100 W IB 2021052100W WO 2021176432 A1 WO2021176432 A1 WO 2021176432A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- food
- user
- image
- database
- nutrition
- Prior art date
Links
- 235000013305 food Nutrition 0.000 title claims abstract description 141
- 238000000034 method Methods 0.000 title claims abstract description 17
- 230000036541 health Effects 0.000 claims abstract description 35
- 235000016709 nutrition Nutrition 0.000 claims abstract description 24
- 230000035764 nutrition Effects 0.000 claims abstract description 13
- 235000005911 diet Nutrition 0.000 claims abstract description 11
- 235000015097 nutrients Nutrition 0.000 claims abstract description 11
- 230000037213 diet Effects 0.000 claims abstract description 9
- 230000008859 change Effects 0.000 claims abstract description 4
- 239000000463 material Substances 0.000 claims description 10
- 235000012631 food intake Nutrition 0.000 claims description 3
- 235000001497 healthy food Nutrition 0.000 claims description 3
- 230000003862 health status Effects 0.000 claims description 2
- 238000012552 review Methods 0.000 claims description 2
- 230000000378 dietary effect Effects 0.000 claims 2
- 235000013361 beverage Nutrition 0.000 claims 1
- 238000013136 deep learning model Methods 0.000 claims 1
- 230000007774 longterm Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000004615 ingredient Substances 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 206010017993 Gastrointestinal neoplasms Diseases 0.000 description 1
- 206010020751 Hypersensitivity Diseases 0.000 description 1
- 208000008589 Obesity Diseases 0.000 description 1
- 230000002009 allergenic effect Effects 0.000 description 1
- 230000007815 allergy Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000036772 blood pressure Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000004186 food analysis Methods 0.000 description 1
- 235000004280 healthy diet Nutrition 0.000 description 1
- 208000019622 heart disease Diseases 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000002503 metabolic effect Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 235000006286 nutrient intake Nutrition 0.000 description 1
- 235000020824 obesity Nutrition 0.000 description 1
- 102000004169 proteins and genes Human genes 0.000 description 1
- 108090000623 proteins and genes Proteins 0.000 description 1
- 230000004580 weight loss Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/60—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to nutrition control, e.g. diets
Definitions
- the present invention relates to the field of image processing technology and the field of healthy eating, specifically a way to determine the amount of food consumed and provide a specific diet to maintain health based on a person's medical history.
- users enter their profile on the web portal or mobile application and create a profile for themselves.
- This profile includes details such as national ID number; age and height and weight and other physical parameters.
- users can enter their health data such as blood pressure or disease records and allergies cases. This can be done manually or by connecting to the local health system database to enter the person's medical/health data into the system.
- the said data can store in the personal health parameter database (30) and can be updated occasionally.
- Fig.1 shows the system has at least three different database including food Image database or Food material (10), Food nutritional values (20), personal health parameters database (30).
- a cellular phone or other electronic devices being used to capturing image of food, uploading the image files to an image server via internet.
- the food and its volume are identified. This can be done with the image recognition system of the food or with entering the text (for example name of the food) or combination of the said two methods. If food is identified, the first output is displayed to the user, according to the type of food and the user health profile. In the first output, brief nutrient- specific information such as calorie content, fat and protein as well as the food name is provided to the user as well as general recommendation about the consumption of the food. At first output the system alert about the health risk of desired food based from personal health parameters database (30).
- Fig.2 shows if identified food is harmful for the user health a similar healthy food will be proposed to the user.
- Fig. 2 shows If the desired food is not available in the database, the system administrator will be notified to add the said food to the list and therefore the detection system will be approved.
- the image recognition can be done by segmenting the food images by an image segmentation algorithm. The similarity matching operation is carried out on the image to be recognized and the food images in a database, the food image with the highest similarity is obtained, the food label corresponding to the food image with the highest similarity serves as the food label of the image to be recognized and this food label display to the users as a text or an image. If the said result is not acceptable or wrong user can enter the name of the food manually as shown in Fig.2.
- Image recognition done through the image identifying system, searching to confirm whether or not the image identifying parameters for food (including type, ingredient, quantity and size of food) built in the food image database (10), analyzing food nutrients of image identifying parameters via the Food nutritional values (20) of the system.
- users may recognize what nutrients contain in the food they would like to eat at once, and take such data to be reference basis for their food management. Still further, users can make their long-term personal records of nutrient absorbing amount of food for management over their daily health.
- User Health Parameters are taken from the User Health Profile and Nutritional value is taken from the Food Nutrition Values.
- Pre Trained Deep Learning decides whether food is suitable for the user. If the food is suitable, the food is approved for the user and the amount of food and other information is informed to the user. Even in this case, the user is asked whether s/he likes the food. If the food is not suitable for the user, healthy food is recommended to the person through the Intelligent Food Recommendation.
- the database will contain the following items: Food nutritional values, Food materials, User health Profiles, User Diets, User Food reviews, User Food history. Where the system scores all foods with Nutritional Values versus User Health Profile.
- the volume of food consumption can be estimated by including an object of known dimensions in the image or inputting by the text manually.
- the information from image analysis and volume estimation can be indexed to estimate the energy and nutrients consumed via the data of Food nutritional values (20).
- the accuracy of this method depends on the accuracy of the food image selected to calculate energy and nutrient consumption.
- Users can make their long-term personal records of nutrient absorbing amount of food for management over their daily health.
- the system displays the estimated change of the personal health parameters by and calories and nutrients of consumed food.
- Fig.3 shows the constant repetition of this method and continuous interaction between the user, web portal and databases, an intelligent algorithm can anticipate the effect of amount of each nutrition to the personal health parameters.
- the method of this invention comprises the following steps:
- the food in the food image can be with high accuracy identified, the nutrition of the consumption food can be calculated, the diet evaluation can be provided for users, the user can be assisted in the diet, and the health can be managed more efficiently by intelligent algorithm.
- FIG.1 shows a schematic overview of a primary database systems and relation of users and system.
- Fig.2 shows flowchart of entering data and improving the database.
- Fig.3 shows continues interaction of user and web portal and intelligent algorithm.
- Fig.4 shows flowchart of intelligent food analyzing wherein intelligent food recognition operation is performed.
- Fig.5 shows flowchart of intelligent food recommendation wherein food health analyzing is performed
- Fig.6 shows flowchart of intelligent food recommendation process.
Landscapes
- Health & Medical Sciences (AREA)
- Nutrition Science (AREA)
- Engineering & Computer Science (AREA)
- Epidemiology (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Primary Health Care (AREA)
- Public Health (AREA)
- Medical Treatment And Welfare Office Work (AREA)
Abstract
The present invention relates a way to determine the amount of food consumed and provide a specific diet to maintain health based on a person's medical history. The method of this invention comprises the following steps: identifying the type and volume of the food by capturing food images via food image database, calculating the nutrition of the food according to the identified food type and the food volume, informing the consumer about the nutrition of the food,capturing food image after consumption of the food,calculating the nutrition of the consumption food and reporting the estimated change of the personal health parameters and nutrients of consumed food.
Description
Description
Title of Invention : | intelligent food image recognition and recommendation method
Technical Field
[0001] The present invention relates to the field of image processing technology and the field of healthy eating, specifically a way to determine the amount of food consumed and provide a specific diet to maintain health based on a person's medical history.
Background Art
[0002] Recent advances in smartphone technologies have directed to a production of food applications based on food image processing, with the aim to overcome disadvantages of the usual food diary which is incorrect and time consuming. In order to afford users feedback with nutritional information accompanied by insightful nutritional information, several methods have been discovered and analyzed.
[0003] By assets of development of smartphone use, some mobile applications have been developed to enable and simplify food journaling, and many have established excessive prospective in operative diet control. For example, teenagers are eager to take food images using a cellphone recorder before consumption; and the nutritional feedback contributes to weight loss. However, many of these applications require significant manual input from users and has low performance in assessing the exact ingredients and food portion, which has delayed the long-term use from users.
Technical Problem
[0004] Many people encounter challenges to maintain healthy diet nowadays, though knowing bad consumption behaviors lead to obesity that rise the risk of heart diseases, metabolic comorbidities, and gastrointestinal cancer. Personal diet management is always necessary in these states, which regularly includes manual food logging that is time consuming and boring. This invention provides a solution for monitoring health parameters thorough
image recognition of food and reporting the consumed nutrient and health parameters after eating the food by users.
Solution to Problem
[0005] First, users enter their profile on the web portal or mobile application and create a profile for themselves. This profile includes details such as national ID number; age and height and weight and other physical parameters. In addition, users can enter their health data such as blood pressure or disease records and allergies cases. This can be done manually or by connecting to the local health system database to enter the person's medical/health data into the system. The said data can store in the personal health parameter database (30) and can be updated occasionally.
[0006] As Fig.1 shows the system has at least three different database including food Image database or Food material (10), Food nutritional values (20), personal health parameters database (30).
[0007] A cellular phone or other electronic devices being used to capturing image of food, uploading the image files to an image server via internet.
[0008] At the first step, the food and its volume are identified. This can be done with the image recognition system of the food or with entering the text (for example name of the food) or combination of the said two methods. If food is identified, the first output is displayed to the user, according to the type of food and the user health profile. In the first output, brief nutrient- specific information such as calorie content, fat and protein as well as the food name is provided to the user as well as general recommendation about the consumption of the food. At first output the system alert about the health risk of desired food based from personal health parameters database (30).
[0009] As Fig.2 shows if identified food is harmful for the user health a similar healthy food will be proposed to the user.
[0010] As Fig. 2 shows If the desired food is not available in the database, the system administrator will be notified to add the said food to the list and therefore the detection system will be approved.
[0011] The image recognition can be done by segmenting the food images by an image segmentation algorithm. The similarity matching operation is carried out on the image to be recognized and the food images in a database, the food image with the highest similarity is obtained, the food label corresponding to the food image with the highest similarity serves as the food label of the image to be recognized and this food label display to the users as a text or an image. If the said result is not acceptable or wrong user can enter the name of the food manually as shown in Fig.2.
[0012] Image recognition done through the image identifying system, searching to confirm whether or not the image identifying parameters for food (including type, ingredient, quantity and size of food) built in the food image database (10), analyzing food nutrients of image identifying parameters via the Food nutritional values (20) of the system.
[0013] Therefore, users may recognize what nutrients contain in the food they would like to eat at once, and take such data to be reference basis for their food management. Still further, users can make their long-term personal records of nutrient absorbing amount of food for management over their daily health.
[0014] As shown in Fig.4, at the beginning of detection process, image noise is removed and the best sharping image filter is applied to detect food. The similar food is estimated through the Pre Trained Food Detector Model, which simultaneously compares the food image with all the food images in the database(10).At this stage, the material similarity is estimated by detecting edges in the photo. In the material recognition stage, several materials may be identified in the image. In this section, the Estimated Material in Dish is compared to the Estimated Food’s material and each food is given a score. Then their scores sort the foods and the highest scored food is recommended to the user.
[0015] On the other hand, as shown in Fig.4 in the Feedback analysis section, the correct recognition of the food name is analyzed, the result goes to the intelligent Food health Analyzer section. Otherwise, the user will be asked to correct the name of the food.
[0016] As shown in Fig.4, At least these parameters are stored in the primary database:Food materials, Food Nutrition Values, User health profile.
[0017] As shown in Fig. 5, User Health Parameters are taken from the User Health Profile and Nutritional value is taken from the Food Nutrition Values. Pre Trained Deep Learning decides whether food is suitable for the user. If the food is suitable, the food is approved for the user and the amount of food and other information is informed to the user. Even in this case, the user is asked whether s/he likes the food. If the food is not suitable for the user, healthy food is recommended to the person through the Intelligent Food Recommendation.
[0018] As shown in Fig.6 , finally, if the system is used several times, the database will contain the following items: Food nutritional values, Food materials, User health Profiles, User Diets, User Food reviews, User Food history. Where the system scores all foods with Nutritional Values versus User Health Profile.
[0019] When the user requests the analysis of the desired food (request Food Analysis), food scores are being updates according to matching user health profile with food materials, and then food scores updates according to matching user diet with foods.
[0020] Users capture images of their foods before and after eating. The volume of food consumption can be estimated by including an object of known dimensions in the image or inputting by the text manually. The information from image analysis and volume estimation can be indexed to estimate the energy and nutrients consumed via the data of Food nutritional values (20). The accuracy of this method depends on the accuracy of the food image selected to calculate energy and nutrient consumption. Users can make their long-term personal records of nutrient absorbing amount of food for management over their daily health. At the final output the system displays the estimated change of the personal health parameters by and calories and nutrients of consumed food.
[0021] As Fig.3 shows the constant repetition of this method and continuous interaction between the user, web portal and databases, an intelligent
algorithm can anticipate the effect of amount of each nutrition to the personal health parameters.
[0022] The method of this invention comprises the following steps:
[0023] 1. Identifying the type and volume of the food by capturing food images via food image database(10) or entering the name of the food before consumption.
[0024] 2. Calculating the nutrition of the food according to the identified food type and the food volume by information of Food nutritional values (20).
[0025] 3. Informing the consumer about the nutrition of the food or existence of allergenic food and advice to consume or not consume the food based on a person's health status comes from the personal health database(30).
[0026] 3. Capturing food image after consumption of the food.
[0027] 4. Calculating the nutrition of the consumption food by comparing the images before and after of food consumption by user.
[0028] 5. Reporting the estimated change of the personal health parameters and calories and nutrients of consumed food
Advantageous Effects of Invention
[0029] Based on the method of the invention, the food in the food image can be with high accuracy identified, the nutrition of the consumption food can be calculated, the diet evaluation can be provided for users, the user can be assisted in the diet, and the health can be managed more efficiently by intelligent algorithm.
Brief Description of Drawings
[0030] [Fig.1 shows a schematic overview of a primary database systems and relation of users and system.
[0031] Fig.2 shows flowchart of entering data and improving the database.
[0032] Fig.3 shows continues interaction of user and web portal and intelligent algorithm.
[0033] Fig.4 shows flowchart of intelligent food analyzing wherein intelligent food recognition operation is performed.
[0034] Fig.5 shows flowchart of intelligent food recommendation wherein food health analyzing is performed
[0035] Fig.6 shows flowchart of intelligent food recommendation process.
Claims
[Claim 1] Intelligent food image recognition and recommendation method comprising following steps: a. Making an individual profile in web or application for users, b. Identifying the type and volume of the food by capturing food images via food image database, c. Estimating similar food through the pre trained food detector model, which simultaneously compares the food image with all the food images in the database, d. Calculating the nutrition of the food according to the identified food type and the food volume by information of nutrient database, e. Deciding whether food is suitable for the user by pre trained deep learning model, f. Informing the consumer about the nutrition of the food and advice to consume or not consume the food based on a person's health status comes from the personal health database, g. Capturing food image after consumption of the food or beverage, h. Calculating the nutrition of the consumption food by comparing the images before and after of food consumption by user, i. Reporting the estimated change of the health parameters of user and nutrients of consumed food.
[Claim 2] The method according to claim 1 , further a similar healthy food is recommended to the user through the intelligent food recommendation.
[Claim 3] The method according to claim 1 wherein image noise is removed and the best sharping image filter is applied to food images.
[Claim 4] The method according to claim 1 wherein the food image be captured by a hand-held electronic device.
[Claim 5] The method according to claim 1 further identifying the type of food done by inputting the text.
[Claim 6] The method according to claim 1 wherein the database be improved by user and portal admin.
[Claim 7] The system of dietary and personal health assessment comprises: a. Food nutritional values, b. Food materials database, c. Personal health database, d. Web portal, e. User interface, f. Intelligent algorithm for anticipate personal health parameters.
[Claim 8] The system of dietary and personal health assessment according to claim 6 comprises further:
User Diets, User Food reviews, User Food history j
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/IB2021/052100 WO2021176432A1 (en) | 2021-03-13 | 2021-03-13 | Intelligent food image recognition and recommendation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/IB2021/052100 WO2021176432A1 (en) | 2021-03-13 | 2021-03-13 | Intelligent food image recognition and recommendation method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2021176432A1 true WO2021176432A1 (en) | 2021-09-10 |
Family
ID=77612593
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/IB2021/052100 WO2021176432A1 (en) | 2021-03-13 | 2021-03-13 | Intelligent food image recognition and recommendation method |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2021176432A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114958579A (en) * | 2022-05-30 | 2022-08-30 | 重庆电子工程职业学院 | Mould detection lamp for special crowds |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110318717A1 (en) * | 2010-06-23 | 2011-12-29 | Laurent Adamowicz | Personalized Food Identification and Nutrition Guidance System |
| US20130105565A1 (en) * | 2011-10-29 | 2013-05-02 | Richard Alan Kamprath | Nutritional Information System |
-
2021
- 2021-03-13 WO PCT/IB2021/052100 patent/WO2021176432A1/en active Application Filing
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110318717A1 (en) * | 2010-06-23 | 2011-12-29 | Laurent Adamowicz | Personalized Food Identification and Nutrition Guidance System |
| US20130105565A1 (en) * | 2011-10-29 | 2013-05-02 | Richard Alan Kamprath | Nutritional Information System |
Non-Patent Citations (1)
| Title |
|---|
| LO FRANK PO WEN; SUN YINGNAN; QIU JIANING; LO BENNY: "Image-Based Food Classification and Volume Estimation for Dietary Assessment: A Review", IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, vol. 24, no. 7, 30 April 2020 (2020-04-30), Piscataway, NJ, USA, pages 1926 - 1939, XP011796257, ISSN: 2168-2194, DOI: 10.1109/JBHI.2020.2987943 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114958579A (en) * | 2022-05-30 | 2022-08-30 | 重庆电子工程职业学院 | Mould detection lamp for special crowds |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110349671B (en) | Physical examination data processing method, system, electronic equipment and storage medium | |
| US20200258632A1 (en) | Hazard based assessment patterns | |
| US20240232266A1 (en) | Deep Multi-Modal Pairwise Ranking Model For Crowdsourced Food Data | |
| US20170323174A1 (en) | Food logging from images | |
| CN120280093A (en) | Systems and methods for food analysis, personalized recommendation, and health management | |
| US20120179665A1 (en) | Health monitoring system | |
| CN111159539B (en) | Food recommendation method and system based on multi-modal information association analysis | |
| Pouladzadeh et al. | You are what you eat: So measure what you eat! | |
| US20210118545A1 (en) | System and method for recommending food items based on a set of instructions | |
| US20250069417A1 (en) | Health management method based on image recognition | |
| CN111599438A (en) | Real-time diet health monitoring method for diabetic patient based on multi-modal data | |
| US20230255556A1 (en) | Biosensors and food logger systems for personalized health and self-care | |
| US20210375478A1 (en) | System and method for up-to-date nutrient database management and nutrient assessment | |
| CN109509117A (en) | A kind of vegetable recommended method, apparatus and system | |
| CN113936765A (en) | Method and device for generating periodic behavior report, storage medium and electronic equipment | |
| Kitamura et al. | Image processing based approach to food balance analysis for personal food logging | |
| CN113035317A (en) | User portrait generation method and device, storage medium and electronic equipment | |
| CN102053158A (en) | Blood sugar analysis system, device and method | |
| WO2021176432A1 (en) | Intelligent food image recognition and recommendation method | |
| Kogias et al. | A two-level food classification system for people with diabetes mellitus using convolutional neural networks | |
| CN117941007A (en) | Digital and Personalized Risk Monitoring and Nutrition Planning System for Prediabetes | |
| Bahadur et al. | Using deep learning techniques, a framework for estimating the nutritional value of food in real time | |
| Hakguder et al. | Smart Diet Management through Food Image and Cooking Recipe Analysis | |
| CN111435610A (en) | Method and device for recommending food and cooking appliance | |
| CN115237966A (en) | Food path recommendation method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21763599 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 21763599 Country of ref document: EP Kind code of ref document: A1 |