TWI908581B - Dual-band image sensing device, image analysis method and applications thereof - Google Patents

Abstract

A dual-band image sensing device comprises a containing unit and a light source and image capturing module positioned above the containing unit, capable of viewing the testing object. The device of the present invention includes at least an infrared light source unit, a visible light source unit, and an image capturing unit for acquiring infrared and visible light images of the testing object. An image analysis and machine learning module is signal-connected to the image capturing unit and includes an image preprocessing unit and an image analysis model unit. The present invention enhances predictive performance by meticulously preprocessing the captured images to effectively eliminate various interferences and noise. The image analysis method not only improves image quality but also facilitates the optimization of subsequent data analysis and applications, thereby achieving more accurate and reliable predictions of the object’s state.

Description

Dual-band image sensing device, its image analysis methods and applications

An image sensing device, particularly an image sensing device that uses dual-band sensing and analysis.

With rising incomes and Westernized eating habits, coupled with the introduction of internationally renowned coffee brands into the market in recent years, numerous coffee shops, both domestic and international, have become a common sight on the streets. The professionalism of the coffee industry has also continued to improve. The skills of Taiwanese people in selecting, roasting, and brewing coffee beans are now comparable to those in coffee-producing regions such as the United States and Europe, which is sufficient proof of the flourishing coffee industry in Taiwan.

Currently, the roasting process mainly relies on baristas to judge the roast level of coffee beans by sight. However, this method has many uncertainties, such as the barista's experience level and the influence of ambient light. This often leads to different roasting conditions for each batch of beans, affecting the taste and making it impossible to maintain the quality of coffee beans consistently.

To address the issue of inconsistent coffee quality caused by the high degree of human intervention and environmental lighting factors in the current coffee roasting process, it is necessary to provide a more accurate and objective testing method to improve coffee quality and production efficiency.

This invention first provides a method for analyzing dual-band images, the steps of which include: Acquiring an infrared image and/or a visible light image of the object under test and transmitting it to an image analysis and machine learning module; An image preprocessing unit in the image analysis and machine learning module preprocesses the infrared image and/or the visible light image, the steps of which include: Step S1-1 Image cropping, Step S1-2 Infrared image threshold selection and setting, and Step S1-3 Visible light image pixel sampling; Then, an image analysis model unit in the image analysis and machine learning module analyzes the preprocessed image, the steps of which include: Step S2-1 Training dataset establishment, Step S2-2... The uniformity distribution of the test object is obtained by comparing the input training dataset with the regression training model.

In step 1-2, shadow or overexposed areas in infrared and/or visible light images are filtered using an effective analysis image region threshold range set by the infrared image. This threshold range is determined by obtaining the maximum and minimum values of pixel information in different regions of the infrared image of the test object as the threshold range of near-infrared light intensity, and defining the final effective region threshold range.

Steps 1-3 involve taking the pixel coordinates within the threshold range and comparing these coordinates with the visible light image.

Step S2-1 involves extracting and filtering infrared (IR) or near-infrared (NIR) light and RGB channel information (R (red), G (green), B (blue)) from the infrared and/or visible light images of the test object, respectively. This information is then integrated with a standard value of the test object to obtain several datasets calculated using the following equations (1), (2), (3), and (4): RGB_input = [R G B]n×3…Equation (1); (N)IR_input = [(N)IR]n×1…Equation (2); RGB (N)IR_input = [R G B (N)IR]n×4…Equation (3); RGB (N)IR SV_input = [R G B (N)IR Standard Value]n×5…Equation (4); where n in equations (1) to (4) above represents the number of images of the object under test obtained.

In step S2-2, the regression training model comparison involves inputting the various datasets from step 2-1 into at least one regression model to perform model comparisons across the various datasets.

Preferably, the regression model includes one or more combinations of multiple linear regression models, polynomial regression models, and partial least squares regression models.

Furthermore, step S2-3 may optionally calculate the mean, mode, and/or median of the uniformity distribution of the test object.

This invention also provides a dual-band image sensing device for performing the aforementioned dual-band image analysis method, comprising: a housing unit for holding a test object; a light source and image acquisition module disposed above the housing unit and having a view of the test object; comprising at least an infrared light source unit, a visible light source unit, and an image acquisition unit; the infrared light source unit and the visible light source unit emitting infrared light and/or visible light to the test object, and the image acquisition unit acquiring infrared light images and/or visible light images; and an image analysis and machine learning module, signal-connected to the image acquisition unit, and comprising an image preprocessing unit and an image analysis model unit electrically and signal-connected.

In some preferred embodiments, a photomask unit is provided between the accommodating unit and the light source and image capturing module.

The visible light source units are arranged in a ring to form a visible light ring; and at least one infrared light source unit is disposed adjacent to the visible light source units. The present invention further provides a coffee roasting apparatus comprising the aforementioned dual-band image sensing device.

As can be seen from the above description, the present invention has the following beneficial effects and advantages:

1. This invention combines optical sensors, algorithms, and optical system design to develop a highly efficient and reliable coffee bean or coffee powder roasting degree detection system. It eliminates uncertainties such as human experience and ambient light sources during current coffee roasting processes, effectively improving the accuracy of coffee roasting degree detection. This invention uses a sensor to read the reflection wavelength of coffee beans/coffee powder in real time, utilizes algorithms for data analysis and prediction, and combines this with an optimized optical system design to achieve accurate measurement of coffee bean roasting degree. Meanwhile, the method provided by this invention can obtain the uniformity distribution of the state value of the test object, and make detailed predictions of the roast level of each coffee bean in each sample. Compared with existing coffee bean roast level detection devices, this invention can improve the roast level deviation caused by sensing only a single area of each sample. Therefore, this invention can promote the improvement of coffee quality, and also contribute to the automation and intelligentization of the coffee production process, driving the coffee industry towards a more technological future.

2. This invention, through detailed preprocessing of images captured by an image sensor, effectively eliminates various interferences and noise in the images, thereby significantly improving prediction performance and achieving a superior level. The image processing not only improves image quality but also helps optimize subsequent data analysis and applications, thus achieving more accurate and reliable results.

The present invention will be described and illustrated in detail below with reference to several preferred embodiments. The accompanying drawings are merely some exemplary representations or embodiments of the present invention. Those skilled in the art to which the present invention pertains can apply the present invention to other similar situations without making any further efforts.

The terms “system,” “device,” “unit,” and/or “module” used in this invention are one method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions. As shown in this invention, unless the context clearly indicates otherwise, words such as “a,” “an,” “an,” and/or “the” do not specifically refer to the singular and may also include the plural. Generally, the terms “comprising” and “including” only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

Flowcharts are used in this invention to illustrate the operations performed by the system according to embodiments of the invention. It should be understood that the preceding or subsequent operations are not necessarily performed precisely in sequence. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

< Dual-band image sensing device >

Please refer to Figure 1, which is a schematic diagram of a preferred embodiment of the dual-band image sensing device 100 of the present invention. In order to clearly explain and illustrate the dual-band image sensing device 100 of the present invention, this embodiment uses a coffee bean roasting degree detection device as an example. However, the dual-band image sensing device 100 provided by the present invention can detect other bean-shaped, granular or powdered food products with the same or similar image characteristics in addition to coffee beans, and is not limited thereto.

A preferred embodiment of the dual-band image sensing device 100 of the present invention includes at least: a housing unit 10 for holding a test object O; a light source and image capture module 20, disposed above the housing unit 10 and having a view of the test object O; comprising at least an infrared light source unit 21, a visible light source unit 22, and an image capture unit 23; the infrared light source unit 21 and the visible light source unit 22 emitting infrared light and/or visible light to the test object O, and the image capture unit 23 capturing infrared light images and/or visible light images; and an image analysis and machine learning module 40, signal-connected to the image capture unit 23, and including an image preprocessing unit 41 and an image analysis model unit 42 electrically and signal-connected.

Preferably, in some preferred cases, a photomask unit 30 is provided between the accommodating unit 10 and the light source and image capture module 20, so as to better enable the light source and image capture module 20 to capture the infrared light image and/or visible light image of the test object O state on the accommodating unit 10.

The infrared light source unit 21 is preferably an infrared light-emitting diode (LED); the visible light source unit 22 preferably includes visible light-emitting diodes (LEDs); the image capture unit 23 can capture images of at least the visible light band and/or the infrared light band, and preferably, when the light brightness is insufficient (i.e., in a dark environment where the visible light band is lacking), the image capture unit 23 can capture images of only the infrared light band. More preferably, in another preferred embodiment of the present invention, the infrared light source unit 21 can emit infrared light or near-infrared light, such as near-infrared light with a wavelength of 850nm, which is not limited herein.

In addition to at least one visible light source unit 22 being disposed above the receiving unit 10, as shown in the preferred embodiment of Figure 1, several visible light source units 22 may be arranged in a ring to form a visible light ring, and at least one infrared light source unit 21 may be disposed on the side or adjacent to the visible light source unit 22.

< Image Analysis Method for Dual-Band Image Sensing Devices >

Referring to Figure 2, the image preprocessing unit 41 of the image analysis and machine learning module 40 performs preprocessing on the infrared and/or visible light images of the object under test O captured by the image capture unit 23. The steps include: Step S1-1: Image cropping; Step S1-2: Infrared image threshold selection and setting; and Step S1-3: Visible light image pixel sampling.

In step 1-1, image cropping mainly involves removing invalid image areas from the captured infrared and/or visible light images that are not images of the object under test (O), retaining only the analyzable image of the object under test (O).

In the infrared image threshold selection and setting of steps 1-2, in order to avoid the analysis results being affected by the shadow areas caused by the overlapping and occlusion between particles or powders in the image of the test object O, as well as the inaccurate color information due to overexposure in the non-overlapping areas, further, the infrared image setting is used to effectively analyze the image area threshold for filtering these shadow or overexposed areas.

Because of infrared light, such as near-infrared light images, pixel information can be used as a reference for threshold selection. Near-infrared images only store the intensity value of the infrared light at the time of capture, with an intensity range of 0-255, which can be used to filter out overexposed or shadowed pixels. In practice, the main approach is to obtain the maximum and minimum values of pixel information in different regions of the infrared image of the object under test O as the threshold range for near-infrared light intensity, and to define the final effective threshold range.

In step S1-3, which involves sampling pixels in the visible light image and establishing a dataset, after defining the effective area threshold, the pixel coordinates within the threshold range are extracted, and these coordinates are applied to the visible light image.

Next, please refer to Figures 1 and 3. The image analysis model unit 42 is included on a desktop or portable device such as a computer, laptop, tablet or mobile phone and performs the following analysis and calculation.

The image preprocessed by the image preprocessing unit 41 is analyzed by the image analysis model unit 42, and the steps include:

Step S2-1: Establish training dataset;

Step S2-2: Input training dataset and compare with regression training model to obtain the uniformity distribution of the test object O;

The training dataset establishment step S2-1 mainly involves extracting and filtering infrared (IR) or near-infrared (NIR) and RGB channel information of R (red), G (green), and B (blue) from the infrared and/or visible light images of the test object O, and integrating it with a standard value of the test object O to obtain several datasets calculated by the following formulas (1), (2), (3) and (4) for subsequent training model establishment and evaluation analysis.

RGB_input = [R G B]n×3…Formula (1);

(N)IR_input = [(N)IR]n×1…Equation (2);

RGB (N)IR_input = [R G B (N)IR]n×4...Equation (3);

RGB (N)IR SV_input = [R G B (N)IR Standard Value]n×5…Equation (4); where n in the above equations (1) to (4) is the number of images of the object under test O obtained.

Step S2-2, regression training model comparison, mainly involves inputting several datasets from step 2-1 into at least one regression model for model comparison across these datasets. This regression model includes, but is not limited to, one or more combinations of multiple linear regression (MLR), polynomial regression (PR), and partial least squares (PLS) regression models.

Furthermore, step S2-3 may optionally calculate the mean, mode, and/or median of the obtained uniformity distribution to verify the accuracy of the O state value of each analyte.

< Implementation Example 1 >

In a preferred embodiment of the present invention, the test substance O is coffee beans, and the roasting status of coffee beans with 16 different roasting levels is analyzed.

First, infrared and/or visible light images of the 16 different roasting levels of coffee beans are acquired using the aforementioned dual-band image sensing device 100. Then, these infrared and/or visible light images are transmitted to the image analysis and machine learning module 40 for analysis.

Please refer to Figures 4A and 4B and Table 1 below. After the image preprocessing unit 41 crops the infrared and/or visible light images to obtain the effective image area, it then performs infrared image threshold selection and setting steps to obtain the effective area threshold range of infrared image pixel intensity values for several preferred embodiments of different coffee beans. In this embodiment, the minimum and maximum values of the infrared image pixel intensity values for 16 different roasting levels of coffee beans are 35-245, and these values are set as the effective area threshold range.

Table 1. Sample Name Infrared image pixel intensity range Elf 90-200 Flw 140-225 Gsha 120-220 Sld 140-215 Yag 130-210 Purcase01_Brz 135-220 Purcase01_EthopiaWater 160-235 Purcase01_Gua 150-245 (maximum value) Purcase02_ColumbiaGsha 135-235 Purcase02_Costalightmedium 115-225 Purcase02_Costamedium 145-215 Purcase02_EthopiaSun 135-240 Self01_light 90-245 Self01_lightmedium 100-230 Self01_roast 70-160 Self01_roastmedium (Minimum value) 35-130

Next, the image preprocessed by the image preprocessing unit 41 is analyzed by the image analysis model unit 42. Please refer to Figure 5A for the original visible light image of this embodiment, Figure 5B for the near-infrared light intensity filtered image, and Figure 5C for the visible light image coordinate filtered image. The RGB three-channel information and NIR information values obtained through these images are integrated with the standard data set of coffee bean standard roast value (Standard RD). This embodiment obtained data of 16 types of coffee beans as shown in Table 1 above, 100 images of each type of coffee bean, for a total of 1600 images. The data is integrated using the above equations (1) to (4) into the following equations (5) to (8), thus obtaining the complete data set of the 6 types of coffee beans in this embodiment.

…Equation (5);

…Equation (6);

…Equation (7);

…Formula (8).

After establishing models with different input datasets, this embodiment uses 16 verification images of coffee beans at different roast levels to evaluate the model's prediction accuracy. After pixel filtering, the values of the R, G, B, and NIR channels of all pixels are sequentially input into at least one regression model in step S2-2 to infer the roast uniformity distribution of each coffee bean sample. Finally, to verify accuracy, the uniformity distribution can optionally be calculated as the mean, median, and/or mode, and compared with the standard roast value to verify the coffee bean roast prediction.

Please refer to Figures 6A-6P and Table 2 below. This embodiment uses a multivariate linear regression model, a polynomial regression model, and a partial least squares regression model, combined with an RGB+NIR four-channel input dataset, and verifies the visual distribution of coffee bean roast uniformity using the mean, calculating 16 different estimated values for coffee bean roast. It also compares the results with the standard roast value detected by currently commercially available infrared coffee bean roast testing machines, confirming that the device and method provided by this invention can indeed accurately predict the state value of the test object O. While this embodiment uses the mean for verification, the median and mode are also verified as valid.

Table 2. Number Standard roast value The invention's predicted value Image of coffee beans and distribution of roast uniformity 1 50.7 55.2 As shown in Figure 6A 2 50.0 57.2 As shown in Figure 6B 3 57.6 59.9 As shown in Figure 6C 4 54.7 56.5 As shown in Figure 6D 5 52.1 54.4 As shown in Figure 6E 6 56.2 54.0 As shown in Figure 6F 7 55.4 59.4 As shown in Figure 6G 8 57.8 60.4 As shown in Figure 6H 9 62.9 59.6 As shown in Figure 6I 10 55.3 57.6 As shown in Figure 6J 11 52.6 53.2 As shown in Figure 6K 12 64.8 61.1 As shown in Figure 6L 13 67.2 59.5 As shown in Figure 6M 14 55.6 50.2 As shown in Figure 6N 15 39.5 36.8 As shown in Figure 60 16 24.5 28.2 As shown in Figure 6P

Flowcharts are used in this invention to illustrate the operations performed by the system according to embodiments of the invention. It should be understood that the preceding or subsequent operations are not necessarily performed precisely in sequence. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the embodiments are modified in some examples with the modifiers "approximately," "approximately," or "substantially." Unless otherwise stated, "approximately," "approximately," or "substantially" indicates that the number is allowed to vary by ±20%. Correspondingly, in some embodiments, the numerical parameters used in the specifications and requests are approximate values, which may vary depending on the specific characteristics required by the individual embodiments. In some embodiments, numerical parameters should take into account the specified significant digits and adopt a general method of digit reservation. Although the numerical ranges and parameters used to confirm their scope in some embodiments of the invention are approximate values, in specific embodiments, the settings of such numerical values are as accurate as feasible.

Finally, it should be understood that the embodiments described herein are merely illustrative of the principles of the embodiments of the present invention. Other variations may also fall within the scope of the present invention. Therefore, alternative configurations of the embodiments of the present invention, as examples and not limitations, may be considered consistent with the teachings of the present invention. Accordingly, the embodiments of the present invention are not limited to those expressly described and illustrated herein.

100: Dual-band image sensing device 10: Reception unit 20: Light source and image acquisition module 21: Infrared light source unit 22: Visible light source unit 23: Image acquisition unit 30: Photomask unit 40: Image analysis and machine learning module 41: Image preprocessing unit 42: Image analysis model unit O: Object under test S1-1~S1-3, S2-1~S2-3: Steps

To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some examples or embodiments of the present invention and are not intended to absolutely limit the technical scope of the present invention. Unless it is obvious from the context or otherwise explained, the same reference numerals in the figures represent the same structure or operation. Among them: Figure 1 is a schematic diagram of a preferred embodiment of the dual-band image sensing device of the present invention. Figure 2 is a flowchart of the preprocessing steps performed by the image preprocessing unit in the image analysis and machine learning module of the present invention. Figure 3 is a schematic flowchart of the image analysis steps performed by the image analysis model unit of the present invention. Figures 4A and 4B show the effective area value range of several coffee bean samples in Embodiment 1 of the present invention. Figures 5A, 5B, and 5C are the original visible light image, near-infrared light intensity filtered image, and visible light image coordinate filtered image of the coffee bean sample in Embodiment 1 of the present invention, respectively. Figures 6A to 6P are the coffee bean images and roast uniformity distribution diagrams of several coffee bean samples in Embodiment 1 of the present invention, respectively.

100: Dual-band image sensing device

10: Containment Unit

20: Light Source and Image Capture Module

21: Infrared Light Source Unit

22: Visible Light Source Unit

23: Image Capture Unit

30: Photomask Unit

40: Image Analysis and Machine Learning Module

41: Image Preprocessing Unit

42: Image Analysis Model Unit

O: Analyzer Test Item

Claims (11)

A dual-band image analysis method includes the following steps: capturing an infrared image and a visible light image of a test object and transmitting them to an image analysis and machine learning module, which includes an image preprocessing unit and an image analysis model unit connected by signals; the image preprocessing unit in the image analysis and machine learning module preprocesses the infrared image and the visible light image, including the following steps: Step S1-1: image cropping, removing invalid image areas from the infrared image and the visible light image that are not images of the test object, retaining only the analyzable image of the test object; Step S1-2: infrared image threshold selection and setting to filter out shadow or overexposed areas in the infrared image; and Step S1-3: visible light image pixel sampling; The image analysis model unit analyzes the pre-processed image, and the steps include: Step S2-1: Establishing the training dataset; and Step S2-2: Inputting the training dataset into the regression training model and comparing it to obtain the uniformity distribution of the state of the object under test. The dual-band image analysis method as described in claim 1, wherein: step 1-2 filters the shadow or overexposed areas in the infrared and visible light images by setting an effective analysis image region threshold range using the infrared image; the threshold range is obtained by taking the maximum and minimum values of pixel information in different regions of the infrared image of the test object as the threshold range of near-infrared light intensity, and defining the final effective region threshold range. The dual-band image analysis method as described in claim 1, wherein: steps 1-3 involve taking pixel coordinate values within the threshold range and comparing these coordinate values with the visible light image. As described in claim 1, the dual-band image analysis method includes the following steps: Step S2-1 involves extracting and filtering infrared (IR) or near-infrared (NIR) and RGB channel information (R (red), G (green), B (blue)) from the infrared and visible light images of the object under test, and integrating this information with a standard value of the object under test to obtain several datasets calculated by the following formulas (1), (2), (3), and (4): RGB_input = [R G B]n×3…Formula (1); (N)IR_input = [(N)IR]n×1…Formula (2); RGB (N)IR_input = [R G B (N)IR]n×4…Formula (3); RGB (N)IRSV_input = [R G B (N)IR Standard…Formula (3) Value]n×5…Equation (4); where n in the above equations (1) to (4) is the number of images of the object to be tested. The dual-band image analysis method as described in claim 1, wherein: step S2-2, regression training model comparison, involves inputting several datasets from step 2-1 into at least one regression model for model comparison of the several datasets; and the regression model includes one or more combinations of multiple linear regression models, polynomial regression models, and partial least squares regression models. The dual-band image analysis method as described in any one of claims 1 to 5, wherein: the mean, mode and/or median of the uniformity distribution of the state of the object under test are further calculated. A dual-band image sensing device for performing the dual-band image analysis method as described in claims 1-6, comprising: a receiving unit for holding a test object; a light source and image capture module disposed above the receiving unit and having a view of the test object; comprising at least an infrared light source unit, a visible light source unit, and an image capture unit; the infrared light source unit and the visible light source unit emitting infrared light and/or visible light to the test object, and the image capture unit capturing infrared light images and/or visible light images; and an image analysis and machine learning module signal-connected to the image capture unit, comprising an image preprocessing unit and an image analysis model unit electrically and signal-connected. The dual-band image sensing device as described in claim 7, wherein: a photomask unit is provided between the accommodating unit and the light source and image acquisition module. The dual-band image sensing device as described in claim 7 or 8, wherein: the infrared light source unit emits infrared light and/or near-infrared light. The dual-band image sensing device as described in claim 7 or 8, wherein: a plurality of visible light source units are arranged in a ring to form a visible light ring; and at least one infrared light source unit is disposed adjacent to the visible light source unit. A coffee roasting degree detection device comprising a dual-band image sensing device as described in any one of claims 7-10.