CN102184405B - Image Acquisition and Analysis Method - Google Patents

Abstract

The present invention relates to the technical field of image processing, and specifically discloses an image acquisition and analysis method, comprising: Step 1. Making several scales and placing them on each row of supermarket shelves, the scales have several groups of codes; Step 2 .Collect the image of the shelf with the scale; step 3. separate the color channels for the collected image, and detect the code of the scale on each channel after separation; step 4. according to the detection result of the code Correct the image; Step 5. Segment the region containing the commodity in the image on the corrected image according to the detection result of the code; Step 6. Use the color feature and local texture feature to classify the commodity in the region , and then get the relevant information about the product placement. The invention can efficiently and accurately obtain information such as the placement position and quantity of commodities on the shelf. Moreover, the method is simple and easy, and saves a lot of manpower and material resources.

Description

Image Acquisition and Analysis Method

technical field

The invention relates to the technical field of image processing, in particular to a method for analyzing collected images and obtaining information on placing commodities on shelves.

Background technique

With the development of the retail industry, supermarkets have become the main sales channel for mass commodities. Mastering the placement of commodities in supermarkets is helpful for commodity manufacturers to keep abreast of market trends and adjust marketing strategies in a timely manner. At present, the main way for commodity manufacturers to obtain commodity placement information is to send staff to conduct on-the-spot investigations in supermarkets. After the dispatched staff arrive at the supermarket, they will make statistics on whether the goods are neatly placed and the number of goods placed by visual inspection and manual counting. There are following problems in the method of manual on-site measurement. First of all, the manufacturer needs to send a large number of staff to different sales points for investigation, which is inefficient; second, the methods of visual inspection and manual counting are not accurate, and it is difficult for the staff to count the quantity of a certain product in a short period of time and accurately evaluate it at the same time. The orderliness of the goods is too subjective; again, the authenticity of the data collected by this method cannot be guaranteed, and the negligence of the staff will lead to inaccurate data, and even intentional falsification of data. All in all, the current method needs to invest a lot of manpower and material resources but cannot get accurate results.

The use of computer as an auxiliary tool for commodity information statistics can improve efficiency and ensure the accuracy of statistical results. Current related methods include object detection and object classification methods in pattern recognition. This type of method first collects supermarket shelf images, then finds the products on the supermarket shelves through target detection, then uses the method of target classification to classify the detected products into their categories, and finally counts the number of products in each category. This method also has some drawbacks. First of all, it cannot handle the situation of repeated patterns. For supermarket shelves, there are often multiple items of the same type of product. This type of method can usually only judge the type of a detected product, but cannot judge the same type of product detected. Whether the products in the category are the same product on the shelf, this may cause double counting during statistics; secondly, this type of method usually has high requirements on lighting conditions, and the lighting conditions of different supermarkets vary greatly, which makes this type of method not fully accurate. promote. In general, even with the use of computers, existing methods cannot guarantee accuracy while being highly efficient.

Contents of the invention

(1) Technical problems to be solved

The technical problem to be solved by the present invention is how to efficiently and accurately identify and analyze captured images of supermarket shelves, and then obtain information such as the quantity and category of commodities on the shelves.

(2) Technical solution

In order to solve the above technical problems, the present invention provides an image acquisition and analysis method, comprising:

Step 1, making several scales and placing them on each row of supermarket shelves, the scales have several groups of codes;

Step 2, collecting the image of the shelf with the ruler;

Step 3, separating the color channels of the collected image, and detecting the coding of the scale on each separated channel;

Step 4, correcting the image according to the detection result of the encoding;

Step 5. Segment the region containing the commodity in the image on the corrected image according to the detection result of the code;

Step 6: Classify the commodities by using the color features and local texture features in the area, and then obtain the relevant information about the placement of the commodities.

Wherein, the code is composed of numbers and two-dimensional codes.

Further, the two-dimensional code is a colored two-dimensional code, the numbers correspond to the colored two-dimensional code one by one, and each group of codes is different.

In a further technical solution, said step 3 includes:

Separate the image into color channels, use template matching to detect numbers and two-dimensional codes on each separated channel, and use the one-to-one correspondence between numbers and two-dimensional codes to perform cross-validation, and finally obtain the scale Coordinates in the image plane.

In a further technical solution, the correction in step 4 includes color correction and distortion correction.

The color correction includes linear transformation of the color values of each channel, normalizing the brightness and contrast of the area where the two-dimensional code is located to a preset constant through the linear transformation, and applying the linear transformation to the entire image The color correction is now complete.

The distortion correction includes finding a two-dimensional affine transformation matrix for the entire image, and adjusting the shape of the scale in the image to be a horizontal straight line.

In a further technical solution, the step 5 includes: segmenting the region containing the commodity in the image according to the coordinates of the scale and the relative positional relationship between the scale and the commodity placement area.

Said step 6 comprises:

Count the color features of the goods to be classified in the area, and compare them with the color list made in advance;

The local texture features of the commodity to be classified are extracted in the region, and compared with the texture list made in advance, to obtain the category to which the commodity to be classified belongs.

Among them, before step 1, it also includes:

Pre-collect a number of commodity images as training samples, perform color correction and distortion correction on the training samples, count the frequency of color appearance in the corrected image, find the local maximum value of the frequency, determine and adjust according to the local maximum value The number of mixed Gaussian models, using the mixed Gaussian model to fit the frequency of statistical color appearance, when the central probability density of the Gaussian distribution of a certain color is greater than the set threshold, the color corresponding to the Gaussian model is the main color , compose the final main colors into a color list;

For each type of commodity, count the frequency of appearance of each color in the color list, and use the statistical frequency as the color feature of this type of commodity;

A window of 16*16 pixels is arbitrarily taken for the training sample, the local texture feature of the window is calculated, the local maximum value of the local texture feature is obtained, and the individual of the mixed Gaussian model is determined and adjusted according to the local maximum value. number, and then use the mixed Gaussian model to fit the occurrence frequency of statistical local texture features. If the central probability density of a certain Gaussian distribution is greater than the preset threshold, the corresponding feature of the Gaussian model is the main local texture feature, and its Add the list of local texture features, and finally get the list of local texture features including the main local features of all commodities.

The counting of the color features of the commodities to be classified in the area, and comparing with the pre-made color list specifically includes:

Extract the color feature of the product to be classified in the area, and compare it with the color feature in the color list, when the sum of the color feature differences between the product to be classified and a certain type of training sample is the smallest, the product to be classified The product belongs to the category of the training sample in terms of color features.

The local texture features of the commodity to be classified are extracted in the region, and compared with the local texture list made in advance, and the category of the commodity to be classified specifically includes:

For every two types of commodities under the same color feature obtained after classification according to the color feature, count several local texture features with the largest difference in frequency of occurrence, and use the difference in frequency of occurrence of the local texture features to judge the accurate classification of the current commodity.

After the step 6, also include:

According to the category result obtained in step 6, the quantity and the position on the shelf of each type of commodity are counted, and the statistical result is output.

(3) Beneficial effects

The above technical solution has the following beneficial effects: by using scales with numbers and two-dimensional codes to mark the placement information of commodities, and collecting images of shelves with scales, by correcting the images, segmenting regions, and counting colors, etc., it can Efficiently and accurately obtain information such as the placement and quantity of commodities. Moreover, the method is simple and easy, and saves a lot of manpower and material resources.

Description of drawings

Fig. 1 is the flowchart of the image collection and analysis method of the embodiment of the present invention;

Fig. 2 is a schematic diagram of a shelf structure of an embodiment of the present invention;

Fig. 3 is a scale schematic diagram of an embodiment of the present invention;

Fig. 4 is a schematic diagram of distortion correction according to an embodiment of the present invention;

Fig. 5 is a schematic diagram of a color two-dimensional code according to an embodiment of the present invention.

Detailed ways

The specific implementation manners of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

As shown in Figure 1, it is a flow chart of the image acquisition and analysis method of the embodiment of the present invention. This embodiment includes the following steps:

Step 1. Make several scales, and place the scales on each row of supermarket shelves, with several sets of codes composed of numbers and QR codes on the scales;

This method designs a special color scale printed with numbers and two-dimensional codes, as shown in Figure 2, and places the scale horizontally on the product label in front of the shelf, that is, the D, E, and F areas in Figure 2. In this embodiment, one scale is placed in each row area, each scale is segmented, and each segment has a set of codes. Each set of codes has a fixed length and is printed with different numbers and QR codes. By identifying the number and the two-dimensional code, it is possible to deduce which position of the scale the group of codes is on, and then know the position of the product corresponding to the group of codes on the shelf. As shown in FIG. 3 , specifically, the coding on the scale includes: two digits and a two-dimensional code, which can be located at both ends of each segment. The QR code is divided into two rows and several columns. Each small square formed by the rows and columns is printed with a different color, and the color arrangement of each segment of the code is different. The two-dimensional code is a color two-dimensional code, and the colored two-dimensional code can be coded separately on each color channel, so it can obtain a larger coding space than the black and white two-dimensional code. Ni (such as N1, N2, etc.) in the figure represents a number , Ci (such as C1, C2, etc.) is the color of the color QR code. There is a one-to-one correspondence between numbers and QR codes, and the codes of all scales do not appear repeatedly. In this way, as long as any one of the number and the two-dimensional code is recognized, the position of this group of codes in the scale can be determined. To recognize numbers and QR codes, template matching can be used.

Step 2, collecting the ruler image on the supermarket shelf;

Use image acquisition devices such as cameras and mobile phones to collect images of rulers placed on supermarket shelves; use the projection transformation relationship between multiple rulers to obtain the angle between the photographer and the shelf plane, that is, the angle between the photographer and the shelf plane can be obtained through the rulers. The relative position of the shelves. It is also possible to set up a camera on the opposite side of the shelf or at a suitable location, connect the camera to the host, and control the camera to collect shelf images at regular intervals or when needed; it can further save manpower and material resources and improve the accuracy of statistical commodity information.

Step 3, separating the color channels of the collected image, and detecting the specific mark on the scale on each channel;

The specific signs of this embodiment are the codes on the scale: numbers and two-dimensional codes;

When detecting a two-dimensional code, first of all, it is necessary to separate the color channels of the collected image.

The method of separating color channels is just the opposite of the method of generating color QR codes. If the color QR code is encoded separately on the three channels of RGB and then superimposed, the image will be decomposed into three channels of RGB when detecting the mark. If the color QR code is encoded separately on the three CMY channels and then superimposed, the image will be decomposed into the three CMY channels when detecting the mark.

Then, use the template matching method to detect numbers and two-dimensional codes on each separated channel, and use the one-to-one correspondence between numbers and two-dimensional codes to perform cross-validation, and finally obtain the coordinates of the scale on the image plane, which can be used represented by a straight line equation.

Step 4, correcting the image according to the detection result of the specific mark;

The image correction in this step includes color correction and distortion correction.

Wherein, the color correction utilizes the result of separating the color channels in step 3, and normalizes each channel. During normalization, a linear transformation is first performed on the color values of the three channels. After the transformation, the brightness (average value of grayscale) and contrast (variance of grayscale) of the area where the two-dimensional code is located are normalized to a preset constant. Applying this transformation to the entire image completes the color correction. Color correction needs to be done separately for each color channel. The brightness and contrast of each color channel can also be obtained by using the color information in the QR code, and the influence of different lighting conditions can be eliminated after normalization. The linear transformation here is to remix the color values of each channel in different proportions. For example, there are three channels before the linear transformation, and the color values of the three channels are r, g, and b respectively. The color values of the three channels after correction may be: the color value of the first channel is 0.9r+0.1g, and the color value of the third channel is 0.9r+0.1g. The color value of the second channel is 0.9g+0.1b, and the color value of the third channel is 0.9b+0.1r.

Distortion correction also uses the result of step 3. Distortion correction First, according to the angle between the photographer and the shelf plane in step 2, a two-dimensional affine transformation matrix is obtained for the entire image. After this transformation, the scale will be a horizontal line in the image, as shown in Figure 4. Apply this transformation Distortion correction is completed to the entire image.

Step 5. Segment the commodity area to be analyzed on the corrected image according to the detection result of the specific mark;

After image correction, according to the coordinates of the ruler, use the relative positional relationship between the ruler (the three areas D, E, and F in Figure 2) and the commodity placement area (the three areas A, B, and C in Figure 2) to segment the image. The region containing the product, in which features are extracted during subsequent processing.

Step 6. Using the color information and texture information to classify the commodities in the area obtained in step 5, and then obtain the relevant information on the placement of the commodities.

For the region obtained by step 5 segmentation:

Before adopting this method, it is necessary to collect a large number of commodity images in advance as training samples, correct the color and distortion of the sample images according to the scale, and count the occurrence frequency of various colors in the corrected images, that is, extract color features from these training samples. According to the frequency of all colors in the statistical training samples, the local maximum value of the color frequency is obtained by using the Mean-Shift algorithm, and the number of mixed Gaussian models is determined and adjusted according to the local maximum value, and then the mixed Gaussian model is used to analyze the statistics. The frequency of color occurrence is fitted. If the central probability density of a Gaussian distribution is greater than the preset threshold, the color corresponding to the Gaussian model will be the main color, and it will be added to the color list. In the end, you get a color list that contains the main colors of all items.

Then count the color features: for each type of product, count the frequency of each color in the color list, and use this frequency as the color feature of this type of product.

Finally, count local texture features: when extracting local texture features, take a window of 16*16 pixels in the target image, and calculate its local texture features, such as Gabor features. For each commodity image, it is densely sampled with a window of 16*16 pixels, and then the local maximum value of the local texture feature is obtained by using the Mean-Shift algorithm, and the individual value of the mixed Gaussian model is determined and adjusted according to the local maximum value. number, and then use the mixed Gaussian model to fit the occurrence frequency of statistical local texture features. If the central probability density of a certain Gaussian distribution is greater than the preset threshold, the corresponding feature of the Gaussian model is the main local texture feature, and its Added to the list of local texture features. Finally, a list of local texture features including the main local features of all commodities is obtained. For each type of product, the frequency of various textures in the local texture feature list is counted, and this frequency is used as the local texture feature of this type of product.

When identifying the commodities on the shelf in step 6, this method uses two types of features, in which the color feature is used for rough classification, and the local texture feature is used for fine classification.

The method of rough classification with color features is as follows: when performing statistical color features on the region obtained in step 5, the color of each pixel in the region is considered to belong to the most similar color in the color list. Specifically, the color feature of the commodity to be classified, that is, the frequency of occurrence of the main color, and the color feature of the known commodity in the pre-established color list can define similarity. For example, for each main color, the difference between the frequency of the product to be classified and the frequency of the color of the known product can be used as a measure of similarity. The smaller the difference, the closer the product to be classified is to the known category. commodity. Next, the frequency of appearance of each color in each type of product image is counted, which is used as the color feature (template) to describe this type of product; in the process of product classification, the color feature of the product to be classified is related to the template The closest, that is, the class of commodity templates with the smallest sum of the probability differences between the commodity to be classified and several known categories of commodities, that is, the probability differences of each main color in the training sample; the commodity to be classified is considered to belong to this category.

Since different types of commodities may have the same color statistical characteristics, the classification result of this step is not accurate, and only multiple possible results can be output (that is, a certain commodity may belong to certain categories). Further classification needs to utilize local texture features.

After obtaining the rough classification results, this method distinguishes different types of commodities through local texture features. Since multiple possible commodity types obtained by rough classification have similar color features, local texture features are only extracted on grayscale images.

For every two types of commodities under the same color feature obtained after rough classification, count several local texture features with the largest difference in frequency of occurrence, that is, several local texture features with the largest frequency difference in the two types of commodities; use the The difference in the frequency of local texture features can accurately classify the current commodity.

Finally, according to the category results obtained in step 6, the quantity of various commodities and their positions on the shelves are counted, and the statistical results are output.

Because the products need to be updated, new products will enter the market, and outdated products will exit the market. The fixed product color template and local texture features cannot achieve satisfactory classification results. This method uses the current classification result as a new training set, and automatically performs self-adaptive training at regular intervals. Use the latest training results as a reference in future processing.

As can be seen from the above embodiments, the embodiment of the present invention uses scales with numbers and two-dimensional codes to mark the placement information of commodities, and collects images of shelves with scales, and corrects the images, divides regions, and counts colors. And other processing, can efficiently and accurately obtain information such as the location and quantity of commodities. Moreover, the method is simple and easy, and saves a lot of manpower and material resources.

The above is only a preferred embodiment of the present invention, it should be pointed out that for those of ordinary skill in the art, without departing from the technical principle of the present invention, some improvements and modifications can also be made, these improvements and modifications It should also be regarded as the protection scope of the present invention.

Claims (11)

1. the IMAQ analytical approach is characterized in that, comprising:

Step 1, make several scales and it is placed on every capable supermarket shelves, have some group codings on the said scale;

Step 2, collection are placed with the image of the shelf of said scale;

Step 3, said separation of images color channel to collecting, the coding of detection scale on each passage after the separation; Said coding is made up of numeral and two-dimension code; With said separation of images color channel; On each passage after the separation, utilize the method for template matches to detect numeral and two-dimension code; Utilize the one-to-one relationship between numeral and the two-dimension code to carry out cross validation simultaneously, finally obtain the coordinate of said scale at the plane of delineation;

Step 4, basis are proofreaied and correct image the testing result of said coding;

Be partitioned into the zone that comprises commodity in the image on step 5, the image of testing result after correction of basis to said coding;

Step 6, on said zone, utilize color characteristic and local textural characteristics that commodity are classified, and then obtain the relevant information that commodity are put.

2. IMAQ analytical approach as claimed in claim 1 is characterized in that, said two-dimension code is the color 2 D sign indicating number, and said numeral and color 2 D sign indicating number are corresponding one by one, and each group coding is all inequality.

3. IMAQ analytical approach as claimed in claim 1 is characterized in that the correction of said step 4 comprises colour correction and distortion correction.

4. IMAQ analytical approach as claimed in claim 3; It is characterized in that; Said colour correction comprises asks linear transformation to the color-values of each passage;, said linear transformation is applied to entire image has promptly accomplished colour correction the constant that the brightness and contrast of two-dimension code region is normalized into prior setting through said linear transformation.

5. IMAQ analytical approach as claimed in claim 3 is characterized in that, said distortion correction comprises asks two-dimentional affine transformation matrix to entire image, is horizontal linear with the shape adjustments of said scale in image.

6. IMAQ analytical approach as claimed in claim 5 is characterized in that, said step 5 comprises: the relative position relation according to the coordinate of said scale, said scale and commodity placement area is partitioned into the zone that comprises commodity in the image.

7. IMAQ analytical approach as claimed in claim 6 is characterized in that, said step 6 comprises:

Statistics is treated the color characteristic of classified commodity in said zone, and contrasts with the colors list of prior making;

In said zone, extract the local grain characteristic of treating classified commodity, and do contrast, obtain the classification of treating that classified commodity is affiliated with the local grain tabulation of prior making.

8. IMAQ analytical approach as claimed in claim 7 is characterized in that, before step 1, also comprises:

Gather some commodity images in advance as training sample; Training sample is carried out colour correction and distortion correction; Add up the frequency that its color occurs in the image after correction, ask for the local maximum of frequency, confirm and adjust the number of mixed Gauss model according to said local maximum; Utilize said mixed Gauss model that the frequency of the color appearance of statistics is carried out match; When the center probability density of the Gaussian distribution of a certain color during greater than setting threshold, the color that this Gauss model is corresponding is main color, and the main color that obtains is at last formed colors list;

To each type commodity, add up the frequency that shades of colour occurs in the said colors list, with the color characteristic of the frequency of adding up as such commodity;

Said training sample is got the window of a 16*16 pixel arbitrarily; Calculate the local grain characteristic of said window; Try to achieve the local maximum of said local grain characteristic, confirm and the number of adjustment mixed Gauss model, utilize mixed Gauss model that the local grain characteristic frequency of occurrences of statistics is carried out match then according to local maximum; If the center probability density of a certain Gaussian distribution is greater than prior preset threshold; Then this Gauss model characteristic of correspondence is main local grain characteristic, and it is added the local grain feature list, obtains comprising the local grain feature list of the main local feature of all commodity at last.

9. IMAQ analytical approach as claimed in claim 8 is characterized in that, said in said zone statistics treat the color characteristic of classified commodity, and do contrast with the colors list of prior making and specifically comprise:

Extract the color characteristic of treating classified commodity in the said zone; And compare with color characteristic in the said colors list; When the said color characteristic difference sum of treating classified commodity and certain type of training sample hour, the said classified commodity of treating belongs to the classification of such training sample on color characteristic.

10. IMAQ analytical approach as claimed in claim 9; It is characterized in that; The said local grain characteristic treat classified commodity of in said zone, extracting, and do contrast with the local grain tabulation of prior making, obtain treating that the classification under the classified commodity specifically comprises:

To according to the per two types of commodity under the same color characteristic that obtains after the color characteristic classification, add up several maximum local grain characteristics of its frequency of occurrences difference, utilize the otherness of the said local grain characteristic frequency of occurrences that current commodity are classified.

11. IMAQ analytical approach as claimed in claim 7 is characterized in that, also comprises after the said step 6:

The classification result who obtains according to said step 6 adds up the quantity of all kinds of commodity and the position on shelf, the output statistics.

Images