RU2488164C1 - Method for localisation of road signs and recognition of their groups and subgroups - Google Patents

Abstract

FIELD: information technologies.SUBSTANCE: method for localisation of road signs and recognition of their groups and subgroups in video frames of images consists in identification of areas of signs location by means of viewing of lines and columns of the image in a sliding window, if orders of colour alternation in the window match the reference one, the appropriate lines and columns are memorised, and the area of image on their crossing is identified as suspicious for location of a sign in it, in which they further identify an area limited with a red border of the sign, and in it, using lines and columns, they build two colour-independent histograms of all counts, which are normed in such a way that their ranges and the number of counts match with the appropriate characteristics of reference histograms, then using the normalised histograms they form a common histogram, and a corresponding vector, which is compared to vectors of references in the specified groups of signs, the decision on whether the current sign belongs to one of the groups of signs is taken on the basis of the criterion of the minimum measure of vector proximity, then in each group they define sign subgroups using histograms of black colour dots, which are formed and compared on the basis of the same rules as histograms used in detection of sign groups.EFFECT: reduced time of localisation and increased reliability of road signs recognition.3 dwg

Description

The invention relates to the field of digital image processing and may find application in active safety systems of automobiles to generate additional warning signals to the driver.

Known methods for localizing characters in images described in the works:

Shneier, M. Road sign detection and recognition [Text] / M.Shneier // Proc. IEEE Computer Society Int. Conf. on Computer Vision and Pattern Recognition. - 2005. - P.215-222.

Torresen, J.A. Camera Based Speed Limit Sign Recognition System [Text] / J. Torresen, J.W. Bakke, Y. Yang // 13th ITS World Congress and Exhibition. - 2006. - P.115-129.

Saturino Maldonado-Bascon, Sergio Lafuente-Arroyo, Pedro Gil-Jimenez Hi-lario Gomez-Moreno and Francisco Lopez-Ferrers. Road-Sign Detection and Recognition Based on Support Vector Machines // IEEE Transaction on Inteligent Transportation System, vol. 8, No.2, June 2007.

Ren, F. General traffic sign recognition by feature matching [Text] / F. Ren, J. Huang, R. Jiang, R. Klette // IEEE 24th Int. Conf. Image and Vision Computing. - 2009. - Issue: Ivcnz. - P.409-414.,

in which the color components are analyzed to determine the area of the sign. The approach to detecting signs based on color filtering is not demanding on computational resources, however, well-known methods cannot always distinguish (for example, red) a sign from other objects of the same color, nor can they distinguish types of characters with a border of the same color (warning and prohibiting), which leads to a significant increase in computational costs in subsequent stages and, ultimately, to a decrease in recognition quality, because the current image has to be compared with a large number of character standards in the database.

The closest prototype of the invention is the “Device, method and computer program for determining traffic signs in the image” (patent No. US 2010/0067805, IPC G06K 9/46, published 03/18/2010), which implements stages of detection, tracking, normalization and recognition. To establish the fact of the presence of a sign on each current image (video frame), the sizes of fragments of ellipses and straight lines, which are boundary for prohibitory and warning signs, are determined. At the same time, Haar color-sensitive wavelets constructed in various (up to 9) color representations are used as signs. Digital color filters are used to measure symptoms. The final decision is made by comparing the results of recognition in various color representations of images on a sequence of frames.

The disadvantage of this method is the large delay in the recognition and generation of warning signals to the driver, due to the fact that the identification of areas of signs with digital color filters is carried out by determining the sizes of fragments of ellipses and straight lines, using a large number of representations on a sequence of frames, which requires large computational costs . In addition, at the detection stage, groups of warning, prohibiting, and other signs, as well as subgroups in groups of signs, determined by the similarity of images, are not classified, therefore, at the final stage of recognition itself, the selection is made from the whole set of signs. Since it is necessary to compare the current image with a large number of standards, in the end, this leads to a large delay and reduces the reliability of recognition.

The problem to which the invention is directed, is to reduce the time of localization and increase the reliability of recognition of road signs.

This goal is achieved by the fact that in the method of localizing road signs and recognizing their groups and subgroups, which consists in highlighting the areas of signs on the video frames with digital color filters, according to the invention, the area of the location of the sign is carried out in two stages, in the first stage to highlight areas of the signs look at the rows and columns of the image with a sliding window, the size of which is set based on the maximum possible at a given allowable resolution, for each the position of the window determines the order of the alternation of colors and compares it with pre-established order of alternating colors on the standards of road signs, if the order of alternating colors coincides, the corresponding rows and columns are remembered, and the image area at their intersection is identified as suspicious of finding a sign in it, at the next stage the suspicious area found, an area bounded by the red border of the sign is selected, and two histograms, independent of color, are built on the rows and columns of the selected area of all samples, both histograms are normalized so that their ranges and the number of samples coincide with the corresponding characteristics of the standard histograms, then from the constructed normalized histograms, by adding them, form a common histogram and its corresponding vector, this vector is compared, in the sense of a given measure of proximity, with the formed according to the same rule, by vectors of standards of given groups of characters, the decision on whether the current character belongs to one of the groups of characters is made according to any known criterion min minimum of the measure of proximity of the vectors, then each group define subgroups marks Histogram black dots, which are formed and compared to the same rules as the general, independent of the color histogram used in the determination of a group of characters.

The following is a description of an example of an embodiment of the claimed proposal, which implements a method for localizing and recognizing groups and subgroups of signs that have a red border or a red background: warning, prohibiting, give way sign, stop sign, no traffic sign (prescriptive and information signs are processed separately according to the same principle, but using blue samples).

Figure 1 shows examples of alternating colors of characters: a) "Overtaking is prohibited"; b) “Give way”; c) “Speed limit” (50 km / h), where the designations for colors are accepted: k - red, b - white, h - black.

Figure 2 shows the normalized general histograms of groups of signs: a) warning; b) “Give way”; c) prohibiting "No entry"; e) prohibiting "Movement without stopping is prohibited."

Figure 3 shows the normalized histograms of samples of black color reference images from the group of warning signs: a) for a subgroup including signs 1.20.1-1.20.3 - "Narrowing the road"; b) for the subgroup including signs 2.3.2-2.3.7 - “Adjacent secondary road” (GOST R 52290-2004).

At the first stage, in order to identify areas suspicious of finding the image of a road sign, rows and columns of the image are sequentially scanned by a sliding window, the size of which is set based on the maximum possible for a given allowable resolution. Permissible resolution is determined by the requirement for reliable character recognition and is provided by the registration of video frames at a certain distance. At each position of the window, the order of alternating colors is determined and remembered. The formed tuple describing the order of the alternation of colors is compared with possible predetermined tuples of colors for the standards of road signs. For example, for most warning signs, this tuple consists of alternating “white” and “black” colors, moreover, the tuple, for almost all signs, begins and ends with a sign indicating “red” color. Figure 1 shows examples of tuples of red, white and black colors for the warning sign "overtaking is prohibited", "give way" and the prohibiting sign "maximum speed limit" (up to 50 km / h).

If the formed tuple coincides with one of the tuples of the base of reference images, the corresponding rows and columns are remembered, and the image area at their intersection is identified as suspicious of a sign in it. At the intersection of the middle row and the middle column of the selected area, the coordinates of the center point are determined. Further, in the highlighted suspicious area, the inside of the sign is “cut out”, limited by a red border that looks like a circle for prohibiting and a triangle for warning signs.

The selection of the inner region of the sign bounded by the red border is carried out using an algorithm of the type "fill". The “fill” algorithm consists in viewing and saving the coordinates of the samples in all directions from the center point, which was determined at the intersection of the middle row and middle column of the selected area at the previous stage, to the periphery until the first intersection with the red border. Next, the “cut out” image is viewed in rows and columns, and in each row and each column, the total number of pixels in the selected area is counted. The result is two histograms (for rows - i and columns - j) of all analyzed samples in the selected area.

Further, the histograms are normalized. To do this, determine the coordinates of the beginning - i ₁ , j ₁ and the end - i _n , j _{n of} each of the two histograms, i.e. serial numbers of samples of the first and last nonzero values. These coordinates specify the dimensions of the rectangle into which the selected area of the sign image is inscribed. The obtained sizes are compared with their minimum possible values given taking into account the permissible resolution required for character recognition.

, p(j), $$j=\overline{j_1,j_n}$$

перенумеровывают по правилу:Both obtained histograms are scaled so that the number of samples of each histogram is equal to the number of samples i _e , j _{e of} reference histograms of groups of signs. For this, the values of the histogram samples p (i), $$i=\overline{i_{\mathrm{one}},i_n}$$

, p (j), $$j=\overline{j_{\mathrm{one}},j_n}$$

renumber by the rule:

, $$i=\overline{i_1,i_{э}}$$

, $$p\left(i\right)=p\left(\overline{i}\right)$$

, $$i=\overline{i_{\mathrm{one}},i_{\mathrm{uh}}}$$

,

, $$j=\overline{j_1,j_{э}}$$

, $$p\left(j\right)=p\left(\overline{j}\right)$$

, $$j=\overline{j_{\mathrm{one}},j_{\mathrm{uh}}}$$

,

Where

, $$\overline{j}=[a_{j}j]$$

, $$\overline{i}=[a_{i}i]$$

, $$\overline{j}=[a_{j}j]$$

,

, $$a_j=\frac{L_j}{\left(j_n-j_1\right)}$$

- коэффициенты преобразования, a L_i, L_j - заданные эталонные длины гистограмм по направлениям изменения индексов i, j соответственно. $$a_i=\frac{L_i}{\left(i_n-i_{\mathrm{one}}\right)}$$

, $$a_j=\frac{L_j}{\left(j_n-j_{\mathrm{one}}\right)}$$

are the conversion coefficients, a L _i , L _j are the specified reference histogram lengths in the direction of change of the indices i, j, respectively.

The obtained scaled histograms are normalized according to the rule:

, $$\overline{p}\left(i\right)=\frac{p\left(i\right)}{{\displaystyle \sum _{i=\overline{i_{\mathrm{one}},i_{\mathrm{uh}}}}p\left(i\right)}}$$

,

$$\overline{p}\left(j\right)=\frac{p\left(j\right)}{{\displaystyle \sum _{j=\overline{j_{\mathrm{one,}}{j}_{\mathrm{uh}}}}p\left(j\right)}}$$

and by attaching two histograms, a common histogram is formed that characterizes the distribution of all samples in the selected area.

и устанавливают принадлежность к одной из них - $${\widehat{Z}}_m.$$

В данном случае для установления степени «похожести» гистограмм использовался критерий максимума функции корреляции:Next, the normalized general histogram is compared with the given general histograms p (i), p (j) of the standards of recognizable sign groups given for the same values of L _i , L _j - Z _m , $$m=\overline{\mathrm{one,}M}$$

and establish belonging to one of them - $${\widehat{Z}}_m.$$

In this case, to establish the degree of "similarity" of the histograms, the criterion of the maximum correlation function was used:

, $$m=\overline{\mathrm{1,}M}$$

, $${\widehat{Z}}_m:Q\left({\widehat{Z}}_m\right)=\underset{m}{\max }Q\left(Z_m\right)$$

, $$m=\overline{\mathrm{one,}M}$$

,

Where $$Q\left(Z_m\right)=\frac{〈\overline{p}\left(i\right),P\left(i\right)〉}{\Vert \overline{p}\left(i\right)\Vert \Vert P\left(i\right)\Vert }.$$

Figure 2 shows examples of normalized general histograms of groups of signs: a) warning; b) “Give way”; c) prohibiting "No entry"; e) prohibiting "Movement without stopping is prohibited."

After groups of signs are determined by the general normalized histograms, subgroups of signs are determined in each group. To do this, in the selected area according to the same rules described above, normalized histograms of black samples are constructed, which are compared by the criterion of the maximum of the correlation function with normalized histograms of black samples of reference images of signs. As a result of this comparison, subgroups of groups of warning and prohibiting signs are determined.

Figure 3 shows examples of normalized histograms of black samples of reference images from the group of warning signs: a) for a subgroup including signs 1.20.1-1.20.3 "Narrowing the road"; b) for a subgroup including signs 2.3.2-2.3.7 “Junction of a secondary road” (GOST R 52290-2004).

Then, using the coordinates of the boundary points determined by the general histogram of the selected area and the coordinates of the corresponding points on the reference images, an affine transformation of the selected area of the sign is carried out in order to bring the current image to the reference in scale and angle of rotation.

The final stage of recognition of a particular character can be implemented by any of the known methods. In the present example, the selected area of the current image is converted to binary, and a threshold is set at 0.75 of the average brightness value. All reference images are also presented in binary form according to the same rule. Next, the selected areas of the current and reference binary images are compared. The decision is made at the minimum of the relative number of mismatched pixels on the set of characters included in the subgroup, which is defined in the previous steps.

In this case, the recognition time is significantly reduced, and the reliability increases, because the number of characters with which the current binary image is mapped is relatively small due to preliminary recognition of the group to which the current character belongs by the general histograms and the subgroup by the “black” histograms. For example, the shape of the “black” histograms confidently recognizes a subgroup that includes the following signs: “Intersection with a minor road” and “Adjacent secondary road”. Therefore, to determine a specific warning sign in this subgroup, it is enough to solve the problem of choosing from 7 characters (signs 2.3.1-2.3.7, GOST R52290-2004).

A significant reduction in the overall computational costs for the localization and recognition of road signs, as well as an increase in the recognition reliability, is ensured by the introduction of recognition procedures for groups of signs by common normalized histograms, as well as subgroups in groups by normalized histograms of "black" color. Thus, the claimed method meets the criteria of the invention of "novelty." Reducing overall computing costs reduces the time it takes to generate warning signals to the driver, which ultimately provides improved road safety.

Claims (1)

A method of localizing road signs and recognizing their groups and subgroups, which consists in highlighting the areas of signs on the video frames with digital color filters, characterized in that the area of the location of the sign is carried out in two stages, at the first stage, lines and columns of the image are sequentially viewed a sliding window, the size of which is set based on the maximum possible at a given permissible resolution, at each position of the window, the order of black is determined colors and compare it with pre-established color rotation patterns on the standards of road signs, if the color rotation orders coincide, the corresponding rows and columns are remembered, and the image area at their intersection is identified as suspicious of the sign in it, at the next stage in the suspicious area found the area bounded by the red border of the sign, and in the selected area, two histograms of all samples, both histograms normal so that their ranges and the number of readings coincide with the corresponding characteristics of the reference histograms, then from the constructed normalized histograms, by forming them, a common histogram and its corresponding vector are formed, this vector is compared, in the sense of a given proximity measure, to the vectors formed according to the same rule standards of given groups of signs, the decision on whether the current sign belongs to one of the groups of signs is made according to any known criterion of the minimum measure of proximity of vectors, then to zhdoy group define subgroups marks Histogram black dots, which are formed and compared to the same rules as the general, independent of the color histogram used in the determination of a group of characters.

Images