JP2007114885A - Classification method and apparatus based on image similarity - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and an apparatus for obtaining information on a change such as an operation of an imaging target from a captured image and classifying a change pattern in the image in parallel with the imaging.
SOLUTION: A plurality of imaging means capable of imaging an object that can operate, a storage means for recording at least image information obtained by the imaging means, and an operation for analyzing at least the image information obtained by the imaging means And a method of classifying a pattern of change in a plurality of images obtained by imaging the target object, the target object is continuously imaged by the imaging means, and the difference between the image information continuously captured The amount of change in image information generated during the imaging unit time is obtained, and the change pattern within a predetermined target time from the current time is classified as a variable (for example, using cluster analysis), thereby obtaining a predetermined amount from the current time. The image information within the target time is classified almost in real time.
[Selection] Figure 2

Description

  The present invention relates to a method of classifying a captured image based on similarity, such as a characteristic action that occurs in a habitual manner many times by a person, and a device that performs the classification.

In recent years, with the mass production of electro-electric light receiving elements, the price of imaging means such as video has been drastically reduced. Now that it is easy to obtain inexpensive monitoring imaging means, it has become easy to install a large number of monitoring imaging means and monitor company facilities from various viewpoints.
However, as the number of image pickup means increases, the number of images picked up by them increases, so the current situation is that the image pickup means and image data are used.
When a display unit such as a display associated with the image pickup unit in a one-to-one relationship is considered, the size and installation space of the display unit are also a problem. It is difficult for a single observer to monitor as many as 50 or 100 units.

For example, there is a method in which one display means is assigned to four image pickup means, the time is divided into four, and the image pickup means for supplying image information to the display means is automatically switched.
However, this means that 3/4 video has been thrown away, and events that you want to monitor may occur in the meantime, so it is not a perfect method.

On the other hand, there is a method in which only a moving image is displayed and a non-moving image is discarded.
However, for example, when operations occur simultaneously in 50 imaging devices, it is eventually necessary to check images for 50 devices.

  When highly similar imaging data is output from a plurality of imaging means, there is no problem even if one of the imaging means is stopped. However, this similarity determination largely depends on human experience and subjectivity, and there has been no apparatus that automatically performs this.

For example, if the audience is listening to the speakers at the hall, some of the audience may be whispering, some may be twisted, and others may be stationary. In order to classify their actions, it is a conventional general method to prepare an imaging means that can show the entire audience, and then determine and classify the actions of each audience using human hands. The only way to determine the status of acceptance of a topic is to observe the entire audience during the lecture and make a judgment based on the situation.
Even if an imaging device is installed for each audience, the current technology will eventually classify each audience manually.
If the speaker can know the audience's whispering, neck twisting, immobility ratio change, and the matching state of the audience's movement during the time from the start of the lecture to the present, talk accordingly. The contents of can be rationally changed in a flexible manner.

In addition, it is a well-known phenomenon that the action of the head or the like appears even when speaking without looking at the other party's face, for example, when making a call. Humans tend to unintentionally express head movements that are unique to each person during dialogue, and it is often the so-called individuality that has a habit depending on the individual. If the meaning of non-linguistic information related to the motion of the imaging target, such as the head motion of the human body, is known, it can be used in various situations, such as being used to assist a man-machine interface.
However, conventionally, non-linguistic information related to the operation of an imaging target has been effectively discarded without being used or classified without being used.

  SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a method for obtaining information related to changes such as an operation of an imaging target from a captured image, and classifying the change pattern in the image based on similarity, and an apparatus for implementing the method. .

In order to solve the above-described problems, a moving image classification method according to the present invention includes a plurality of imaging units capable of imaging an object that can be operated, and a memory that records at least image information obtained by the imaging unit. And a calculation means for analyzing at least the image information obtained by the imaging means, and in the method for classifying the pattern of changes in a plurality of images obtained by imaging the object, the object is continuously detected by the imaging means. The amount of change in the image information generated during the imaging unit time is obtained from the difference between the image information captured continuously, and the similarity is obtained using the change pattern within the predetermined target time from the current time as a variable. By comparing and classifying actions with high similarity as the same action, the image information within a predetermined target time is classified at any time according to the type of action. To.
A method of classifying by similarity includes, for example, cluster analysis by digitizing and comparing each feature.

  Here, the amount of change in the image information generated over time is Fourier transformed, and the frequency and intensity included in the continuous captured image information obtained thereby are used as variables for similarity classification. You may contribute to classification. As a result, it is possible to reduce a plurality of continuous operations or combined operations to a single operation or an individual operation, and avoid being classified as another operation because it cannot be isolated.

 For example, when cluster analysis is used for classification, the analysis may be performed using a variable weighted to the timing at which a change occurs in the captured image information, and may be contributed to effective classification according to the purpose.

 An image representative of each classified similar operation may be output to the display means to contribute to visual recognition.

 Typical change patterns are classified into four states: frequent periodic changes, infrequent periodic changes, non-periodic changes, and no changes, and the classification status to the four states is displayed immediately after classification. You may output and show and contribute to utilization of a classification result.

 A similarity relationship between a plurality of change patterns may be determined, and the captured image may be hierarchically displayed on the display unit based on the similarity relationship between the change patterns, thereby contributing to the utilization of the classification result.

 A typical change pattern example may be prepared in advance, and the input of the image information classified as the typical example may be immediately output to the display means to contribute to the utilization of the classification result.

The apparatus of the present invention for performing such classification is obtained by a plurality of imaging means capable of imaging an object that can operate, a storage means for recording at least image information obtained by the imaging means, and an imaging means. An apparatus for classifying change patterns in a plurality of images obtained by imaging a target object, and an imaging means for continuously imaging the target object, and an imaging means. The amount of change in the image information generated during the imaging unit time is calculated from the difference between the storage means for recording the captured image information and the image information stored in the storage means, and a predetermined target time from the current time And calculating means for classifying image information within a predetermined target time from the current time in substantially real time by analyzing the change pattern in the image as a variable. . The analysis of change patterns as variables is based on classification according to similarity.
In the above description, the classification in substantially real time means that the classification process is performed immediately after the unit operation of the classification target is completed.

  According to the present invention, since a captured image can be classified immediately after operation from changes in continuously captured images, the classification state can be used as needed or used to detect the occurrence of a specific object or event.

Embodiments of the present invention will be described below with reference to the drawings.
In this embodiment, moving images captured simultaneously by a plurality of image capturing means such as video cameras are targeted for classification. In addition, image data output from the image pickup means is input to a computer connected to a display means such as a CRT, which has a calculation means such as a CPU and a storage means such as an HD, and is processed. Here, it is assumed that the distance between the imaging unit and the subject and the data transmission delay between the imaging unit and the computer can be substantially ignored.

In order to compare a plurality of captured images and determine whether they are “similar” or “similar”, the following is performed.
Suppose that the moving image M with the image pickup means α is picked up, and the same moving image M is picked up by the image pickup means β with a delay of 1 second. In this case, although there is a time difference of 1 second, the content of the moving image is the same, but here it is determined that it is not similar. Here, when different imaging units set at different positions image the same subject and obtain similar moving images M, they are determined to be “similar”. Therefore, what is important is the same timing from the start to the end of the operation and the similarity of the operation content itself.

In order to facilitate the processing, the moving image information is processed along the time axis.
The processing here means expressing the difference between frames, for example, the sum of absolute values of luminance differences, as a numerical value in each pixel of one frame image constituting the moving image.

 If the imaging data is monochrome image information, the luminance at coordinates (x, y) in the frame image captured by the imaging means at time t can be expressed as: If the imaging data is color image information, the red, green, and blue intensities at coordinates (x, y) in the frame image captured by the imaging means at time t can be expressed as,, respectively.

The following equation is generally established between the luminance and the intensity of red, green and blue.
(Equation 1)

 Here, Cr, Cg, and Cb are correction coefficients for red, green, and blue, and generally, Cr = 0.298912, Cg = 0.586611, and Cb = 0.114478 are often used as empirical rules. It is also desirable to change the values of Cr, Cg, and Cb depending on conditions such as the characteristics of the imaging means and the light source of the image projected by the imaging means. From this equation, the following method can be used for both monochrome images and color images.

Assuming that an image is composed of vertical h pixels and horizontal w pixels, the numerical value represented by d (t) in the following expression is the previous time by Δt corresponding to the time of one frame at time t, that is, time (t−Δt). ) Is the sum of absolute values of luminance differences from the image.
(Equation 2)

FIG. 1 is a graph in which information d (t) for each frame obtained from three imaging means α, β, and γ set at different positions is arranged on the time axis.
Shown here is the operation that occurred during the comparative sampling period S, which is sufficiently larger than the time Δt between frames. The inter-frame time Δt is, for example, Δt = 0.0667 seconds if the imaging apparatus can capture 15 frames per second. The comparison sampling period S is arbitrary, but for example, 5 seconds or the like can be applied.

The selection of the various variables obtained from this figure is important. There are some that are important and some that are not important in this classification of similarity as follows.
For example, the total motion amount Sα (= Σd (t)), Sβ, Sγ in each imaging means generated during the sampling period S in this case is Sβ <Sα = Sγ.
Therefore, considering only the total motion amount, the image information by the imaging unit α is equal to the image information by the imaging unit γ, and the image information by the imaging unit α and the image information by the imaging unit γ are different from the image information by the imaging unit β.
However, it is unlikely that the imaging unit α and the imaging unit γ have captured the same subject. Rather, whether or not the same subject has been imaged is similar to the imaging unit α and the imaging unit β in terms of the timing of operation timing.
As will be understood from this example, the comparison of the total operation amount Σd (t) between the imaging means is not important. One factor is that the total operation amount Σd (t) depends on the distance between the imaging means and the subject. Rather, it is necessary to emphasize the coincidence of operation timing.

The coincidence of timing can be measured using various conventionally known methods, but here, an operation non-operation determination variable D (binary number) is used as an example.
The operation / non-operation determination variable D is expressed in binary notation with the number of digits s, and each digit corresponds to d (1) to d (s). 1 if d (t)> 0, 0 if d (t) = 0. Ideally, d (t) = 0 for a movie that has been shot in a static place, but it is common for electrical and electronic noise to occur between the previous and next frames. In general, N is set to 1 if d (t) ≧ N, and is set to 0 if d (t) <N, to remove the influence of minute noise. In this way, for example, Dα = 01011011010101 is represented.
s is the number of frames used for comparison, and can be expressed as a quotient by the inter-frame time Δt of the sampling period S. For example, if an imaging apparatus capable of capturing 15 frames per second and the sampling period is 5 seconds, s = S / Δt = 5 / 0.0667 = 75 [frame].

For example, it is assumed that the operation / non-operation determination variable Dα in the imaging unit α is represented as 1100111111, the operation / non-operation determination variable Dβ in the imaging unit β is represented as 1100111111, and the operation / non-operation determination variable Dγ in the imaging unit γ is represented as 1111110011.
Then, the sameness of the timing of the image information by the imaging means α and the image information by the imaging means β can be determined by examining the Hamming distance between Dα and Dβ. The Hamming distance between Dα and Dβ is 0, and the Hamming distance between Dα and Dγ is 4. Even when noise and other disturbances that cannot be removed are taken into account, the smaller the hamming distance, the higher the timing identity.
Note that the Levenshtein distance and the like can be used as appropriate for the sameness of the timing of the image information as well as the Hamming distance.

Next, considering the content of the operation, what is important in the operation information is a change pattern of the operation.
For example, the imaging means α images rhythmic long-period vibrations such as only the movement of an arm of a certain person Q, and the imaging means β performs movements of an arm of a certain person Q and rhythmical short-period vibrations of the neck. Suppose that an image is taken. In this case, for example, if the motion generation part is regarded as a composite wave of a plurality of sin waves having different periods and intensities, it is possible to obtain the frequency and intensity included in the moving image by performing Fourier transform on the composite wave. It is.

Classification based on similarity, for example, cluster analysis, is appropriately performed using the operation timing identity, the operation content period, and the intensity information obtained as described above as variables. In the case of FIG. 1, the imaging unit α and the imaging unit β are close to each other to form one cluster, and the imaging unit γ is located farther than the cluster formed by the imaging unit α and the imaging unit β. .
Thereby, although it is a rough judgment, classification can be performed by determining that the imaging unit α and the imaging unit β capture the same subject, and that the imaging unit γ captures a different subject.
Since it is determined that the imaging unit α and the imaging unit β are capturing the same subject, information from either one is sufficient. Therefore, useful information here is the imaging unit α (or the imaging unit β) and the imaging unit. It becomes information of γ.

 If the position of the image pickup means is not changed, the image pickup means that are easily classified into the same cluster can be understood by accumulating information obtained by this method. Imaging devices that are likely to be classified into the same cluster capture an image of a close space, and conversely, imaging devices that are easily classified as a different cluster capture an image of a distant (different) space.

On the other hand, with respect to an already captured image, it is more efficient to reduce the number of calculations and analyze it by reducing the number to an arbitrary n rather than analyzing all m moving images depending on the number of imaging means. It is assumed that the narrowing-down classification of n similar images has been completed by the above-described means. n corresponds to the number of moving images selected by the user. For example, if m = 10 and n = 2, it means that two are selected from ten moving image sources.
Next, what is needed is an analysis of each of the n moving images.

For analysis of moving images, conventionally known methods can be used as appropriate. Here, a simple example is shown in which the variable d (t) that has already been calculated by the above-described means is reused for determination. The method according to this example can reduce the amount of calculation, and is intended to suppress the amount of calculation rather than the accuracy of the operation analysis.
For example, if a moving image obtained from one imaging means is analyzed in real time, the current computer speed is sufficient. However, if a large number of image pickup means are used, or if the arithmetic unit itself is inexpensive and low power consumption type, it is meaningful to give priority to the reduction of the calculation amount.

An example of motion analysis is Japanese colloquium that is spoken between friends.
The common spoken language used by Japanese people often uses ambiguous sentence patterns compared to common spoken languages in other countries, such as omission of the subject, omission of the end of the sentence, and conversation at a timing that invites co-speaking. Many non-verbal actions such as back-channeling can be seen during conversation with nature.
Therefore, here, the ultra-long cycle operation is discarded as meaningless or requiring high-level processing for determination. For example, for the purpose of explanation, keep swinging a swivel chair slowly from side to side due to large long-term non-periodic motion, such as drawing a virtual map in the air with your fingertips for explanation, boredom, etc. Such an operation with a very long period is automatically ignored if it is longer than the sampling period S.

Assuming that the orientation of the imaging means, the orientation of the subject, and the distance between them are very different, it is possible to measure the increase / decrease period of d (t) depending on the luminance difference from the image one frame before. The most reasonable.
Even if d (t) from the current time t of the moving image obtained from a specific imaging means α to the sampling period S is analyzed by the above-described method, it can be classified into, for example, the following four motions as motion analysis. it can.
(B): The most frequent periodic movement (change)
(B): Other periodic operations (changes)
(C): Aperiodic operation (change)
(D): Non-operation (immobility)
Although there are individual differences, when a moving image obtained from a specific imaging means α is general back-channeling, these motion classifications a to i occur frequently other than whispering (b) and whispering, respectively. It can be roughly regarded as a periodic motion (b), a non-periodic motion (c) such as a so-called gesture, and a non-motion (d).

FIG. 2 is an explanatory diagram showing the main part of the system configuration of this embodiment.
Each moving image obtained from the m imaging devices is input to the moving image receiving device, subjected to calculation processing by the arithmetic device, and recorded and stored in the storage device as d (t). Each d (t) may be displayed on the analysis result display device.

The arithmetic unit analyzes d (t) of a series of continuous motion parts at appropriate times that are preferably longer than the sampling period S, and performs classification computation processing into four states of motion classifications A to D. This means that, for example, moving images from a specific imaging device α that captures the motion of the person Q are classified into four states of motion classifications a to d.
If the person Q is performing some kind of action at present or before the maximum sampling period S (when Dα> 0), matching is performed to determine which of the action classifications A to C is the most similar. This examines the similarity of the operation period and intensity indicated by d (t) within the sampling period S as described above, and determines which of the operation classifications A to C is the most similar. A conventionally known cluster analysis using the period and intensity as variables may be used. For example, if it is determined that the person is most similar to the motion classification A, it can be determined that the person Q has whispered within the sampling period S.

The classification results of similar moving images obtained by comparing d (t) within the sampling period S from the current time, and the analysis results of continuous motion parts of all m d (t) at appropriate times are It may be displayed on the analysis result display device.
Also, as a result of the similarity classification, n dissimilar moving images selected from all m imaging devices may be displayed on the moving image display device in accordance with the switching instruction transmission from the arithmetic device to the moving image switching device. .

By analyzing which of the motion classifications a to d previously classified in the above-mentioned manner, the motion generated within the sampling period S from the current time indicated by n moving images is less than n q It is possible to show human actions immediately after classification.
Of all the q people, how many people are whipping (b), how many are periodic movements (b) other than whipping, how many are non-periodic movements (c), how many are immobile (d), Each ratio may be immediately displayed on the analysis result display device after classification.

Motion analysis within a video can also be applied to identification of people and special actions.
For example, the following d (t) pattern is recorded in advance in the storage device.
(B): Movement of a housekeeper trying to open a window normally (b): Typical movement of a thief trying to break a window and invade (c): Other actions (d): Non-operation (immobility) )
If the computing device determines whether the change in d (t) within the currently recorded sampling period S is most similar to any of the above operation classifications a to d, the thief breaks the window and enters When the operation information (b) to be obtained is obtained, an output such as issuing an alarm by an attached alarm device may be performed.
In this way, recording d (t) corresponding to the motion rhythm and comparing it with a typical motion can be used in various situations.

  According to the present invention, captured images can be classified almost in real time, and can be used for the use of the classification status as needed or for detection by classification. For example, it can be applied to a variety of situations, such as when a speaker changes the story according to the audience's response classification, or when a specific event such as a house intrusion is detected and an alarm is issued. It is.

A graph in which information d (t) for each frame obtained from three imaging means α, β, γ set at different positions is arranged on the time axis Explanatory drawing which shows the principal part of the system configuration | structure of a present Example

Claims (8)

A plurality of imaging means capable of imaging a target object capable of movement;
Storage means for recording at least image information obtained by the imaging means;
An arithmetic means for analyzing at least the image information obtained by the imaging means,
A method of classifying patterns of change in a plurality of images obtained by imaging a target object,
The object is continuously imaged by the imaging means,
From the difference between these continuously captured image information, the amount of change in the image information that occurred during the imaging unit time is obtained,
By classifying the change pattern within a given target time from the current time as a variable according to similarity,
A classification method based on similarity of images, characterized in that image information within a predetermined target time from the current time is classified in substantially real time. The classification method according to the similarity of images according to claim 1, wherein the amount of change in image information is subjected to Fourier transform, and the frequency and intensity included in continuous captured image information obtained thereby are used as classification variables. The classification method according to the similarity of images according to claim 1 or 2, wherein classification is performed using a variable that weights the timing at which a change occurs in captured image information. The classification method according to the similarity of images according to claim 1, wherein each characteristic image classified by similarity is output and displayed on a display means. Typical change pattern
Classify into four states: periodic changes that occur frequently, periodic changes that do not occur frequently, aperiodic changes, and no changes.
The classification method according to the similarity of images according to claim 1, wherein the classification state indicating the four states is output and displayed on the display means in substantially real time. Define similar relationships between multiple change patterns,
The classification method according to the similarity of images according to claim 1, wherein the captured images are hierarchically displayed on the display means based on the similarity between the change patterns. Prepare variables for classification that show typical change patterns in advance.
The classification method according to the similarity of images according to any one of claims 1 to 6, wherein when image information classified into the typical change pattern is input, the image information is immediately output to the display means. A plurality of imaging means capable of imaging a target object capable of movement;
Storage means for recording at least image information obtained by the imaging means;
An arithmetic means for analyzing at least the image information obtained by the imaging means,
An apparatus for classifying patterns of change in a plurality of images obtained by imaging a target object,
Imaging means for continuously imaging the object;
Storage means for recording image information continuously captured by the imaging means;
From the difference between the image information accumulated in the storage means, the amount of change in the image information generated during the imaging unit time is calculated,
By analyzing the change pattern within a given target time from the current time as a variable,
A classification device based on similarity of images, comprising: arithmetic means for classifying image information within a predetermined target time from a current time in substantially real time.

Images