CN113447111B - Visual vibration amplification method, detection method and system based on morphological component analysis - Google Patents

Abstract

The invention discloses a visual vibration amplification method, a detection method and a system based on morphological component analysis, which belong to the technical field of computer vision. Acquiring a video file including the target object, and determining the region of interest of the target object in each frame of the video file; using a quality factor adjustable wavelet dictionary and a discrete sine-cosine dictionary to represent the structural components and texture components of each frame of video respectively ; Using the threshold selection algorithm based on the generalized Gaussian density distribution to determine the threshold value of the structural components; using adaptive MCA to separate the image structural components and texture components; A large video is constructed, and at the same time, the visual vibration signal is extracted for the structural components to realize the measurement of multiple vibration frequencies.

Description

Visual vibration amplification method, detection method and system based on morphological component analysis

technical field

The invention belongs to the technical field of computer vision, and in particular relates to a visual vibration amplification and detection method and system based on morphological component analysis.

Background technique

Frequency information is an important basis for vibration analysis, which is widely used in mechanical or building structural health inspection and quality inspection. People generally detect the vibration frequency by analyzing the vibration signal of the object. In the field of vibration measurement, the measured object is often stimulated by different excitation sources, and it appears as a mixture of multiple frequencies in the frequency spectrum.

In the traditional vibration detection, the method of measuring the frequency of the measured object is mainly to paste the acceleration sensor on the surface of the measured object to obtain the acceleration signal, obtain the displacement signal according to the acceleration signal, and obtain the frequency spectrum of the measured object for the displacement signal. There are obvious shortcomings in the type of vibration detection method: due to the weight of the acceleration sensor itself, it is difficult to detect tubular objects, and the existence of the acceleration sensor itself will affect the natural frequency of the measured object; and the sensor sticking process is time-consuming and labor-intensive. Sensor fusion is also required. The non-contact laser sensor can only perform single-point measurement. To perform full-field measurement, multiple laser sensors are required, which is expensive and has the disadvantage of time delay.

Therefore, a non-contact vibration signal acquisition method based on vision has been proposed. The main technical idea is: each pixel is used as a visual sensor, and the vibration signal at any point can be extracted. Field measurement. However, due to changes in the lighting conditions of the camera, the uneven reflection of the object surface disturbs the vibration signal extracted from the video, resulting in inaccurate measurement results.

In recent years, some people have begun to study the changes in illumination. For example, phase-based motion amplification methods are widely used to measure and amplify imperceptible motion in videos (Wadhwa, M. Rubinstein, F. Durand, and W. Freeman, "Phase- based video motion processing," ACM Trans. Graph., vol. 32, no. 4, Jul. 2013, Art. no. 80.). (A. Davis et al., "Visual vibrometry: Estimating materialproperties from small motions in video," IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 4, pp. 732–745, Apr. 2017) ; Extract the motion signal from the video to calculate the frequency of the vibrating object, which is used to infer the properties of the material, such as Young's modulus, stiffness, damping, geometric dimensions, etc. (Chen J G, Wadhwa N, Cha Y J, et al. Modal identification of simple structures with high-speed video using motion magnification [J]. Journal of Sound and Vibration, 2015, 345: 58-71.) Using this method for measurement Vibration frequency and displacement of cantilever beams and plastic pipes.

However, in some application scenarios, taking the engine frequency spectrum at idle speed as an example, under normal circumstances, the six modal frequencies of the engine assembly mounting system are all below 25hz. Therefore, the vibration spectrum of the engine idling state is often a mixture of multiple frequencies below 25 Hz. However, due to the above-mentioned defects of the phase wrapping or unwrapping algorithm, the extracted vibration waveform is inaccurate, which affects the accuracy of micro-vibration frequency measurement, especially when there are multiple mixed frequencies in the system, it is easier to detect non-existent anomalies frequency.

To sum up, the existing vibration measurement technology cannot effectively solve the measurement problem of micro-amplitude vibration. In this case, for the measurement of micro-amplitude vibration, a visual vibration amplification method is urgently needed, which can reduce the interference of shooting light and accurately detect multiple modal frequencies of the measured object.

SUMMARY OF THE INVENTION

In order to solve or at least partially solve the above problems, the present invention provides a visual vibration amplification method based on morphological component analysis, which removes the interference of photographing light and can accurately detect the vibration frequency caused by the deviation of the position and angle of the rubber parts inside the engine, Multiple modal frequencies of the engine can be obtained.

In order to achieve the above object, the present invention provides the following technical solutions:

A first aspect of the present invention provides a visual vibration amplification method based on morphological component analysis

Obtain a video file including a target object, and determine the region of interest of the target object in each frame of image in the video file;

Constructing a structural component representing the region of interest in each frame of the video file; constructing a texture component representing the region of interest in each frame of the video file;

The structure component and texture component in the region of interest are separated, and the separated structure component and texture component are combined with Euler's perspective to amplify the micro-vibration signal of the region of interest, and an enlarged video file containing the target object is reconstructed.

Further, the structure represents the structural component of the region of interest in each frame of the image of the video file; the step of constructing the texture component of the region of interest of each frame of the image in the video file includes:

The quality factor adjustable wavelet dictionary is constructed to represent the structural components of the image, and the local discrete cosine transform and discrete sine transform dictionaries are constructed to represent the texture components of the image.

Further, the step of separating the structural component and the texture component in the region of interest includes:

According to the MCA method, the region of interest of each frame image of the video file is separated, and the structure component texture component and the noise component of each frame image are obtained;

constructing a threshold selection algorithm based on a generalized Gaussian density distribution to determine hard thresholds for the texture component and the structure component;

According to the hard thresholds of the texture component and the texture component, the texture component and the texture component are solved by an iterative threshold method.

Further, the step of determining the region of interest of the target object in each frame of image in the video file includes:

Build an SVM recognition model, input a certain frame of image of the video file data into the SVM recognition model, and obtain the key area of the target object as a region of interest; wherein the SVM recognition model is obtained through training set training .

Further, the step of obtaining the training set includes:

Obtain multiple sets of images containing the target object, and use the watershed algorithm to segment candidate regions of each set of images. Each candidate region only contains a set of key objects of the target object. mark;

Using the sliding window method, the image patches in the candidate region marked with key objects are taken as positive examples, and the background image patches without key object markings are taken as negative examples to construct a training set for multi-value classification.

Further, the micro-vibration signal of the region of interest is amplified in combination with the Euler perspective, and the step of reconstructing the amplified video file data including the target object includes:

Combined with Euler's perspective, the vibration change of the target object is characterized according to the brightness change of each pixel in the image structure component of each frame in the video file, and a visual vibration signal is obtained;

Receiving the magnification, multiplying the magnification by the visual vibration signal, and then adding the amplified visual vibration signal to the brightness change of each pixel in the structural component of each frame of image;

Design band-pass filtering according to the theoretical frequency band range of the target object, so that the vibration of the region of interest with a small vibration amplitude in a specific frequency band is amplified;

The enlarged structural and texture components of the region of interest are recombined to generate a vibration-enlarged video.

A second aspect of the present invention provides a visual vibration detection method based on morphological component analysis

The obtained visual vibration signal is obtained through fast Fourier transform to obtain the spectrum corresponding to the visual vibration signal;

The frequency spectrum corresponding to the visual vibration signal is combined with the vibration amplification video obtained by the above method to detect the vibration of the target object.

A third aspect of the present invention provides a visual vibration amplification system based on morphological component analysis, comprising:

an extraction module, which is used to obtain a video file including a target object, and determine the region of interest of the target object in each frame of image in the video file;

a construction module, which is used for constructing a structural component representing the region of interest of each frame of the video file; constructing a texture component representing the region of interest of each frame of the video file; and

A reconstruction module, which is used to separate the structural component and the texture component in the region of interest, amplify the micro-vibration signal of the region of interest by combining the separated structural component and texture component, and combine the Euler perspective to reconstruct the target A zoomed-in video file of the object.

A fourth aspect of the present invention provides an electronic device, including a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are connected in sequence, and the memory is used to store a computer program, and the computer The program includes program instructions, and the processor is configured to invoke the program instructions to perform the above-described method.

A fifth aspect of the present invention provides a readable storage medium storing a computer program, the computer program including program instructions, the program instructions, when executed by a processor, cause the processor to perform the above method.

Compared with the prior art, the embodiments of the present invention at least have the following beneficial effects:

(1) In the present invention, by separating the structural component and the texture component in the region of interest of the target object, for the structural component, the Euler method is used to calculate the vibration spectrum, and the theoretical value of the vibration frequency in the idle state of the engine can be compared. The abnormal frequency of the engine is detected; at the same time, the video of the engine failure state can be enlarged, and the vibration pattern of the engine failure can be observed and analyzed intuitively. Compared with the existing patents, which basically use contact acceleration sensors or non-contact laser sensors, the final result is limited to giving the vibration spectrum of the engine. The difference in nature has obvious advantages.

(2) The present invention focuses on using a fixed dictionary to characterize the image, and does not involve the processing of a large number of experimental samples; and uses morphological component analysis to obtain structural components and texture components. Essentially, each current frame and The structural component difference of the first frame corresponds to normal vibration, and the texture component difference of each current frame and the first frame corresponds to abnormal disturbance. Using the principle of Euler's perspective, this example only enlarges the structural component and reduces the texture component band. At the same time, the visual vibration signal extracted from the structural components is not affected by abnormal disturbances (such as light, uneven surface), and does not involve the phase unwrapping algorithm, which effectively reflects the vibration of the target object. On this basis, do Fourier transform and find its frequency spectrum, which can accurately calculate the frequency of multiple micro-vibration.

(3) The present invention uses morphological component analysis (Morphological Component Analysis, MCA) to separate the structural component, texture component and noise of the region of interest in the video file image, wherein the structural component and the texture component correspond to the phase and Amplitude, the separated structural components are combined with Euler's perspective to extract the visual vibration signal. This method can also remove texture interference, improve the robustness of the algorithm to light changes, and does not involve phase unwrapping; The pulling method calculates the frequency spectrum of the engine, and further combines the amplified structural components and texture components to generate a vibration-amplified video to obtain a visual effect of the vibration of the target object, which can better assist in the manual detection of the target object in the running state of the engine.

Description of drawings

The above and other objects, features and advantages of the present application will become more apparent from the detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present application, constitute a part of the specification, and are used to explain the present application together with the embodiments of the present application, and do not constitute a limitation to the present application. In the drawings, the same reference numbers generally refer to the same components or steps.

Fig. 1 The overall flow chart of the visual vibration detection method for engine faults based on morphological component analysis;

Fig. 2 is a flow chart of engine fault frequency detection and visualization method based on Euler's perspective;

Fig. 3 is the idle state diagram of the engine assembly;

4 illustrates a block diagram of an electronic device according to an embodiment of the present application;

Fig. 5 is the frequency spectrum of the mount system in the idle state of the engine assembly;

6 is a flowchart of a visual vibration amplification method based on morphological component analysis provided by an embodiment of the present invention;

FIG. 7 is a block diagram of a visual vibration amplification system based on morphological component analysis provided by an embodiment of the present invention.

Detailed ways

Hereinafter, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.

Application overview

Taking engine detection as an example, in the process of engine vibration detection, due to the deviation of the position and angle of the internal rubber parts, the natural frequency of the rubber system will change, resulting in abnormal vibration frequency when the engine is running at idle speed. In order to detect the abnormal modal frequency of the engine idling state (essentially the detection of multiple mixed frequencies), in traditional vibration detection, acceleration sensors are often attached to the surface of the engine block and metal parts. The vibration detection method has obvious shortcomings: due to the weight of the acceleration sensor itself, it will affect the natural frequency of the target object, and the pasting process is time-consuming and labor-intensive, and multi-point measurement also requires sensor fusion. For another example, a non-contact laser sensor can only perform single-point measurement. If a full-field measurement is required, multiple laser sensors are required, which is expensive and has the disadvantage of time delay.

The present invention uses morphological component analysis (Morphological Component Analysis, MCA) to separate the structural component, texture component and noise of the region of interest in the video file image, wherein the structural component and the texture component correspond to the phase and amplitude of the image, respectively. The separated structural components are combined with Euler's perspective to extract the visual vibration signal. This method can also remove texture interference, improve the robustness of the algorithm to light changes, and does not involve phase unwrapping. For the structural component, the Euler method is used to calculate the frequency spectrum of the engine, and the amplified structural component and texture component are further combined to generate an amplified vibration video, and the visual effect of the vibration of the target object is obtained, which can better assist manual detection. Engine running state.

Exemplary method

As shown in Figures 1 and 6, this example provides a visual vibration amplification method based on morphological component analysis, which specifically includes the following steps:

S100: Acquire a video file including a target object, and determine a region of interest of the target object in each frame of image in the video file;

Specifically, this example mainly processes video files containing target objects, including mechanical equipment and large buildings, such as cables, cable-stayed bridges, engines, centrifugal pumps, etc., which may generate vibration during actual use. structure. The area of interest refers to the area where the target object needs vibration amplification, and the vibration is generally obvious in this area, such as the engine block and metal parts in the engine.

This example video file can include a working video recording the target object, or it can be a video stream including the target object. Obtain a certain frame of image containing the target object in the video stream or video file, for example, read the image through the camera, and use the camera to read the Api to directly read the frame of the current image stream; according to a certain frame of the video file data The geometric features of the target object are determined to determine the region of interest of the target object in a certain frame of the video file. For the method of determining the region of interest in the image, the region of interest (ROI) of the target object can be extracted through a feature recognition algorithm or a trained image recognition model, but it is not limited thereto.

S200: constructing a structural component representing the region of interest in each frame of the video file; constructing a texture component representing the region of interest in each frame of the video file;

Specifically, the structure component of the video image is represented by constructing a wavelet dictionary with adjustable quality factor, and the texture component of the video image is represented by constructing a discrete sine and cosine dictionary. Since the structural components of the video image show significant directional edges and the texture structure shows periodic content, in this example, the Quality Factor Tunable Wavelet Dictionary (TQWT) is selected to represent the structural components, and the discrete sine and cosine dictionary (DCT) to represent the texture components. .

S300: Separating structural components and texture components in the region of interest, and amplifying the micro-vibration signals of the target object with the separated structural components and texture components in combination with Euler's perspective, and reconstructing an enlarged video file containing the target object.

In some embodiments, the step of separating structural and texture components in the region of interest includes:

According to the MCA method, the region of interest of each frame image of the video file is separated to obtain the structure component, texture component and noise component of each frame image; a threshold selection algorithm based on generalized Gaussian density distribution is constructed to determine the texture component and the structure. The hard threshold of the component; according to the hard threshold of the texture component and the structural component, the structural component and the texture component are solved by an iterative threshold method.

Specifically, an image separation method based on adaptive MCA is first constructed, the purpose of which is to separate the structural components and texture components of each frame of the region of interest (ROI) in the video, so as to obtain the structural components, texture components, and structural components and texture components. The difference between the sum and the original image is the noise component N.

Use TQWT to decompose the image into five layers, select the generalized Gaussian density function of the fifth layer signal, in which the fifth layer signal basically only contains texture components, calculate the morphological parameters α, β corresponding to the texture components of the image, and determine the most The optimal rank ρ; after obtaining the optimal rank ρ, further use the iterative threshold method to solve the structural component and the texture component.

Further, as shown in FIG. 2 , the step of amplifying the micro-vibration signal of the region of interest in combination with the Euler perspective, and reconstructing the enlarged video file data including the target object includes: combining the Euler perspective, according to the video file. The brightness change of each pixel in the image structure component of each frame is used to characterize the vibration change of the target object, and the visual vibration signal is obtained; after receiving the magnification, the magnification is multiplied by the visual vibration signal, and then the value of each pixel in the image structure component of each frame is calculated. The brightness change plus the amplified visual vibration signal; the band-pass filter is designed according to the theoretical frequency band range of the target object, so that the vibration of the region of interest with a small vibration amplitude in a specific frequency band is amplified; the amplified region of interest is amplified. Structural components are recombined with texture components to generate vibrational magnification videos.

Specifically, according to the principle of Euler's perspective, the brightness change of each pixel in the image structure component of each frame of the video is used to represent the vibration change of the target object, the visual vibration signal is obtained, the magnification is manually determined, and the magnification is multiplied by the visual vibration signal. The brightness change of each pixel in the structural components of each frame of the image is added to the amplified visual vibration signal, combined with the texture components obtained by the MCA method in the above steps, and the amplified structural components and texture components are recombined to generate a vibration amplified video. . In this video, the band-pass filter is designed according to the theoretical frequency band range of the target object, so that the vibration with a small vibration amplitude in a specific frequency band can be amplified to a degree that is easy to perceive, and the visualization effect of the vibration of the target object can be obtained. A fault frequency detection and visualization method based on Euler's perspective is constructed. The purpose is to use visual magnification technology to enhance the visualization of vibration signals for areas where the amplitude of micro-vibration changes is small, which is conducive to analyzing the vibration of the target object.

As a variation example, in step S100, the region of interest (ROI) is obtained in the following manner:

Build an SVM recognition model, input a certain frame of image of the video file data into the SVM recognition model, and obtain the key area of the target object as a region of interest; wherein the SVM recognition model is obtained through training set training . The critical area refers to an area where the target object vibrates significantly, such as an engine block and metal parts in an engine.

In order to have a better effect in identifying regions of interest (ROI), preferably, the step of acquiring the training set includes:

Obtain multiple sets of images containing the target object, and use the watershed algorithm to segment candidate regions of each set of images, each candidate region only contains one key region of the target object; for example, the tubular region in the engine, Linear area, block area, etc.

Extract different video files containing the target object, and mark the key areas in them; adopt the sliding window method, take the image blocks marked with key areas as positive examples, and collect background image blocks as negative examples to construct all the multi-value classification. The training set described above; the background image patches here are images that do not contain key regions.

The following takes the acquisition of the region of interest of the engine as an example for description.

S101: Use the watershed algorithm to detect the image containing the engine as the target object, and obtain the boundaries between different regions, so as to obtain candidate regions divided by different boundaries, and each candidate region contains only one type of component; the candidate region here can be Refers to the tubular area, linear area, block area in the engine.

S102: According to the method of step 101, extract different engine videos, and mark key objects in different candidate regions in the multi-frame images in each group of engine videos to enrich the training samples, for example, mark the tubular region as "metal piece" , and mark the "block area" as "engine block"; engine videos of different models can be engine videos taken at different angles and under different lighting conditions to enrich the content of the training set.

S103: In a sliding window manner, image blocks marked with key regions are used as positive examples, and background image blocks are collected as negative examples to construct the training set for multi-value classification. In the SVM recognition model testing process, the residual network input image block, after each level of convolution, excitation nonlinear transformation and pooling operations, and finally input to the two-layer fully connected layer, the output of the candidate category score, and The largest score is used as the predicted category of the component; in this example, the engine block and metal parts are selected as the region of interest (ROI, region of interest) for the extraction of the engine visual vibration signal.

In some embodiments, the step S200 may include the following steps:

S201: Use a quality factor adjustable wavelet dictionary to represent the structural component of the video image, and use a discrete sine and cosine dictionary to represent the texture component of the video image;

S202: Use the MCA method to separate each frame of image in the video to obtain the structural component Y ₁ including the image (that is, the intercepted region of interest image), the texture component Y ₂ and the difference between the sum of the two components and the original image, That is, the noise component N;

δx=Y ₁ +Y ₂ +N (1),

The described threshold selection algorithm based on generalized Gaussian density distribution includes the following steps:

Step S311: Solve y ₁ and y ₂ by iterative threshold method:

According to the MCA algorithm, the form of the objective function is determined as:

Among them, Φ ₁ represents the odd function of TQWT; Φ ₂ represents the odd function of DCT; T represents rank transfer; y ₁ represents the coefficient of the TQWT domain; y ₂ represents the corresponding coefficient of the DCT domain;

Solve for y ₁ and y ₂ using the iterative threshold method:

表示y1的估计值；Y_n表示噪声；

表示Y₂在前一次迭代的估计误差。in,

represents the estimated value of y1; Y _n represents noise;

represents the estimated error of _Y2 at the previous iteration.

和n₁分别表示

和噪声的TQWT系数。in,

and n ₁ respectively represent

and TQWT coefficients for noise.

表示y₂的估计误差，和y₂一样，为谐振分量(结构分量)；因此，当

用TQWT系数表示时，

的秩要低于r₁的秩，即

Further, in order to accurately estimate y ₁ , a method needs to be devised to accurately remove

represents the estimation error of y ₂ , which, like y ₂ , is the resonance component (structural component); therefore, when

When expressed by the TQWT coefficient,

The rank of is lower than the rank of r ₁ , that is

的最优秩，其目的是为了准确估算式(3)中的

在这里，我们把y₁和广义高斯密度函数联系起来，广义高斯密度函数：S312: OK

The optimal rank of , whose purpose is to accurately estimate the equation (3)

Here, we relate y ₁ to the generalized Gaussian density function, the generalized Gaussian density function:

where α, β represent generalized morphological parameters;

使其最接近y₁，可以写成目标函数的形式：The morphological parameter α corresponding to the optimal rank ρ should be able to give a

To make it closest to y ₁ , it can be written in the form of the objective function:

Among them, α _ρ represents the optimal morphological parameter in the generalized morphological parameter α; β _ρ represents the optimal morphological parameter in the generalized morphological parameter β.

Here, TQWT is used to decompose the image into 5 layers, and the generalized Gaussian density function of the fifth layer signal (basically only contains the shock component) is selected, the corresponding morphological parameters α and β are calculated, and the optimal value is determined according to formula (17). rank ρ.

Then determine the optimal rank ρ; after obtaining the optimal rank ρ, further use the iterative threshold method to solve the impact component (texture component) and the resonance component (structural component).

Step S320: The described adaptive MCA-based image separation method includes the following steps:

S321: Solving for structural components

可以通过最优化方法求解，如下式：

It can be solved by the optimization method, as follows:

表示

的最优解；

表示

表示Y₂在前一次迭代的估计误差；||x||_F是x的Frobenious范数；通过使用奇异值分解(SVD：singular-value decomposition)，r₁可以分解成r₁＝U∑V^T，其中，∑＝diag(σ₁，….，σ_n)；V^T表示；U表示；in,

express

the optimal solution;

express

represents the estimation error of Y ₂ in the previous iteration; ||x|| _F is the Frobenious norm of x; by using singular value decomposition (SVD: singular-value decomposition), r ₁ can be decomposed into r ₁ =U∑V ^T , where ∑=diag(σ ₁ ,....,σ _n ); V ^T represents; U represents;

According to the Eckart-Young-Mirsky theorem, equation (9) can be decomposed into:

where η _ρ (Σ)=diag(σ ₁ ,...,σ _n ,0,...,0)

如下：rough estimate of y ₁

as follows:

表示y1的最优解；

表示y2的最优解；最终的

可以通过如下所示的硬阈值决定：in

represents the optimal solution of y1;

represents the optimal solution of y2; the final

This can be determined by a hard threshold as shown below:

绝对值的中位数；H表示决定硬阈值的算法；where λ ₁ =MAD/0.6745, MAD represents

median of absolute values; H represents the algorithm for determining the hard threshold;

S322: Solve for texture components

According to formula (3)

Similar to r ₁ , r ₂ can be defined as

和n₂分别为

和噪声的DCT系数；此处的硬阈值λ₂正比于

误差R_T定义如下：

and _n2 are respectively

and the DCT coefficients of noise; here the hard threshold _λ2 is proportional to

The error R _T is defined as follows:

From this, we can estimate

可以通过式(10)计算得到；此处的硬阈值λ₂可以设置为，in

It can be calculated by formula (10); the hard threshold λ ₂ here can be set as,

可以通过如下所示的硬阈值决定：final

This can be determined by a hard threshold as shown below:

In some embodiments, taking the engine as the target object as an example, the step of amplifying the micro-vibration signal of the region of interest in combination with the Euler perspective, the step of reconstructing the enlarged video file data including the target object includes:

Step S331: According to the principle of Euler's perspective, use the brightness change of each pixel in the image structure component of each frame of the video to represent the vibration change of the target object, manually determine the magnification, and multiply the magnification by the brightness change of each pixel. The brightness change of each pixel in the image structural component is added to the amplified visual vibration signal, combined with the texture component obtained by the MCA method in the above steps, and the amplified structural component and texture component are recombined to generate a vibration-amplified video. In this video, the band-pass filter is designed according to the theoretical frequency band range of the target object, so that the vibration with a small vibration amplitude in a specific frequency band can be amplified to a degree that is easy to perceive, and the visualization effect of the vibration of the target object can be obtained. The specific formula as follows:

δ=∑ _k h(α·(Y _1k -Y ₁₁ ))+Y ₂ (19)

Among them, α represents the magnification factor, h represents the bandpass filter, k represents the current frame, and δ represents the amplified video; Y ₁₁ represents the structural component of the first frame; Y ₂ represents the texture component; Y _1k represents the structural component of the kth frame ;

Step S332: For the obtained visual vibration signal, according to the FFT transformation, obtain the frequency spectrum corresponding to the visual vibration signal, and detect the frequency corresponding to the target object. The specific formula is as follows:

δy(t)=∑ _k ∑ _i h(Y _1k (i)-Y ₁₁ (i)) (20)

Among them, δy(t) represents the visual vibration signal, and i represents the pixel in each frame of video.

It should be noted that during the research process, the applicant discovered: the existing paper [Wadhwa, M. Rubinstein, F. Durand, and W. Freeman, "Phase-based video motion processing," ACMTrans.Graph., vol.32, no.4, Jul.2013, Art.no.80.] uses a phase-based motion amplification method, which is susceptible to phase wrapping and unwrapping algorithms, resulting in inaccurate detection of mixed frequencies. The existing paper [Learning-based Video Motion Magnication, 2018] uses deep learning to separate shape components and texture components, but its deep learning network is only used to achieve motion amplification, and the deep learning method relies on a large number of samples. Currently, Not suitable for actual vibration measurement. The existing paper [ANew Approach to the Phase-Based VideoMotion Magnification for Measuring Microdisplacements, 2019] uses the Hilbert transform to achieve the decomposition of the complex controllable pyramid, and its essence is still phase-based motion amplification, which cannot solve the effect of phase winding.

Different from the above methods, this example focuses on using a fixed dictionary and does not involve a large number of experimental samples. This example uses morphological component analysis to obtain structural components and texture components. In essence, the difference between the structural components of each current frame and the first frame corresponds to normal vibration, and the difference between the texture components of each current frame and the first frame corresponds to normal vibration. Corresponding to the abnormal disturbance, using the principle of Euler's perspective, this example only amplifies the structural components, reducing the interference caused by the texture components. At the same time, the visual vibration signal extracted from the structural components is not affected by abnormal disturbances (such as light, uneven surface), and does not involve the phase unwrapping algorithm, which effectively reflects the vibration of the target object. Fourier transform, find its frequency spectrum, can accurately calculate the frequency of multiple micro-vibration.

As shown in Figures 1 and 2, this example also provides a visual vibration detection method based on morphological component analysis. The obtained visual vibration signal is transformed by FFT to obtain a frequency spectrum corresponding to the visual vibration signal; the visual vibration signal corresponding to The frequency spectrum of the target object is combined with the vibration amplification video obtained in the above embodiment to detect the vibration of the target object; the engine failure is analyzed by engineers.

Specifically, the brightness change of each pixel in the image structure component of each frame of the video is used to represent the vibration change of the target object, and the visual vibration signal is obtained. On this basis, FFT (Fourier) transformation is used to obtain the spectrum corresponding to the visual vibration signal. , to detect the frequency corresponding to the target object; in this video, a band-pass filter is designed according to the theoretical frequency band range of the target object, so that the vibration with a small vibration amplitude in a specific frequency band can be amplified to a degree that is easy to perceive, and the target object can be obtained. The visualization of the vibration situation. Combine engine failure visualization video and engine failure spectrum to analyze engine failures. The engine in this example may be a diesel engine, a gasoline engine, an electric vehicle motor, or the like.

Exemplary device

As shown in Figure 7, a visual vibration amplification system based on morphological component analysis includes:

Extraction module 20, which is used to obtain the video file data including the target object, and selects the region of interest of the target object according to the geometric feature of a certain frame of the video file data;

A construction module 30, which is used for constructing a structural component representing the region of interest of each frame of the video file; constructing a texture component representing the region of interest of each frame of the video file; and

The reconstruction module 40 is used to separate the structural component and the texture component in the region of interest, and the separated structural component and the texture component are combined with the Euler perspective to amplify the micro-vibration signal of the target object, and reconstruct the enlarged video file containing the target object .

Exemplary Electronics

Hereinafter, an electronic device according to an embodiment of the present application will be described with reference to FIG. 1 . The electronic device can be the mobile device itself, or a stand-alone device independent of it, which can communicate with the mobile devices to receive collected input signals from them and transmit to them selected target decision-making behaviors.

FIG. 4 illustrates a block diagram of an electronic device according to an embodiment of the present application.

As shown in FIG. 4 , the electronic device 10 includes one or more processors 11 and a memory 12 .

Processor 11 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in electronic device 10 to perform desired functions.

Memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and/or cache memory, or the like. The non-volatile memory may include, for example, read only memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the decision-making behavior decision method and/or the above-described various embodiments of the present application Other desired features.

In one example, the electronic device 10 may also include an input device 13 and an output device 14 interconnected by a bus system and/or other form of connection mechanism (not shown). For example, the input device 13 may include various types such as on-board diagnostic system (OBD), unified diagnostic service (UDS), inertial measurement unit (IMU), camera, lidar, millimeter-wave radar, ultrasonic radar, vehicle-to-vehicle communication (V2X), etc. equipment. The input device 13 may also include, for example, a keyboard, a mouse, and the like. The output device 14 may include, for example, displays, speakers, printers, and communication networks and their connected remote output devices, among others.

Of course, for simplicity, only some of the components in the electronic device 10 related to the present application are shown in FIG. 4 , and components such as buses, input/output interfaces, and the like are omitted. Besides, the electronic device 10 may also include any other suitable components according to the specific application.

Exemplary computer program product and computer readable storage medium

In addition to the methods and apparatuses described above, embodiments of the present application may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the "exemplary methods" described above in this specification The steps in the decision-making method according to various embodiments of the present application described in the section.

The computer program product can write program codes for performing the operations of the embodiments of the present application in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc. , also includes conventional procedural programming languages, such as "C" language or similar programming languages. The program code may execute entirely on the user computing device, partly on the user device, as a stand-alone software package, partly on the user computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on.

In addition, embodiments of the present application may also be computer-readable storage media having computer program instructions stored thereon, the computer program instructions, when executed by a processor, cause the processor to perform the above-mentioned "Example Method" section of this specification The steps in the decision-making method according to various embodiments of the present application described in .

The computer-readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses or devices, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, advantages, effects, etc. mentioned in the present application are only examples rather than limitations, and these advantages, advantages, effects, etc., are not considered to be Required for each embodiment of this application. In addition, the specific details disclosed above are only for the purpose of example and easy understanding, rather than limiting, and the above-mentioned details do not limit the application to be implemented by using the above-mentioned specific details.

The block diagrams of devices, apparatus, apparatuses, and systems referred to in this application are merely illustrative examples and are not intended to require or imply that the connections, arrangements, or configurations must be in the manner shown in the block diagrams. As those skilled in the art will appreciate, these means, apparatuses, apparatuses, systems may be connected, arranged, configured in any manner. Words such as "including", "including", "having" and the like are open-ended words meaning "including but not limited to" and are used interchangeably therewith. As used herein, the words "or" and "and" refer to and are used interchangeably with the word "and/or" unless the context clearly dictates otherwise. As used herein, the word "such as" refers to and is used interchangeably with the phrase "such as but not limited to".

It should also be pointed out that in the apparatus, equipment and method of the present application, each component or each step can be decomposed and/or recombined. These disaggregations and/or recombinations should be considered as equivalents of the present application.

The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Therefore, this application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for the purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims (7)

1. A visual vibration amplification method based on morphological component analysis is characterized by comprising the following steps:

s100: acquiring a video file comprising a target object, and determining an interested area of the target object in each frame of image in the video file;

s200: constructing a structural component for representing the interested region in each frame of image of the video file; constructing texture components for representing the interested areas of each frame of image in the video file; constructing a quality factor adjustable wavelet dictionary for representing structural components of the image, and constructing a local discrete cosine transform and discrete sine transform dictionary for representing texture components of the image;

s300: separating the structure component, the texture component and the noise component in the region of interest, amplifying the micro-vibration signal of the region of interest by combining the separated structure component and texture component with an Euler visual angle, and reconstructing an amplified video file containing a target object; the structural component difference of each current frame and the first frame corresponds to normal vibration, the texture component difference of each current frame and the first frame corresponds to abnormal disturbance, only the structural component is amplified by utilizing an Euler visual angle principle, and the interference brought by the texture component is reduced;

the step of separating the structure component and the texture component in the region of interest comprises:

separating the interested region of each frame of image of the video file by using an MCA method to obtain a structure component texture component and a noise component of each frame of image; the difference between the sum of the structural component and the texture component and the original image is a noise component;

constructing a threshold selection algorithm based on generalized Gaussian density distribution, and determining hard thresholds of the texture component and the structure component;

solving the structure component and the texture component by an iterative threshold method according to the texture component and the hard threshold of the structure component;

the step of reconstructing the enlarged video file data containing the target object by enlarging the micro-vibration signal of the region of interest in combination with the Euler view angle comprises the following steps:

representing the vibration change of a target object according to the brightness change of each pixel in each frame of image structure component in the video file to obtain a visual vibration signal;

receiving an amplification factor, multiplying the amplification factor by a visual vibration signal, and adding the brightness change of each pixel in each frame of image structure component to the amplified visual vibration signal;

designing band-pass filtering according to the theoretical frequency band range of the target object, so that the vibration amplitude of the region of interest in a specific frequency band is amplified;

and recombining the structure component and the texture component after the region of interest is enlarged to generate the vibration enlarged video.

2. The visual vibration amplifying method based on morphological component analysis as claimed in claim 1, wherein the step of determining the interest region of the target object in each frame of image in the video file comprises:

constructing an SVM recognition model, inputting a certain frame of image of the video file data into the SVM recognition model, and acquiring a key area of the target object as an interesting area; the SVM recognition model is obtained by training through a training set.

3. The visual vibration amplifying method based on morphological component analysis as claimed in claim 2, wherein the step of obtaining the training set comprises:

acquiring a plurality of groups of images containing target objects, segmenting candidate regions of each group of images by adopting a watershed algorithm, wherein each candidate region only contains one group of key objects of the target objects, and marking the candidate regions;

and adopting a sliding window mode, taking the candidate area image block with the key object mark as a positive example, and taking the background image block with the irrelevant object mark as a negative example, and constructing a training set of multi-value classification.

4. A visual vibration detection method based on morphological component analysis is characterized by comprising the following steps:

obtaining a frequency spectrum corresponding to the visual vibration signal through fast Fourier transform of the obtained visual vibration signal;

and detecting the vibration condition of the target object by combining the corresponding frequency spectrum of the visual vibration signal with the vibration amplification video obtained by any one of claims 1 to 3.

5. A visual vibration amplification system based on morphological component analysis, comprising:

the extraction module is used for acquiring a video file comprising a target object and determining an interest area of the target object in each frame of image in the video file;

the construction module is used for constructing structural components for representing the interested region of each frame of image of the video file; texture components representing the interested region of each frame of image of the video file are constructed, wherein a quality factor adjustable wavelet dictionary is constructed and used for representing the structural components of the image, and a local discrete cosine transform and discrete sine transform dictionary is constructed and used for representing the texture components of the image; and

the reconstruction module is used for separating the structure component and the texture component in the region of interest, amplifying the micro-vibration signal of the region of interest by combining the separated structure component and the texture component with an Euler visual angle, and reconstructing an amplified video file containing a target object;

the structural component difference of each current frame and the first frame corresponds to normal vibration, the texture component difference of each current frame and the first frame corresponds to abnormal disturbance, only the structural component is amplified by utilizing an Euler visual angle principle, and the interference brought by the texture component is reduced;

the step of separating the structure component and the texture component in the region of interest comprises:

separating the interested region of each frame of image of the video file by using an MCA method to obtain a structure component texture component and a noise component of each frame of image; the difference between the sum of the structural component and the texture component and the original image is a noise component;

constructing a threshold selection algorithm based on generalized Gaussian density distribution, and determining hard thresholds of the texture component and the structure component;

solving the structure component and the texture component by an iterative threshold method according to the texture component and the hard threshold of the structure component;

the step of reconstructing the enlarged video file data containing the target object by enlarging the micro-vibration signal of the region of interest in combination with the Euler view angle comprises the following steps:

representing the vibration change of a target object according to the brightness change of each pixel in each frame of image structure component in the video file to obtain a visual vibration signal;

receiving an amplification factor, multiplying the amplification factor by a visual vibration signal, and adding the brightness change of each pixel in each frame of image structure component to the amplified visual vibration signal;

designing band-pass filtering according to the theoretical frequency band range of the target object, so that the vibration amplitude of the region of interest in a specific frequency band is amplified;

and recombining the structure component and the texture component after the region of interest is enlarged to generate the vibration enlarged video.

6. An electronic device comprising a processor, an input device, an output device, and a memory, the processor, the input device, the output device, and the memory being connected in series, the memory being configured to store a computer program comprising program instructions, the processor being configured to invoke the program instructions to perform the method of any of claims 1-4.

7. A readable storage medium, characterized in that the storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the method according to any one of claims 1-4.

Images