TWI634344B - 3D trajectory detection system - Google Patents

Abstract

A three-dimensional trajectory detection system includes an image capturing device, a radar device, and a computing module, wherein the image capturing device is configured to capture a dynamic image and calculate a pixel displacement of one of the moving object images according to the dynamic image. a quantity and a pixel displacement amount in the Y direction, and the radar device transmits a wireless signal to a moving object, and receives a reflected signal from the moving object, and the radar device calculates the movement according to the Doppler phase shift amount of the reflected signal One of the objects is displaced in the Z-axis direction, and the calculation module establishes a three-dimensional trajectory of the moving object according to the X-direction pixel displacement amount of the moving object, the Y-direction pixel displacement amount, and the Z-axis direction displacement of the moving object.

Description

3D trajectory detection system

The present invention relates to a three-dimensional trajectory detection system, and more particularly to a three-dimensional trajectory detection system that combines computer vision and Doppler radar.

With the rapid development of mobile devices, its functions and applications are more and more extensive, and the application potential of Augmented Reality (AR) is extremely high and has become the focus of current mobile devices. Hand gesture recognition (HGR) is a very important interactive technology for augmented reality. The gesture recognition allows the user to observe the direct interaction with the virtual image on the display. The current gesture recognition The mainstream technology is identified by a 3D camera, such as Microsoft Kinect, Intel RealSense, and Lead Motion. Since the 3D camera takes an image with an RGB camera and then uses an infrared camera to obtain the depth information of the image, Gesture recognition allows the mobile device to be equipped with an infrared camera to provide gesture recognition. This will not only make the mobile device more expensive, but also cause the battery to drain too fast. In addition, The depth detection capability of an infrared camera with a well-lit outdoor environment will be greatly reduced, and these problems all cause difficulties in gesture recognition of mobile devices.

The main purpose of the present invention is to capture the motion image by the image capturing device and calculate the X-direction pixel displacement amount and the Y-direction pixel displacement amount of the moving object image, and detect the displacement of the same moving object in the Z-axis direction by the radar device, and finally calculate The module calculates the three-dimensional trajectory of the moving object based on the pixel displacement amount in the X direction, the pixel displacement amount in the Y direction, and the displacement in the Z axis direction.

A three-dimensional trajectory detection system includes an image capturing device, a radar device, and a computing module. The image capturing device is configured to capture a moving image having a moving object image. The capturing device calculates an X-direction pixel displacement amount and a Y-direction pixel displacement amount according to the moving object image, and the radar device transmits a wireless signal to a moving object, and receives a reflection signal by the moving object. And the radar device obtains a Z-axis displacement of the moving object according to the Doppler phase shift amount of the reflected signal, the computing module is coupled to the image capturing device and the radar device, and the computing module is configured according to the movement The X-direction pixel displacement amount of the object image, the Y-direction pixel displacement amount, and the Z-axis direction displacement of the moving object calculate a three-dimensional trajectory of the moving object.

The three-dimensional trajectory detecting system of the present invention acquires the X-direction pixel displacement amount and the Y-direction pixel displacement amount of the moving object image by the image capturing device, and acquires the Z-axis of the moving object by using the radar device The directional displacement can be obtained by the calculation module according to the X-direction pixel displacement amount, the Y-direction pixel displacement amount, and the Z-axis direction displacement to obtain the three-dimensional trajectory of the moving object. Since the radar device is a passive radar, there is no dedicated RF transmission signal source, but the detection of the Z-axis displacement of the moving object by using the existing wireless signal in the environment can effectively avoid the wireless interference of the mobile device. Communication, and the power consumption requirements of the mobile device are also low. In addition, the radar device using electromagnetic waves for detection is not interfered by outdoor light, so that the three-dimensional trajectory detection system of the present invention can solve the current mobile device. The difficulty of gesture recognition.

1 is a functional block diagram of a three-dimensional trajectory detection system 100, which includes an image capture device 110, a radar device 120, and a computing module. The image capturing device 110 and the radar device 120 are coupled to the computing module 130. In the embodiment, the image capturing device 110, the radar device 120, and the computing module 130 are independent electronic components. In other embodiments, the image capturing device 110, the radar device 120, and the computing module 130 are integrated into a mobile device.

Referring to FIGS. 2 and 3, the image capturing device 110 has an RGB camera 111 and an image processing module 112. The RGB camera 111 is configured to capture a moving image P, and the moving image P includes a moving object O. For a moving object image O _P and its background, please refer to FIG. 3 , the moving image P has the moving object image O _P , and the moving object image O _P moves in the moving image P with time, in In this embodiment, the moving object image O _P is the right hand of a user. In other embodiments, the moving object image O _P may be other moving objects. Referring to FIG. 2, the image processing module 112 of the image capturing device 110 receives the moving image P. Since the RGB camera 111 can only capture two-dimensional images, the image processing module 112 can only The motion image P calculates the X-direction pixel displacement amount of the moving object image O _P And a Y-direction pixel displacement . In this embodiment, the image processing module 112 of the image capturing device 110 calculates the X-direction pixel displacement of the moving object image O _P by a background subtraction method. And the amount of pixel displacement in the Y direction The image processing module 112 of the image capturing device 110 creates a background model according to the dynamic image P, by calculating a difference between a plurality of pixels of the moving image P and a plurality of pixels of the background model. The X-direction pixel displacement of the moving object image O _P And the amount of pixel displacement in the Y direction Preferably, the background model is a Gaussian mixture model, but the invention is not limited thereto.

Referring to FIGS. 1 and 4 again, the radar device 120 transmits a wireless signal S _T to the moving object O, and receives a reflected signal S _R from the moving object O, since the reflected signal S _R contains the moving object O The displacement of the Doppler phase is caused by the displacement, so the radar device 120 obtains a Z-axis displacement of the moving object O according to the reflected signal S _R . The radar device 120 can be an active radar or a passive radar. Since the three-dimensional trajectory detection system 100 of the present invention is to be disposed in a mobile device, the present invention is preferably The radar device 120 is a passive radar to reduce the overall power consumption of the mobile device.

Referring to FIG. 5, in the embodiment, the radar device 120 is a passive injection-locked radar, wherein the radar device 120 has a coupler 121, an antenna 122, and an injection-locked oscillator 123. An amplifier 124, a demodulation circuit 125, a signal processing unit 126, and a band pass filter 127.

Referring to FIG. 5, the action of the radar device 120 is: the coupler 121 receives the wireless signal S _T from the environment, and the wireless signal S _T can send a transmission signal or a wireless base station to the mobile device (Wireless The access point sends a transmission signal. In this embodiment, the wireless signal S _{T is} a Wi-Fi signal sent by the wireless base station. The coupler 121 couples the wireless signal S _T to the antenna 122 and the injection-locked oscillator 123. In this embodiment, the coupler 121 is a branch-line coupler, and the coupler 121 After the wireless signal S _{T is} coupled to the injection-locked oscillator 123, the injection-locked oscillator 123 can be placed in an injection-locked state. The coupler 121 couples the wireless signal S _T to the band pass filter 127 and transmits the signal to the antenna 122. The antenna 122 transmits the wireless signal S _T to the moving object O, and the antenna 122 is moved by the mobile device. The object O receives the reflected signal S _R , wherein the reflected signal S _R includes a Doppler phase shift caused by the displacement of the moving object O, and the reflected signal S _R received by the antenna 122 is transmitted to the coupler 121. The coupler 121 couples the reflected signal S _R to the amplifier 124. The demodulation circuit 125 is coupled to the injection-locked oscillator 123 and the amplifier 124 to lock one of the outputs of the injection-locked oscillator 123 (lock-in The signal S _L and one of the amplifiers 124 output RF amplification signal S _O perform quadrature phase demodulation.

Referring to FIG. 5, in the embodiment, the demodulation circuit 125 is an orthogonal (I/Q) demodulation circuit, wherein the demodulation circuit 125 has a 90 degree phase difference power divider 125a and a power divider. 125b, a first mixer 125c, a second mixer 125d, a first low pass filter 125e and a second low pass filter 125f. The 90-degree phase difference power divider 125a is coupled to the injection-locked oscillator 123 to receive the phase-locked signal S _L . The 0-degree output of the 90-degree phase difference power divider 125a outputs an in-phase local oscillation signal S _{LO_I} . One of the 90 degree output terminals of the 90 degree phase difference power splitter 125a outputs a quadrature local oscillation signal S _{LO_Q} . The power splitter 125b is coupled to the amplifier 124 to receive the RF amplified signal S _O , and the power splitter 125b divides the RF amplified signal S _O into two paths. The first mixer 125c receives the RF amplification signal S _{O of the} in-phase local oscillation signal S _{LO_I} and one of the paths to perform mixing to obtain a first mixed signal M1, and the second mixer 125d receives the The orthogonal local oscillation signal S _{LO_Q} and the RF amplification signal S _{O of} another path are mixed to obtain a second mixed signal M2, and the first low pass filter 125e receives the first mixer 125c. The first mixed signal M1 is filtered to output an in-phase signal I, and the second low-pass filter 125f receives the second mixed signal M2 of the second mixer 125d for filtering and outputting. a quadrature signal Q, the signal processing unit 126 is coupled to the demodulation circuit 125 to receive the in-phase signal I and the quadrature signal Q, and the signal processing unit 126 is configured according to the in-phase signal I and the orthogonal signal Q calculates the Z-axis displacement of the moving object O In this embodiment, the signal processing unit 126 obtains the Doppler phase shift amount of the reflected signal S _R by arctangent calculation, and further obtains the Z-axis displacement of the moving object O. .

Referring to FIG. 1 , the computing module 130 is coupled to the image capturing device 110 and the radar device 120 , and the computing module 130 needs to perform a calibration process. Please refer to FIGS. 6 and 7 . The first angle of the moving object O along one of the XZ planes Doing a linear motion and a second angle of the moving object O along a YZ plane Doing a linear motion, and the first angle With the second angle The X-direction pixel displacement amount of the moving object image O _P is calculated according to the moving object O under the above two kinds of linear motions for the known fixed angle. , the pixel displacement of the Y direction And the Z-axis displacement of the moving object O Calculating the displacement of one of the moving objects O in the X-axis direction The X-direction pixel displacement amount of the moving object image O _P One of the first conversion ratios And the displacement of one of the moving objects O in the Y-axis direction The amount of pixel displacement in the Y direction with the moving object image O _P A schematic view of a second one of the ratio of m _y conversion, the first calculation equation in terms of the ratio of the second m _x m _y ratio in terms of:

The three-dimensional trajectory detection system 100 of the present invention can obtain the X-direction pixel displacement amount Δ P _x and the Y-direction pixel displacement Δ P _{y of} the moving object image O _P by the image capturing device 110. The first conversion ratio m _x and the second conversion ratio m _y obtained by the calculation module 130 by the calibration program, and the X-direction pixel displacement of the moving object image O _P under any motion track of the moving object O The quantity Δ P _{x is} converted into the X-axis direction displacement Δ x of the moving object O, and the Y-direction pixel displacement amount Δ P _{y of} the moving object image O _P is converted into the Y-axis direction displacement of the moving object O. y, and by the radar apparatus 120 obtains the Z-axis direction of displacement of the moving object O △ z, to create the three-dimensional trajectory of the moving object O T. Since the radar device 120 is a passive radar, there is no dedicated RF transmitting signal source, but the wireless signal of the environment is used to detect the Z-axis displacement Δ z of the moving object O, which can effectively avoid interference action. The wireless communication of the device has lower requirements on the power consumption of the mobile device. In addition, the radar device 120 using electromagnetic waves for detection is not interfered by outdoor light, so that the three-dimensional trajectory detection system 100 of the present invention does It can solve the difficulty of gesture recognition of current mobile devices.

The image capturing device 110 is a smart phone, and the radar device 120 is a smart phone. The image capturing device 110 is a smart phone. A passive radar (not shown), the antenna 122 of the radar device 120 is a flat panel antenna, the computing module 130 is a notebook computer, and the image capturing device 110 and the radar device 120 are coupled to the antenna The calculation module 130. Referring to FIG. 8 , the left hand of the user (not shown) takes the image capturing device 110 , and the right hand moves from left to right in a path of an upward convex curve and gradually moves away from the image capturing device. 110. It can be seen from the established three-dimensional trajectory that it is indeed similar to the trajectory of the user's right hand. Referring to FIG. 9, the user's left hand (not shown) takes the image capturing device 110, and the right hand first moves slightly upward from left to right and gradually moves away from the image capturing device 110, and then makes a turn. Moving from right to left and down and gradually approaching the image capture device 110, it can be seen from the established three-dimensional trajectory that it is indeed similar to the trajectory of the user's right hand. It can be proved that the present invention can establish a three-dimensional trajectory according to the dynamic image captured by the image capturing device 110 and the Z-axis displacement detected by the radar device 120.

The scope of the present invention is defined by the scope of the appended claims, and any changes and modifications made by those skilled in the art without departing from the spirit and scope of the invention are within the scope of the present invention. .

100‧‧‧3D Trajectory Detection System

110‧‧‧Image capture device

111‧‧‧RGB camera

112‧‧‧Image Processing Module

120‧‧‧Radar device

121‧‧‧ Coupler

122‧‧‧Antenna

123‧‧‧Injected Locked Oscillator

124‧‧‧Amplifier

125‧‧‧Demodulation circuit

125a‧‧90 degree differential power splitter

125b‧‧‧Power splitter

125c‧‧‧First Mixer

125d‧‧‧second mixer

125e‧‧‧First low pass filter

125f‧‧‧second low pass filter

126‧‧‧Signal Processing Unit

127‧‧‧ bandpass filter

130‧‧‧Computation Module

P‧‧‧ motion pictures

O _P ‧‧‧moving object images

‧‧‧X-direction pixel displacement

‧‧‧Pixel displacement in the Y direction

‧‧‧First conversion ratio

‧‧‧Second conversion ratio

‧‧‧X-axis displacement

‧‧‧Y-axis displacement

‧‧‧Z-axis displacement

T‧‧‧3D trajectory

O‧‧‧moving objects

S _T ‧‧‧Wireless signal

S _R ‧‧‧Reflective signal

S _L ‧‧‧Lock signal

S _O ‧‧‧RF amplification signal

S _{LO_I ‧‧‧In-} phase local oscillation signal

S _{LO_Q ‧‧‧Orthogonal} local oscillation signal

M1‧‧‧ first mixed wave signal

M2‧‧‧Secondary mixed signal

I‧‧‧In-phase signal

Q‧‧‧Orthogonal signal

Figure 1 is a functional block diagram of a three-dimensional track detection system in accordance with an embodiment of the present invention. Figure 2 is a functional block diagram of an image capture device in accordance with an embodiment of the present invention. Figure 3: A schematic diagram of a mobile device capturing a motion picture in accordance with an embodiment of the present invention. Figure 4: Schematic diagram of a radar device detecting displacement of a moving object in accordance with an embodiment of the present invention. Figure 5: Circuit diagram of the radar device in accordance with an embodiment of the present invention. Figure 6: Schematic diagram of linear motion of the moving object along a fixed angle on the XZ plane in accordance with an embodiment of the present invention. Figure 7: Schematic diagram of linear motion of the moving object along a fixed angle on the YZ plane in accordance with an embodiment of the present invention. Figure 8: A schematic diagram of a three-dimensional trajectory system for establishing a three-dimensional trajectory in accordance with an embodiment of the present invention. Figure 9 is a schematic diagram of the three-dimensional trajectory detection system establishing a three-dimensional trajectory according to an embodiment of the present invention.

Claims (11)

A three-dimensional trajectory detection system includes an image capturing device for capturing a moving image, wherein the moving image has a moving object image, and the image capturing device calculates an X-direction pixel displacement of the moving object image And a Y-direction pixel displacement amount; a radar device transmits a wireless signal to a moving object, and receives a reflected signal from the moving object, and the radar device obtains the movement according to a Doppler phase shift amount of the reflected signal One of the objects is displaced in the Z-axis direction; and a computing module is coupled to the image capturing device and the radar device, and the computing module is configured according to the X-direction pixel displacement amount and the Y-direction pixel displacement amount of the moving object image And calculating, in the Z-axis direction of the moving object, a three-dimensional trajectory of the moving object, wherein the computing module has a calibration program for linearly moving the moving object along a first angle on an XZ plane And the moving object moves linearly along a second angle on a YZ plane, and the first angle and the second angle are known to be fixed angles, wherein The calculation module calculates an X-axis direction displacement and the movement of the moving object according to the X-direction pixel displacement amount of the moving object image, the Y-direction pixel displacement amount, and the Z-axis direction displacement of the moving object a first conversion ratio of the X-direction pixel displacement amount of the object image, and a second conversion ratio of the Y-axis direction displacement of the moving object and the Y-direction pixel displacement amount of the moving object image. The three-dimensional trajectory detection system of claim 1, wherein the image capturing device calculates the X-direction pixel displacement amount and the Y-direction pixel of the moving object image by a background subtraction method. The amount of displacement. For example, the three-dimensional trajectory detection system described in claim 2, wherein the image capture The device generates a background model according to the motion image, and calculates the X-direction pixel displacement amount and the Y-direction pixel displacement amount of the moving object image by using a plurality of pixels of the motion image and a plurality of pixels of the background model. The three-dimensional trajectory detection system of claim 1, wherein the radar device is a passive radar. The three-dimensional trajectory detection system of claim 4, wherein the radar device has a coupler, an antenna, an injection-locked oscillator, an amplifier, and a demodulation circuit. Receiving the wireless signal, and the coupler couples the wireless signal to the antenna and the injection-locked oscillator, so that the injection-locked oscillator is in an injection-locked state, and the antenna transmits the wireless signal To the moving object, and the antenna receives a reflected signal from the moving object, the reflected signal is transmitted to the coupler, the coupler couples the reflected signal to the amplifier, and the demodulation circuit is coupled to the injection-locked oscillator and The amplifier performs quadrature phase demodulation on a phase locked signal of the injection locked oscillator output and one of the amplifier output RF amplified signals. The three-dimensional trajectory detection system of claim 5, wherein the radar device has a signal processing unit, the signal processing unit is coupled to the demodulation circuit to receive an in-phase signal and an orthogonal signal, and the signal is The processing unit calculates the displacement of the moving object in the Z-axis direction according to the in-phase signal and the orthogonal signal. The three-dimensional trajectory detection system of claim 5, wherein the demodulation circuit has a 90 degree phase difference power divider, a power divider, a first mixer, a second mixer, and a a first low pass filter and a second low pass filter, the 90 degree phase difference power splitter is coupled to the injection lock oscillator to receive the lock phase signal, and the 90 degree phase difference power splitter outputs an in-phase local oscillation signal and a An orthogonal local oscillation signal, the power splitter is coupled to the amplifier to receive the RF amplified signal, and the work is performed The rate divider divides the RF amplification signal into two paths, and the first mixer receives the RF amplification signal of the in-phase local oscillation signal and one of the paths for mixing, and the second mixer receives the positive Transmitting the local oscillation signal and the RF amplification signal of another path for mixing, the first low pass filter receiving a first mixed signal of the first mixer for filtering to output the in-phase signal, the first The second low pass filter receives a second mixed signal of the second mixer for filtering to output the orthogonal signal. The three-dimensional trajectory detection system of claim 1, wherein the calculation formula of the first conversion ratio and the second conversion ratio is: Wherein, m _{x is} the first conversion ratio, Δ z is the Z-axis displacement of the moving object, Δ P _x is the X-direction pixel displacement amount of the moving object image, and θ _{x is} the first angle, Δ x For the X-axis direction displacement of the moving object, m _{y is} the second conversion ratio, Δ P _y is the Y-direction pixel displacement amount of the moving object image, θ _{y is} the second angle, and Δ y is the moving object The Y-axis direction is displaced. The three-dimensional trajectory detection system of any one of claims 1 to 8, wherein the image capturing device, the radar device and the computing module are integrated in a mobile device. The three-dimensional trajectory detection system of claim 9, wherein the wireless signal is selected from a signal sent by the mobile device or a signal sent by a wireless base station. A three-dimensional trajectory detection system includes an image capturing device for capturing a moving image, wherein the moving image has a moving object image, and the image capturing device calculates an X-direction pixel displacement of the moving object image And a pixel displacement amount in the Y direction; a radar device that transmits a wireless signal to a moving object and receives a reflected signal from the moving object, and the radar device obtains a Z-axis displacement of the moving object according to a phase shift amount of the reflected signal. The radar device has a coupler, an antenna, an injection-locked oscillator, an amplifier, and a demodulation circuit. The coupler receives the wireless signal, and the coupler couples the wireless signal to The antenna and the injection-locked oscillator cause the injection-locked oscillator to be in an injection-locked state, the antenna transmits the wireless signal to the moving object, and the antenna receives a reflected signal from the moving object Transmitting the reflected signal to the coupler, the coupler coupling the reflected signal to the amplifier, the demodulation circuit coupled to the injection-locked oscillator and the amplifier to lock a phase-locked signal to the injection-locked oscillator output And one of the amplifier outputs an RF amplification signal for quadrature phase demodulation; and a computing module coupled to the image capture device and Radar apparatus, and the displacement amount calculation module according to the X-direction pixels of the image of the moving object, the Z-axis direction to the Y direction of the pixel shift amount of the moving object and the calculated three-dimensional trajectory of displacement of the one moving object.