TWI830407B - Calculating method of ball trajectory based on image and radar - Google Patents

Abstract

The present invention aims to provide a method that can utilize a camera and a radar sensor provided in one sensing device or provided independently of each other when calculating the movement trajectory of a ball hit by a golfer's golf shot. The ball trajectory calculation method based on images and radar is used to match the ball trajectory information calculated through the analysis and calculation of the acquired image with the ball trajectory information calculated through the analysis and calculation of the radar sensing data collected by the radar sensor to calculate the entire trajectory of the ball movement.

Description

Ball trajectory calculation method based on images and radar

The present invention relates to a method of calculating the ball's trajectory using a camera image of a ball that moves as a golfer hits the ball with a golf club and a radar sensor's sensing data.

In the case of sports events utilizing balls, especially golf balls, attempts have been made to accurately sense the physical characteristics of a ball that is moved by being struck by a golfer, and to use the sensed values to perform golf ball analysis or transfer them. Implemented as an image for use in analog golf fields such as so-called screen golf.

As representative devices that sense a ball that moves when a golfer hits it to calculate motion characteristic information of the ball, there are a camera sensing device based on image analysis and a radar sensing device based on radar signals.

As a device that separately analyzes the image of the ball that is acquired by the camera and moves as the golfer hits it to calculate the position information of the ball, and uses the calculated position information to generate the trajectory information of the moving ball, a sensing device based on image analysis Although it has the advantage of being able to detect the position of the ball quite accurately within the shooting angle of the camera, due to the performance of the camera, environmental effects such as surrounding brightness, and image processing performance, it is completely impossible to detect the position of the ball when it is out of the shooting angle of the camera and when it is far away from the camera. Problem with detecting the ball.

In addition, the problem with the camera sensing method is that since it cannot sense the entire flight range of the ball that is hit and moves, it is also completely unable to sense the ball's flight such as fade or draw. A spherical mass in which the ball curves from the middle, so that its trajectory cannot be calculated.

As a device capable of transmitting a radar signal toward a moving ball and receiving a signal reflected from the ball, and calculating information on various characteristics of the ball's motion through analysis of the received signal, the advantage of a radar sensing device is that, due to the radar The wide sensing range enables the ball to be sensed even if the distance between the device and the ball is somewhat far, and difficult-to-sensing information such as the ball's spin can be sensed with considerable accuracy through image analysis , but it is susceptible to peripheral interference to the radar signal and may contain noise, so there is a problem that information such as the trajectory of the ball cannot be accurately calculated.

In particular, in the initial interval of a predetermined distance from the position where the ball is hit, noise is easily generated in the radar signal due to the interference of people and golf clubs, and signal distortion occurs seriously, so it is difficult to calculate the movement of the ball. The problem of trajectory.

In order to calculate the trajectory of the ball that moves as the player hits the ball with a golf club, people are trying to compensate for each other's shortcomings by mixing the camera sensing method and the radar sensing method, but for the camera sensing method, due to the acquisition of two-dimensional In images, the data about the trajectory is also 2D data. As for the radar sensing method, since the position information of the ball in the calculation space is basically 3D data in the three-dimensional space, there are limitations in using 2D data and 3D data at the same time. , and whether the camera and radar are implemented as one device or used independently, due to different sensing positions, there is a problem that it is difficult to fuse the data processing results of the two.

Existing patent documents: Korean Authorized Patent Gazette No. 10-1848864; Korean Public Patent Gazette No. 2019-0085152; Japanese Authorized Patent Gazette No. 4773176; Korean Authorized Patent Gazette No. 10-1845503.

The present invention aims to provide a method that can utilize a camera and a radar sensor provided in one sensing device or provided independently of each other when calculating the movement trajectory of a ball hit by a golfer's golf shot. The ball trajectory calculation method based on images and radar is used to match the ball trajectory information calculated through the analysis and calculation of the acquired image with the ball trajectory information calculated through the analysis and calculation of the radar sensing data collected by the radar sensor to calculate the entire trajectory of the ball movement.

A ball trajectory calculation method based on images and radar according to an embodiment of the present invention includes the steps of: acquiring an image of a moving ball with a camera and analyzing it, and calculating an image processing trajectory as the movement trajectory of the ball; The radar signal sent by the radar sensor is reflected by the moving ball and the received signal calculates the radar sensing data about the movement of the ball, respectively generating multiple radar ball trajectory candidates; calculating the image processing trajectory and the multiple radar a matching rate for each of the ball trajectory candidates; and calculating a ball trajectory by integration of a radar ball trajectory candidate corresponding to the highest matching rate among the matching rates and the image processing trajectory.

In addition, preferably, the step of calculating the image processing trajectory includes the step of calculating the trajectory from the hitting position of the ball until a ball object corresponding to the ball can be detected from the acquired image.

Furthermore, preferably, the step of calculating the ball trajectory through the integration includes the step of extracting a trajectory from a position where the image processing trajectory ends on the radar ball trajectory candidate corresponding to the highest matching rate and connecting to the image processing trajectory.

In addition, preferably, the step of calculating the image processing trajectory includes: the step of collecting images acquired by the camera based on the hitting time point of the ball; and detecting the image corresponding to the hitting position of the ball from the image. A step of calculating a ball object of the ball and a line connecting the ball objects detected until the ball object cannot be detected in the image.

In addition, preferably, the camera is a single camera that acquires a two-dimensional image from a perspective looking in the direction in which the ball is traveling. The step of calculating the image processing trajectory includes: connecting the hitting position of the ball to the detected to obtainable The step of detecting the position of the ball object in the image and displaying it on the two-dimensional image corresponding to the shooting angle of the camera to calculate the image processing trajectory.

In addition, preferably, the step of respectively calculating the plurality of radar ball trajectory candidates includes: using the collected radar sensing data to calculate ball position coordinate information; using the ball position coordinate information to calculate a basic ball as a radar-based ball trajectory. The steps of the trajectory; and the step of changing the viewing angle and zoom ratio of the basic ball trajectory to generate the plurality of radar ball trajectory candidates respectively.

In addition, preferably, the step of calculating the matching rate includes: comparing the image processing trajectory with the plurality of radar ball trajectory candidates to digitize the similarity of the trajectories, and the step of calculating the ball trajectory includes: The step of determining the radar ball trajectory candidate with the highest value of trajectory similarity as the radar ball trajectory candidate corresponding to the highest matching rate; and selecting the radar ball trajectory candidate corresponding to the final position of the image processing trajectory from the determined radar ball trajectory candidate. The trajectory from the position is connected to the image processing trajectory to calculate the ball trajectory.

In addition, preferably, the step of respectively calculating the plurality of radar ball trajectory candidates includes: calculating a basic ball trajectory based on the collected radar sensing data; and changing the viewing angle and zoom ratio of the basic ball trajectory to change the plurality of radar ball trajectory candidates. Each of the radar ball trajectory candidates is generated into a two-dimensional view form corresponding to the angle of view of the two-dimensional image of the camera. The step of calculating the matching rate includes: comparing the image processing trajectory as a two-dimensional image with the two-dimensional image. The step of comparing multiple radar ball trajectory candidates in the form of a dimensional view to digitize the similarity of the trajectories. The step of calculating the ball trajectory includes: comparing the two-dimensional view of the radar ball trajectory candidate with the highest similarity value of the trajectory. The trajectory on is connected to the image processing trajectory to calculate the ball trajectory step.

On the other hand, a ball trajectory calculation method based on images and radar according to another embodiment of the present invention includes: acquiring an image of a moving ball with a camera and analyzing it, and calculating an image processing trajectory as the movement trajectory of the ball to generate a corresponding The step of obtaining a two-dimensional image from the perspective of the camera; the radar sensing data about the movement of the ball is calculated by collecting and analyzing the radar signal sent by the radar sensor and being reflected by the moving ball. The radar-based The step of generating a ball trajectory into a two-dimensional view form corresponding to the perspective of the two-dimensional image of the camera; so that the radar-based ball trajectory of the two-dimensional view form is consistent with the image processing on the two-dimensional image corresponding to the perspective of the camera The step of adjusting the angle of view and zoom ratio of the radar-based ball trajectory in the two-dimensional view form by trajectory matching; and the step of calculating the ball trajectory by matching the image processing trajectory with the radar-based ball trajectory.

In addition, preferably, the step of calculating the image processing trajectory to generate a two-dimensional image corresponding to the angle of view of the camera includes: the step of collecting images acquired by the camera based on the hitting time point of the ball; and from the step of: A ball object corresponding to the ball is detected in the image obtained from the hitting position of the ball, and a line connecting the ball objects detected until the ball object cannot be detected in the image is generated on the two-dimensional image by this image processing trajectory step, the radar-based ball trajectory is generated as the corresponding The steps of creating a two-dimensional view form from the perspective of the two-dimensional image of the camera include: using the collected radar sensing data to calculate the ball position coordinate information in the three-dimensional space; using the ball position coordinate information to calculate the ball position coordinate information in the three-dimensional space. The steps of a radar ball trajectory; and the step of generating the radar-based ball trajectory in the three-dimensional space on a two-dimensional view corresponding to the perspective of the two-dimensional image of the camera.

The ball trajectory calculation method based on images and radar of the present invention has the advantage of being able to utilize cameras and radars provided in one sensing device or provided independently of each other when calculating the movement trajectory of a ball hit by a golfer's golf shot. The sensor matches the ball trajectory information calculated through the analysis of the image obtained by the camera and the ball trajectory information calculated through the analysis and calculation of the radar sensing data collected by the radar sensor to calculate the effect of the entire trajectory of the ball movement.

10: golfer

100:Camera device

100a, 100b: Image

110, 330: two-dimensional image

20: ball

200:Radar sensor

210:Signal transmission department

220: Signal receiving department

230:Signal Analysis Department

240:Information and Computing Department

251~255: Two-dimensional view

300: Trajectory calculation device

B:Ball

BO: ball object

Ob:object

P0~Pf: the position of the ball object

RA1~RA3: receiving antenna

Sr:

St:

TRI: image processing trajectory

TRr1~TRr5: radar ball trajectory candidates

fTR: final ball trajectory

rP1, rP2, rP3

t1, t2, t3

x1, x2, x3

y1, y2, y3

z1, z2, z3

FIG. 1 is a diagram illustrating a state in which a camera device and a radar sensor installed behind a golfer sense the movement of a ball as the golfer hits a golf shot on a golf course.

FIG. 2 is a diagram showing an example of an image acquired by a camera at a predetermined shooting speed in the state shown in FIG. 1 and an example of an image processing trajectory generated using the same.

Figures 3 and 4 are used to illustrate that the radar sensor calculates the position coordinate information of the moving ball through the analysis of radar signals.

FIG. 5 is a flow chart illustrating a ball trajectory calculation method based on images and radar according to an embodiment of the present invention.

FIG. 6 is a diagram illustrating the generation and matching of an image processing trajectory and a plurality of radar ball trajectory candidates based on the flowchart of FIG. 5 .

7 is a diagram illustrating the calculation of the final ball trajectory by combining the radar ball trajectory candidate with the highest matching rate with the image processing trajectory among the plurality of radar ball trajectory candidates shown in FIG. 5 and the image processing trajectory.

FIG. 8 is a flowchart illustrating a ball trajectory calculation method based on images and radar according to another embodiment of the present invention.

The specific contents of the ball trajectory calculation method using radar sensing data of a struck ball and the radar sensing device using the same according to the present invention will be described below with reference to the accompanying drawings.

The image- and radar-based ball trajectory calculation method of the present invention will be described in detail below with reference to the accompanying drawings.

A ball trajectory calculation method according to an embodiment of the present invention is a method that uses the results sensed by a camera sensing device and a radar sensing device to calculate the ball trajectory when a golfer hits the ball with a golf club in an outdoor area such as a golf course. The method of ball trajectory within the overall range of ball movement.

FIG. 1 is a diagram illustrating a state in which the camera device 100 and the radar sensor 200 installed behind the golfer 10 sense the movement of the ball 20 as the golfer 10 performs a golf shot on the golf course.

The ball trajectory calculation method based on images and radar according to an embodiment of the present invention not only includes a camera device 100 and a radar sensor 200 that are independently configured to sense from a perspective looking toward the direction of the ball 20 as shown in FIG. 1 The situation may also include the situation where the camera and the radar sensor are configured together in a sensing device housing to perform image analysis and radar analysis respectively.

In addition, the camera may be a single camera that acquires a two-dimensional image from its angle of view, or it may be a camera device that is composed of two cameras in a three-dimensional manner like a launch monitor. The stereoscopic camera device can calculate the three-dimensional position coordinates of the ball by using the data on the two-dimensional image acquired by each camera.

Regardless of whether it is a single camera or a three-dimensional camera device, when the ball is far away from the camera device, the ball cannot be detected from the image. Therefore, the camera device can only sense the distance interval where the ball can be detected from the initial starting position of the ball. Measure the moving ball and calculate the trajectory of the ball up to that interval.

The basic feature of the present invention is that the trajectory of the ball is calculated by analyzing the image captured by the camera from the initial departure of the ball to the period when the ball can be detected in the image captured by the camera, and the trajectory of the ball thereafter is calculated using the radar sensing device. The ball trajectory is calculated based on the sensed radar sensing data, and the image-based trajectory and the radar-based trajectory are matched to calculate the entire ball trajectory.

The ball trajectory calculation method based on images and radar according to an embodiment of the present invention can be connected to a camera device and a radar sensor to calculate the ball trajectory using the image analysis results of the camera device and the radar analysis results of the radar sensor. The client device may be executed, or may be executed by components used for information calculation in a sensing device equipped with both a camera and a radar.

As mentioned above, the main body of the ball trajectory calculation method of the present invention can be implemented through various forms of hardware settings or software configurations. For example, the present invention can be implemented by a separately installed client device or a sensor equipped with a camera and a radar sensor. In the following, the main body that executes the ball trajectory calculation method of the present invention will be named "trajectory calculation device" for description.

FIG. 1 shows a device for calculating a ball trajectory calculation method according to an embodiment of the present invention, which is connected to a camera device 100 and a radar sensor 200 (communication can be performed in a wired or wireless manner). ground-connected) trajectory calculation device 300. However, the present invention is not limited to this. As mentioned above, the trajectory calculation device can be implemented in various forms.

For example, the above-mentioned trajectory calculation device can be implemented as a separate terminal (a terminal that connects a camera and a radar sensor through a wired/wireless manner to perform calculations for trajectory calculation using data received from the camera and radar sensor), It can also be implemented in a smartphone that drives an application for trajectory calculation by connecting to a camera and radar sensor and performing calculations.

FIG. 2 shows an example of an image acquired by a camera at a predetermined shooting speed in the state shown in FIG. 1 .

Figure 2(a) is the n-th frame image 100a acquired by the camera from its perspective, Figure 2(b) is the n+m-th frame image 100b, and there are multiple frames of images between the image 100a and the image 100b.

The image processing unit provided in the camera device or the trajectory calculation device that calculates the trajectory of the ball can detect the object corresponding to the ball in the image for each frame of the image to specify the position of the ball object in the image.

As shown in (a) of Figure 2, the camera device can find and detect the object corresponding to the ball in the image 100a, that is, the ball object BO, and can store the detected position of the ball object BO on the two-dimensional image.

(a) of Figure 2 shows a state in which the ball object moves a predetermined distance from the initial starting position PO. Although it cannot be seen from the two-dimensional image, since the ball actually flies forward, the actual ball gradually moves away from the camera device. As the camera acquires images, the size of the ball object in the image becomes smaller and smaller.

In this way, the size of the ball object gradually becomes smaller, and then as shown in Figure 2(b), in the image of the n+mth frame, its size becomes so small that it is difficult to determine whether the detected object Ob is a ball object. Finally, The ball object will not be detected from the image.

The trajectory calculation device can use the results of detecting the ball object in each image acquired by the camera (using the detection results of the ball object until the ball object cannot be detected in the image) to calculate the connection as shown in (c) of Figure 2 The trajectory of the positions P0, P1, P2...Pf of each ball object is the image processing trajectory TRI.

Since the camera that acquires the image is a single camera, the images acquired by the camera (100a, 100b, etc.) are all two-dimensional images corresponding to the camera's perspective. Therefore, the image processing trajectory TRI shown in Figure 2(c) can be connected by connecting the two-dimensional A line is created for the position of the ball object on image 110 .

Since the image processing trajectory TRI as described above is a trajectory until the ball object can be detected in the image captured by the camera, it is impossible to extract the movement of the ball from the time when the actual ball moves away from the camera and the ball object can no longer be detected in the image captured by the camera. Therefore, the trajectory of the ball in the interval from when the ball object cannot be detected can be obtained using the radar sensing data collected by the radar sensor.

Figures 3 and 4 show that the radar sensor calculates the position coordinate information of the moving ball through analysis of the radar signal.

A radar sensor is a device that basically uses the Doppler Effect of radar to calculate various information such as the motion characteristics of a golf club shot. As shown in Figure 3, it can include signal transmission. unit 210, signal receiving unit 220, signal analysis unit 230 and information calculation unit 240.

The radar sensor may be disposed on the ground or near the ground behind a predetermined distance from the position of the ball to be struck by the user, and may be configured to transmit a specific frequency at the disposed position toward the direction of movement of the ball to be struck. Radar signals and track the movement of the ball as it is struck while receiving and analyzing the reflected waves reflected from the ball.

The signal transmitting part 210 may be configured to transmit a specific radar (Radar) signal in a aiming direction, and, although not shown in the figure, may include a transmitting antenna for transmitting the radar signal.

The signal receiving unit 220 is configured to receive a reflected wave signal in which the radar signal transmitted by the signal transmitting unit 210 is reflected from the ball and returned. Due to the Doppler effect, the frequency of the signal transmitted by the signal transmitting unit 210 changes with respect to the reflected wave signal transmitted by the signal transmitting unit 210 and reflected from the ball, thereby causing a Doppler shift (Doppler shift). That is, the signal receiving unit 220 receives a signal that is Doppler shifted.

The signal receiving unit 220 may be configured to include multiple receiving antennas that receive the reflected wave signals, so as to calculate the position, speed, trajectory, direction angle and other information of the moving ball using the phase difference of the received signals of the multiple receiving antennas. .

(a) of FIG. 4 schematically shows an example of the structure of the above-mentioned signal receiving unit 220. As shown in the figure, when the signal receiving unit 220 is appropriately arranged and provided with three or more receiving antennas including RA1, RA2 and RA3, Each of the receiving antennas RA1, RA2, and RA3 can receive the reflected wave signal received from the ball B, and the trajectory (elevation angle) and direction angle of the moving ball B can be respectively calculated using the phase difference of the signals between each receiving antenna.

For example, according to the configuration of each receiving antenna in the signal receiving unit 220 shown in (a) of FIG. 4 , the trajectory of the moving ball B can be calculated using the phase difference of the signals received by RA1 and RA2 respectively, and the trajectory of the moving ball B can be calculated using RA1 and RA2 RA3 respectively receives the phase difference of the signals to calculate the direction angle of the moving ball B.

In addition, as the signal receiving part 220 receives the reflected wave signal, the distance between the ball B and the signal receiving part 220 can also be easily calculated. Therefore, when the signal receiving part 220 receives the reflected wave signal reflected from the ball B, You can know the distance to ball B, the trajectory angle of ball B, and the direction angle of ball B. Degree information, using the above information, the position coordinate information of ball B can be calculated. When the position coordinate information of ball B is calculated, it can also be used to calculate the speed of the ball.

In this way, the signal analysis unit 230 may be configured to calculate the phase value information by analyzing the radar signal reflected by the ball at each predetermined time interval when the ball moves, and calculate the position coordinate information of the moving ball at predetermined time intervals.

That is, the signal analysis unit 230 can calculate radar sensing data about the movement of the ball by analyzing the reflected wave signal of the radar received as described above. The radar sensing data can be the phase information of the ball calculated at a predetermined time interval, or It can be the position coordinate information of the ball calculated based on the phase information.

The information calculation unit 240 can calculate the movement trajectory of the ball using the position coordinate information of the ball calculated by the signal analysis unit 230 (to distinguish it from the image processing trajectory calculated through image analysis, it is called "radar-based ball trajectory").

The radar signal reflected by the ball and received by the signal receiving unit may suffer distortion and phase jitter of the radar signal caused by interference between the user and the club, buckling of the ball surface, etc., and the information calculation unit 240 compensates for such distortion or phase of the signal. Jitter, and calculate the movement trajectory of the ball through statistical analysis of radar sensing data.

However, the signal distortion and phase jitter of the radar signal are particularly serious in the initial movement interval corresponding to the predetermined distance interval after the ball is launched. In the initial movement interval, the ball movement trajectory calculated using radar sensing data may be different from the actual movement interval. The trajectory of the ball is different, so the trajectory calculated through image analysis may be more accurate during the ball's initial motion interval.

In the present invention, the method of calculating the radar-based ball trajectory using radar sensing data can be implemented through a variety of methods.

For example, the reliability of each radar sensing data can be analyzed, a weight value can be assigned to each data based on the reliability, and the weight value can be reflected. The ball trajectory can be calculated through statistical analysis such as the RANSAC algorithm, or it can also be based on the data corresponding to the radar. Partial trajectories are obtained for data with high reliability in the range of ball position coordinates of the sensing data, and the radar sensing data is fitted to the partial trajectories through a physics engine to calculate radar-based ball trajectories.

In the present invention, the specific method of calculating the radar-based ball trajectory using radar sensing data is not characterized in itself, but the feature is that, no matter what method it is, the radar-based ball trajectory calculated by any method is combined with image processing. Trajectories are combined to calculate the entire trajectory of the ball's movement.

The specific process of the ball trajectory calculation method based on images and radar according to an embodiment of the present invention will be described below.

FIG. 5 is a flow chart illustrating the process of a ball trajectory calculation method based on images and radar according to an embodiment of the present invention.

When the camera device and the radar sensor are placed at a predetermined distance based on the user's hitting position (or a sensing device that implements the camera and the radar sensor as one device is placed at a predetermined distance), and the desired hitting position is When the hit ball is in place, the camera device senses the ball, thereby becoming ball ready (meaning that the camera device and the radar sensor are ready to sense the ball) (S100).

As the ball is hit (S110), an image is acquired by the camera (S121), and the signal analysis unit of the radar sensor can collect radar sensing data about the movement of the ball by analyzing the signal received by the signal receiving unit (S131) .

The camera device detects an object corresponding to the captured portion of the actual ball, that is, a ball object in each of the acquired images (S122).

In the detection of the ball object, ball object requirements such as the shape of the ball, that is, the golf ball, the roundness of the object, etc. can be preset, and the objects in the image that meet the requirements can be detected as ball objects, or the corresponding objects can be stored in advance. A template of a ball is generated, and the ball object is detected by comparing the object detected in the image with the template and judging the similarity.

However, the ball object cannot be detected in all images in the entire range of the ball movement. When the distance between the camera and the ball exceeds a predetermined distance, since the position of the camera is fixed, the corresponding image in the image acquired by the camera The size of the part about the ball becomes too small so that the ball object can no longer be detected.

When the detection of the ball object in the image obtained from the camera fails (S123), that is, when the ball object can no longer be detected from the image, the camera device or the trajectory calculation device can use the image detection from each frame until the ball object cannot be detected. The ball object calculates the image processing trajectory and displays it on the camera image (S124).

For example, as shown in (c) of FIG. 2 , the positions P0, P1, P2, ... superior.

On the other hand, the radar sensor can use the collected radar sensing data to calculate the ball position coordinate information as shown in (b) of Figure 4 (S132). Then, the ball position coordinate information can be used to calculate the radar-based ball trajectory (S133). Here, the radar-based ball trajectory calculated in step S133 is called a "basic ball trajectory".

As mentioned above, the ball position coordinates calculated using the radar sensing data collected by the radar sensor are three-dimensional coordinate information. Therefore, the basic ball trajectory can be a trajectory in the three-dimensional space calculated using the three-dimensional coordinate information.

The radar sensor or the trajectory calculation device may respectively generate a plurality of radar ball trajectory candidates from the basic ball trajectory as described above (S134). The multiple radar ball trajectory candidates may respectively be trajectories generated with different viewing angles and/or zoom ratios for the same basic ball trajectory.

For example, the basic ball trajectory can be generated into a two-dimensional view form corresponding to the angle of view of the two-dimensional image of the camera, and the angle of view and zoom ratio of the two-dimensional view can be changed to generate a plurality of different angles and/or zoom ratios. Radar ball trajectory candidates in 2D view form.

The trajectory calculation device that executes the method of the present invention can calculate the matching rate between the image processing trajectory as described above and each of the multiple radar ball trajectory candidates (S140), and select the image processing trajectory with the highest matching rate, wherein, Select a radar ball trajectory candidate combination (S150), and integrate the image processing trajectory and the radar ball trajectory candidate with the highest matching rate to calculate the final ball trajectory (S160).

Regarding the calculation of the matching rate between the image processing trajectory and the radar ball trajectory candidate through the above steps S134 to step S160 and the calculation of the final ball trajectory through the corresponding matching, a more specific description will be made with reference to FIGS. 6 and 7 .

(a) of FIG. 6 shows the image processing trajectory TRI analyzed and calculated by the camera on the two-dimensional image 110 corresponding to the angle of view of the camera. (b) to (f) of FIG. 6 respectively show the analysis and calculation of the image obtained by the radar. Radar ball trajectory candidates are generated from the basic ball trajectory calculated from the sensing data.

The above image processing trajectory TRI is information on the two-dimensional image 110 and is therefore 2D data. However, the basic ball trajectory calculated using radar sensing data is information on three-dimensional space and is 3D data. Therefore, it is difficult to directly combine 2D data and 3D data. , so it is preferable to convert the basic radar-based ball trajectory as 3D data into 2D data to generate radar ball trajectory candidates, and, for image processing trajectories and radar ball trajectories To match the trace candidates, it is preferable to generate the radar ball trajectory candidates into a form such as two-dimensional views 251 and 252 in the form of the two-dimensional image 110 .

As shown in (b) to (f) of Figure 6 , the radar ball trajectory candidates show the basic ball trajectory as a two-dimensional view form, and each radar ball trajectory candidate (two-dimensional views 251, 252, 253, 254, 255, etc.) The generated images may be generated according to different angles of view and/or zoom ratios from each other while adjusting the viewing angle and zoom ratio within a preset range, and the generated number may also be preset.

The greater the number of radar ball trajectory candidates, the more accurate the ball trajectory that can be exported. However, it may take more time to calculate the matching rate with the image processing trajectory. Therefore, it is possible to determine a trajectory that ensures a certain degree of accuracy without compromising the accuracy. It would take too much time to generate a suitable amount of radar ball trajectory candidates.

Each radar ball trajectory candidate generated as a 2D view morphology is generated from the same basic ball trajectory, but each 2D view is generated with a different perspective and/or zoom ratio of the 2D view. Although the trajectories within are generated from the same basic ball trajectory, they may be displayed in different forms from each other as two-dimensional views.

Assume that (b) of Figure 6 is the first radar ball trajectory candidate TRr1 in the form of the two-dimensional view 251, (c) of Figure 6 is the second radar ball trajectory candidate TRr2 in the form of the two-dimensional view 252, and (d) of Figure 6 is The third radar ball trajectory candidate TRr3 in the form of the two-dimensional view 253, Figure 6 (e) is the fourth radar ball trajectory candidate TRr4 in the form of the two-dimensional view 254, and Figure 6 (f) is the fifth radar ball trajectory candidate in the form of the two-dimensional view 255 Radar ball trajectory candidate TRr5, in addition to (b)~(f) in Figure 6, can also generate a larger number of radar ball trajectory candidates.

In the following, the five radar ball trajectory candidates in (b) to (f) of Figure 6 will be described.

As described above, when the image processing trajectory TRI and the plurality of radar ball trajectory candidates TRr1 ~ TRr5 are provided, the trajectory calculation device calculates the matching rate of the image processing trajectory TRI and the plurality of radar ball trajectory candidates TRr1 ~ TRr5 to determine the matching Image processing trajectory-radar ball trajectory candidate combination with higher rate.

That is, the two-dimensional image 110 and the two-dimensional views 251 to 255 have the same form and are easy to compare with each other. Each of the image processing trajectory TRI as the two-dimensional image and the radar ball trajectory candidates TRr1 to TRr5 in the two-dimensional view form can be A comparison is performed to digitize the similarity of the trajectories, and the image processing trajectory-radar ball trajectory candidate combination with the highest numerical value of trajectory similarity can be determined as the combination with the highest matching rate.

The similarity of the trajectory between the image processing trajectory and the radar ball trajectory candidate can be expressed numerically by numericizing the similarity in the morphological and geometric features of the trajectory. At this time, in addition to the morphological characteristics of the trajectory, the ball on the trajectory is also judged. When selecting the ball quality, it can be judged together with the ball quality to take into account the similarity in ball quality between the image processing trajectory and the radar ball trajectory candidate.

In the example shown in Figure 6, the matching rate between the image processing trajectory TRI and the first radar ball trajectory candidate TRr1 can be calculated, and then the matching rate between the image processing trajectory TRI and the second radar ball trajectory candidate TRr2 can be calculated, and then the image The matching rate between the processing trajectory TRI and the third radar ball trajectory candidate TRr3 is then calculated, and each radar ball trajectory is compared with the image processing trajectory one-to-one by calculating the matching rate between the image processing trajectory TRI and the fourth radar ball trajectory candidate TRr4. And analyze the similarity of trajectories to find the combination with the highest matching rate, that is, the highest similarity.

When it is assumed that the image processing trajectory TRI in Fig. 6 has the highest matching rate with the fourth radar ball trajectory candidate TRr4 shown in (e) of Fig. 6, as shown in Fig. 7, the trajectory calculation device can integrate the (e) of Fig. 7 The image processing trajectory TRI shown in a) and the fourth radar ball trajectory candidate TRr4 shown in FIG. 7(b) are used to calculate the final ball trajectory fTR shown in FIG. 7(c).

That is, the final ball trajectory can be calculated by combining the image processing trajectory TRI - the radar ball trajectory candidate TRr4 which determines the trajectory similarity with the highest value, and connecting the radar ball trajectory candidate TRr4 to the image processing trajectory TRI.

When the radar ball trajectory candidate TRr4 is connected to the image processing trajectory TRI, it can be calculated by connecting the trajectory part TRr from the last position of the image processing trajectory TRI in the radar ball trajectory candidate TRr4 to the last position of the image processing trajectory TRI as shown in Figure 7 The final ball trajectory fTR on the two-dimensional image 330 shown in (c).

On the other hand, a ball trajectory calculation method based on images and radar according to another embodiment of the present invention will be described with reference to the flow chart shown in FIG. 8 .

Compared with the previously mentioned ball trajectory calculation method of the embodiment shown in FIG. 5 , the ball trajectory calculation method of the embodiment shown in FIG. 8 combines the image processing trajectory and the radar-based ball trajectory to calculate the overall ball movement. There are similarities in terms of trajectories, but there are differences in the process of matching radar-based ball trajectories with image-processed trajectories to calculate the final ball trajectory.

First, when the camera device and the radar sensor are placed at a predetermined distance based on the user's hitting position (or the sensing device that implements the camera and the radar sensor as one device is placed at a predetermined distance), and the desired hitting position is When the hit ball is in place, the camera device senses the ball, thereby becoming ball ready (meaning that the camera device and the radar sensor are ready to sense the ball) (S200).

As the ball is hit (S210), an image is acquired by the camera (S221), and the signal analysis unit of the radar sensor can collect radar sensing data about the movement of the ball by analyzing the signal received by the signal receiving unit (S231) .

The camera device detects an object corresponding to the captured part of the actual ball in each acquired image, that is, a ball object (S222). When the detection of the ball object in the image acquired from the camera fails (S223), That is, when the ball object can no longer be detected from the image, the camera device or the trajectory calculation device can calculate the image processing trajectory using the ball object from the image detection of each frame until the ball object cannot be detected, and display it on the camera image. (S224).

The process of the above steps S221 to S224 is substantially the same as the process of steps S121 to S124 of Figure 5 mentioned before. As shown in Figure 2 (c), the position of the ball object detected by the image can be obtained from each camera. (P0, P1, P2...Pf) are connected to calculate the image processing trajectory TRI, and it is displayed on the two-dimensional image 110.

On the other hand, the radar sensor can use the collected radar sensing data to calculate the ball position coordinate information in the three-dimensional space (S232).

Then, the ball position coordinate information in the three-dimensional space can be used to calculate the radar-based ball trajectory (S233).

The trajectory calculation device generates the radar-based ball trajectory in the three-dimensional space into a two-dimensional view form corresponding to the angle of view of the two-dimensional image of the camera (S234). For example, a two-dimensional view form of one of the forms in (b) to (f) of Figure 6 can be generated.

The trajectory calculation device adjusts the angle of view and the radar-based ball trajectory of the two-dimensional view form in such a manner that the radar-based ball trajectory in the two-dimensional view form matches the image processing trajectory on the two-dimensional image corresponding to the angle of view of the camera. /or scaling (S240). This adjustment can continue until matching is complete.

When the angle of view and/or zoom ratio of the radar-based ball trajectory in the two-dimensional view form is adjusted in this way to complete the matching of the image processing trajectory and the radar-based ball trajectory (S250), the trajectory calculation device matches the image processing trajectory with the radar-based ball trajectory. The trajectory is matched to calculate the final ball trajectory (S260).

For example, the final ball trajectory can be calculated by extracting the trajectory portion from the last position of the image processing trajectory for the radar-based ball trajectory of the two-dimensional view morphology described above and concatenating it to the last position of the imaging trajectory.

As mentioned above, when calculating the ball trajectory calculation method based on images and radar of the present invention, the camera and radar can be used to calculate the trajectory of the ball hit by the golfer's golf shot. The sensor matches the ball trajectory information calculated through the analysis of the image acquired by the camera with the ball trajectory information calculated through the analysis and calculation of the radar sensing data collected by the radar sensor to calculate the entire trajectory of the ball movement. The trajectory of a partial interval generated by the image analysis uses the radar sensor to extend to the entire interval to generate the characteristics of the trajectory.

10: golfer

100:Camera device

20: ball

200:Radar sensor

300: Trajectory calculation device

Claims (10)

A ball trajectory calculation method based on images and radar, which is characterized by including: the steps of obtaining and analyzing the image of the moving ball by a camera, and calculating the image processing trajectory as the moving trajectory of the ball; by collecting and analyzing the image processed by the radar The radar signal sent by the sensor is reflected by the moving ball and the received signal calculates the radar sensing data about the movement of the ball, and generates multiple radar ball trajectory candidates respectively; calculates the image processing trajectory and the multiple radar ball trajectory candidates. the steps of matching a rate for each of the trajectory candidates; and calculating a ball trajectory by integration of a radar ball trajectory candidate corresponding to the highest matching rate among the matching rates and the image processing trajectory. The ball trajectory calculation method based on images and radar according to claim 1, wherein the step of calculating the image processing trajectory includes: detecting from the hitting position of the ball corresponding to the ball from the acquired image. The trajectory up to the spherical object is calculated as the step of the image processing trajectory. The ball trajectory calculation method based on images and radar according to claim 1, wherein the step of calculating the ball trajectory through the integration includes: extracting the radar ball trajectory candidate corresponding to the highest matching rate from the image processing trajectory. position of the trajectory and connect the steps to that image processing trajectory. The ball trajectory calculation method based on images and radar according to claim 1, wherein, The step of calculating the image processing trajectory includes: collecting images acquired by the camera based on the hitting time point of the ball; and detecting a ball object corresponding to the ball from the image acquired from the hitting position of the ball. , and a line connecting the ball objects detected until the ball object cannot be detected in the image is calculated as a step of processing the trajectory of the image. The ball trajectory calculation method based on images and radar according to claim 1, wherein the camera is a single camera that obtains two-dimensional images from a perspective looking in the direction of the ball, and the step of calculating the image processing trajectory includes: The step of calculating the image processing trajectory by connecting the hitting position of the ball to the position of the ball object until the ball object can be detected from the acquired image and displaying it on the two-dimensional image corresponding to the shooting angle of the camera. The ball trajectory calculation method based on images and radar according to claim 1, wherein the step of calculating the plurality of radar ball trajectory candidates includes: using the collected radar sensing data to calculate ball position coordinate information; using the The steps of calculating the ball position coordinate information as a basic ball trajectory as a radar-based ball trajectory; and the steps of changing the viewing angle and zoom ratio of the basic ball trajectory to generate the plurality of radar ball trajectory candidates respectively. The ball trajectory calculation method based on images and radar according to claim 6, wherein the step of calculating the matching rate includes: The step of comparing the image processing trajectory with the plurality of radar ball trajectory candidates to numerically quantify the similarity of the trajectories. The step of calculating the ball trajectory includes: determining the radar ball trajectory candidate with the highest value of the similarity of the trajectories. Be the step of the radar ball trajectory candidate corresponding to the highest matching rate; and calculate the image processing trajectory by connecting the trajectory from the position corresponding to the last position of the image processing trajectory in the determined radar ball trajectory candidate to the image processing trajectory. Ball trajectory steps. The ball trajectory calculation method based on images and radar according to claim 1, wherein the step of calculating the plurality of radar ball trajectory candidates includes: calculating the basic ball trajectory based on the collected radar sensing data; and calculating the basic ball trajectory based on the collected radar sensing data; The basic ball trajectory changes the angle of view and zoom ratio to generate each of the plurality of radar ball trajectory candidates into a two-dimensional view form corresponding to the angle of view of the two-dimensional image of the camera. The step of calculating the matching rate includes: The step of comparing the image processing trajectory as a two-dimensional image with multiple radar ball trajectory candidates in the two-dimensional view form to digitize the similarity of the trajectories. The step of calculating the ball trajectory includes: A step in which the trajectory on the two-dimensional view of the radar ball trajectory candidate with the highest numerical value is connected to the image processing trajectory to calculate the ball trajectory. A ball trajectory calculation method based on images and radar, characterized in that it includes: obtaining an image of a moving ball by a camera and analyzing it, and calculating an image processing trajectory as the movement trajectory of the ball to generate an image corresponding to the angle of view of the camera. 2D imaging steps; The radar-based ball trajectory is generated into a two-dimensional image corresponding to the camera by collecting and analyzing radar sensing data about the movement of the ball calculated from the radar signal sent by the radar sensor being reflected by the moving ball. the steps of adjusting the radar-based ball trajectory of the two-dimensional view form to match the image processing trajectory on the two-dimensional image corresponding to the camera's perspective; the steps of viewing and scaling the radar ball trajectory; and the steps of calculating the ball trajectory by matching the image processing trajectory with the radar-based ball trajectory. The ball trajectory calculation method based on images and radar according to claim 9, wherein the step of calculating the image processing trajectory to generate a two-dimensional image corresponding to the angle of view of the camera includes: taking the hitting time point of the ball as a reference The steps of collecting images acquired by the camera; and detecting a ball object corresponding to the ball from the image acquired from the hitting position of the ball, and connecting the ball objects detected until the ball object cannot be detected from the image. The step of generating the image processing trajectory of the line on the two-dimensional image, and the step of generating the radar-based ball trajectory into a two-dimensional view form corresponding to the angle of view of the two-dimensional image of the camera includes: using the collected radar The steps of calculating the ball position coordinate information in the three-dimensional space from the sensing data; the steps of using the ball position coordinate information to calculate the radar-based ball trajectory in the three-dimensional space; and generating the radar-based ball trajectory in the three-dimensional space corresponding to Step on the two-dimensional view of the camera's two-dimensional image perspective.