JP2003348424A - Motion tracking apparatus and method - Google Patents

Abstract

(57) [Summary] [Problem] To be able to accurately follow the position of each player when broadcasting a sport, and to superimpose and display information such as a player name on a relay video. SOLUTION: Transmitters 10 ₁ to 10 ₁₀ are attached to fixed positions of soccer players P1 to P5, a ball and a ground, respectively. Transmitter 10 ₁ to 10 ₁₀ includes an infrared LED flashing in a particular pattern. Transmitter 10
_{1-10 10} flashing lights having different pattern for each object respectively attached occurs, the receiver can identify the objects from the pattern of flashing light having an image pickup device. The receiver captures the state of the game, receives the blinking light from the transmitters 10 ₁ to ₁₀ 10 attached to each object, detects the coordinate position of each object on the coordinates of the light receiving surface, and The object ID, its position data, and the imaging signal are output. Object ID
, For example, a player name is superimposed on the captured image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion tracking apparatus and method for tracking the movement of, for example, an athlete.

[0002]

2. Description of the Related Art If it is possible to follow the movement of a player during a sports competition such as a soccer game, it is possible to display a player name or the like during a sports broadcast. Conventionally, by moving a person wearing a marker of a luminous body and photographing its appearance,
A motion tracker that captures a person's movement is known.
In this device, the marker is relatively large, and there is a problem that the marker becomes an obstacle for a player. Further, it is necessary to change the color of the marker and the like for each player, and it is difficult to identify many players. Furthermore, the requirement that the entire system be synchronized makes it unsuitable for use in open environments such as outdoor stadiums.

[0003] A magnetic tracker is known in which a base station generates a magnetic field, and in the presence of the magnetic field, a person moves while wearing a coil and transmits an electric signal induced in the coil to the base station by wire. I have. The magnetic tracker has a problem in that it is difficult to follow a wide range of movement, and the size of the device becomes large.

[0004] Furthermore, it is conceivable to obtain information for identifying players by a cyber code, which is a mosaic-shaped two-dimensional bar code. In order to recognize the cyber code, it is necessary to perform a process of binarizing the captured image and detecting the cyber code from the binarized image. Visual codes such as cyber codes are affected by the image quality of each frame of video and are not suitable for those that follow fast moving objects. Furthermore, it is necessary to enlarge the cyber code so that the cyber code can be recognized from a distance, which poses a design problem. Further, a method of shooting a sports game and manually analyzing the shot image is also possible. However, the manual analysis has a huge cost, and it has been difficult to perform the analysis in real time.

Accordingly, an object of the present invention is to provide a motion tracking apparatus and a method thereof that can solve the above problems and accurately track the position of each player in real time.

[0006]

According to a first aspect of the present invention, there is provided a tracking apparatus for tracking the movement of a plurality of objects. A transmitter for transmitting transmission data to be identifiable as an optical signal; optical signals from a plurality of transmitters being received by a two-dimensional light receiving surface; A motion tracking device, comprising: a receiver for acquiring a plurality of objects; an imaging unit that acquires captured images of a plurality of objects; and a combining unit that superimposes information related to the objects at positions indicated by position data in the captured images. It is. According to a tenth aspect of the present invention, in the captured image,
This is a motion tracking method in which information related to an object is superimposed on a position indicated by position data.

According to a fourth aspect of the present invention, in the tracking device for tracking the movement of a plurality of objects, a plurality of objects and a fixed position defining an area where the object moves are respectively added, and the plurality of objects and the fixed position can be identified. A transmitter that transmits transmission data as an optical signal, and optical signals from a plurality of transmitters are received by a two-dimensional light receiving surface, and position data of a light receiving position of the optical signal on the two-dimensional light receiving surface is fixed for each object and fixed. The motion tracking device includes a receiver for each position and a conversion unit for obtaining position data of the object in an area where the object moves based on information on the fixed position. An invention according to claim 11 is a motion tracking method for obtaining position data of an object in an area where the object moves based on information on the fixed position.

According to a seventh aspect of the present invention, there is provided a tracking device for tracking the movement of a plurality of objects, wherein the transmitter transmits transmission data, which is added to each of the plurality of objects and makes the plurality of objects identifiable, as an optical signal. A receiver that receives optical signals from a plurality of transmitters on a two-dimensional light receiving surface and obtains position data of a light receiving position of the optical signal on the two-dimensional light receiving surface for each object, and a recording unit that records the position data of the object And a motion tracking device. A twelfth aspect of the present invention is a motion tracking method for recording position data of an object.

According to the present invention, a plurality of transmitters and receivers do not need to be synchronized, and motion tracking applicable to a wide open area can be realized. Therefore, according to the present invention, when relaying a sport, a horse race, a car race, and the like, the position of each object such as a player, a racehorse, and a car can be accurately grasped. In particular, the position on the captured image can be accurately followed, and information related to the object such as a player name can be superimposed and displayed on the relay video. The transmitter according to the present invention includes:
It is very small and lightweight, and does not hinder the movement of objects such as players.

[0010]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below. FIG. 1A shows a state during a soccer game, for example, in which each of players P1 to P5 has a transmitter 1
I wear 0 ₁ to 10 ₅ . Figure 1B shows that the player P1 is wearing a transmitter 10 _1. Also,
Transmitter ₁₀₆ is added with respect to the ball B football. Further, transmitters 10 ₇ , 10 ₈ , 10 ₉ , and 10 ₁₀ are attached to corners of the soccer field and intersections of the center line and the side lines, respectively.

As will be described later, the transmitters 10 ₁ to 10 _1
10 is a light emitting body that blinks in a specific pattern, for example, an LED that emits infrared light (Light Emitting Diode: Light Emitting Diode), and a circuit unit for driving the LED, which is enough to not hinder the movement of the player It is small. The transmitters 10 ₁ to 10 ₁₀ are objects to which each is attached (moving people, animals, objects, etc., to which the transmitters such as players, balls, stadiums, and racehorses are attached are collectively referred to as objects). A different pattern of flashing light is generated every time. The correspondence between the blinking light pattern and the object is predetermined, and the object can be recognized by identifying the blinking light pattern.

As shown in FIG. 1C, the transmitters 10 ₁ to 10 _{1
Since 10} generates infrared flashing light, the flashing light cannot be visually recognized, so that the flashing light does not disturb the athlete's game or the spectator's watching the game.
Each object, for example, a player may have two or more transmitters at different parts such as the front and the back.

[0013] The blinking light generated by each of the transmitters 10 ₁ to 10 ₁₀ is received by the receiver. As will be described later, the receiver has an image sensor, captures the state of a soccer game from an arbitrary position, and receives blinking light from the transmitters 10 ₁ to ₁₀ 10 attached to each object. .

FIG. 2 schematically shows a configuration of a receiving side in one embodiment, and reference numeral 20 indicates a receiver. As will be described later, the receiver 20 decodes each of the plurality of received blinking lights to identify the object, detects the coordinate position on the light receiving surface of each object on the coordinates, and detects the ID of each object and its position data. S1 is output. Also,
The receiver 20 outputs an image signal S2 captured by the receiver 20.

The ID and position data (for example, coordinate data) S1 and the image pickup signal S2 from the receiver 20
1 is supplied. The image synthesizing device 31 uses the database 3
2 is queried for information related to the ID of the object, for example, a player name. In response to the inquiry, the relevant information is supplied from the database 32 to the image synthesizing device 31,
Related information is superimposed on the captured image. In this case, the related information is superimposed so as to be displayed at the position of the corresponding object. A video output in which related information is superimposed on a video of a soccer match is obtained from the image synthesizing device 31.

As shown in FIG. 3, a receiver 20a having a function of receiving blinking light from a transmitter and a video camera 20b having a function as a camera for shooting a soccer game are separately provided. Is also good. These receivers 2
The camera 0a and the video camera 20b use lenses having the same angle of view, and are installed so that the optical axes of the lenses are parallel to each other. As shown in FIG. 4, since the distance between the receiver 20a and the video camera 1b and the subject, for example, a player, is considerably long, the angle θ formed by each optical axis with respect to the subject becomes very small. As a result, there is almost no positional deviation between the receiver 20a and the video camera 20b, and it is possible to match the image of the match with the position of the obtained ID.

FIG. 5 shows an example in which the video output of the image synthesizing device 31 is supplied to a display device and displayed. Player names N1 to N5 as related information acquired from the database 32 are displayed near the players P1 to P5 in the video of the game. The players P1 to P5 move along with the player names N1 to N5.
Moves.

In one embodiment, as described with reference to FIG. 1, the transmitter 1 is located at a fixed position such as a corner of a ground.
Since 0 ₇ to 10 ₁₀ are installed, and the receiver 20 also receives blinking light from the transmitters 10 ₇ to 10 ₁₀ , the position of each player with respect to the coordinates of the ground can be obtained. Even when the receiver 20 moves, the position of each player can be detected based on the coordinates of the ground. Therefore, as shown in FIG. 6, the current position of each player is obtained for an image shot from a different angle from the image currently being shot, for example, a bird's-eye view of the ground GN, and the player names N1 to N5 are respectively assigned to the positions.
Can be displayed.

It is also possible to record the movement of an object, for example a player. FIG. 7 shows a schematic of a system for recording movement. The ID and the position data S1 are supplied from the receiver 20 to the coordinate conversion unit 34. Reference ID For example, transmitters 10 ₇ to ₁₀ 10 installed at the corner of the ground
Is stored in the memory 35. The position data of the reference ID is supplied to the coordinate conversion unit 34.
The coordinate conversion unit 4 performs an arithmetic process and outputs the transmitters 10 ₇ to 10 _7.
The coordinate position of the object is obtained for the ₁₀ position data. That is, the position of the object with respect to the coordinates of the light receiving surface of the receiver 20 is converted into the position of the reference ID, for example, the position of the object with respect to the coordinates of the ground.

The obtained coordinate position of the object is recorded in the recording / reproducing device 6. In this case, the current time information is added to the position data and recorded. The recording / reproducing device 6 sequentially records data of coordinate positions on a tape-shaped recording medium, a disk-shaped recording medium, a semiconductor memory, or the like. The recorded coordinate data can be reproduced in order, and the trajectory TR of the movement of one player during the game can be displayed on the display device as shown in FIG. The movement of the player can be analyzed based on the trajectory TR. Position data of all players or specified players is recorded.

Here, a method of coordinate conversion in the coordinate conversion unit 4 will be described. The coordinates of a point on the ground, since in terms of height 0 and _{(x i, y i, 0} ), the coordinates on the image and (X _i, Y _i). At this time, the following relationship (a) is established between the two.

X _i = (a ₁ x _i + a ₂ y _i + a ₃ ) / (a ₇
x _i + a ₈ y _i +1) Y _i = (a ₄ x _i + a ₅ y _i + a ₆ ) / (a ₇ x _i + a ₈ y)
_i +1)

Here, a ₁ , a ₂ ,..., A ₈ are unknown coefficients representing the position, direction, focal length, and the like of the camera. The coefficients can be obtained by solving the following equation if four points whose coordinates on the ground are known in the image.

[0024]

(Equation 1)

If a ₁ , a ₂ ,..., A ₈ are obtained, the coordinates on the ground of another point known to be at the same height as the ground in the image from the relation (a) Is required.

Although not described in detail here, a document (Junichi Komoto, "A method of constructing augmented reality using a two-dimensional matrix code", WISS '96, Kindai Kagakusha,
1996, pp. 199-208), a point at a different height from the ground but with a known height (for example, a transmitter attached to a player's uniform is 1.4 m) On the ground) can also be obtained. If the height fluctuates from the assumed height, the error increases. However, even in this case, the error can be suppressed by using a receiver that captures the athlete in the game from an angle close to directly above the ground. Furthermore, by using a plurality of receivers, the three-dimensional position of the object can be tracked using a stereoscopic method.

The above-mentioned transmitters 10 ₁ to 10 ₁₀ and receiver 2
The digital communication system constituted by 0 is described in more detail. In FIG. 9, reference numerals 10 ₁ , 10
₂ , 10 ₃ and 10 ₄ indicate transmitters, respectively. Although four transmitters are shown, this is an example, and the number of transmitters is arbitrary. In the above description, ten transmitters 1
0 ₁ to 10 ₁₀ are used.

The transmitters 10 ₁ to 10 ₄ have the same configuration as each other. That is, a transmission data storage memory 11 for storing N-bit transmission data, and a light emitting element such as an infrared-emitting LED (Liquid Cryst) according to the content of the transmission data.
al Display) Data processing unit 1 for controlling the blinking operation of 13
2 constitutes a transmitter. The LED 13 is attached to an object, for example, a uniform of a player.

FIG. 10 shows a data frame structure transmitted as a blinking pattern of the LED 13 from the transmitter.
One data frame is composed of N bits of transmission data (payload) and a start bit of bit "1" and a bit "1".
The bit is sandwiched between stop bits of “0.” For example, bit “0” corresponds to turning off the LED 13 and bit “
1 "corresponds to the lighting of the LED 13. The blinking pattern corresponds to the transmission information, for example, the ID of the object.

In FIG. 9, reference numeral 20 indicates a receiver. The receiver 20 includes light receiving elements 22 ₁ , 22 ₂ , 22 ₃ ,
.., 22 _M. Light receiving unit 2
Although not shown, blinking light generated by the LED 13 of the transmitter is incident on the light receiving surface of the light source 1 through a light collecting optical system such as a light collecting lens. The light receiving unit 21 is, for example, a CMOS (Complem
entary Metal Oxide Semiconductor (complementary metal oxide semiconductor). CM
OS image sensors are generally CCD (Charge Coupl
(ed Device: charge transfer element) Since the operation speed is faster than that of the image sensor, the blinking pattern can be read even when the transmitter blinks the LED at a high frequency.

As shown in FIG. 11, the light receiving surface 21a of the light receiving section 21 has a structure in which light receiving elements are arranged in a (n × m = M) two-dimensional matrix. One light receiving element corresponds to one pixel. The light receiving unit 21 not only detects a blinking signal of the LED for each pixel, but also functions as a camera that captures an image formed on the light receiving surface 21a.

As shown in FIG. 9, the light receiving elements 23 ₁ to 23 _M
Detection block 23 ₁ ~ 23 _M are connected to each. The detection block 23 ₁ ~23 _M, flashing signal detected by each light receiving element is detected. Detection block 23
Output signals of ₁ to 23 _M are supplied to the data collection unit 24.
The data collection unit 24 processes the output signal of the detection block in an integrated manner. The data collection unit 24 performs processing of the LED blink signal and processing of the image signal captured by the light receiving unit 21. From the data collection unit 24, received data as a processing result of the blinking signal and a signal of a processing result of the captured image are output.

As shown in FIG. 12, the light receiving section 21 has an operation mode for photographing a scene as a camera (referred to as an image mode) and an operation mode for receiving and processing an optical signal from an LED (referred to as a decode mode). Made alternately. In the decoding mode, for example, sampling of 12 kHz
It repeated 00 times, and decodes the optical signal of the 8-bit carrier frequency e.g. 4kHz to be sent at the transmitter side for all the detection blocks 23 ₁ ~23 _M, 15fps (frame /
Seconds), a received / recognized image of the optical signal can be created. This recognition image is a result of decoding the value of each pixel of the image, and has both transmission data represented by the optical signal and spatial information of the optical signal. When outputting both a scene image acquired in the image mode and a recognized image acquired in the decode mode, the image mode and the decode mode are alternately repeated at 15 fps.

The detection block 23 ₁ ~ 23 _M are shared by the image mode and decoding mode. The detection block includes a bandpass filter that allows only a carrier frequency component of transmission data to pass, a PLL, an A / D converter, a memory for storing imaging data, a binary data memory, and a memory for storing reception data. In the image mode, the light receiving element is exposed for a time corresponding to the shutter speed, and an analog signal corresponding to the exposure amount is output. This analog signal is A / D converted to T-bit digital data and stored in the image data storage memory.

In the decode mode, when the received data is N bits long, T-bit digital data obtained by A / D converting the output of the band-pass filter is binarized by threshold processing, and It is temporarily stored in the digitized data memory. The binarized data indicates that the light receiving element is LED
Indicates whether or not an optical signal has been received.

When the light receiving element receives an optical signal from the LED, the light receiving element is exposed for a certain time corresponding to the sampling frequency after waiting for a certain time corresponding to the carrier frequency. Then, an analog signal corresponding to the exposure amount is subjected to an A / D converter via a band-pass filter,
The T-bit digital data is temporarily stored in the imaging data storage memory. Then, the T-bit digital data is binarized by threshold processing, and the binarized data is stored in a predetermined bit position of the received data storage memory (N bits). Next, the same operation is repeated by incrementing the address of the memory, and the N-bit received data is stored.

The data collection unit 24 reads out N-bit received data from each of the (n × m = M) detection blocks, checks if there is an error in the received data if necessary, and stores the received data. Store in the memory for When storing N-bit received data, the value of the received data and the address of the light receiving element that outputs the received data are stored as a set. For all the light receiving elements, the received data and the information (address) of the position of the light receiving element are stored as a set for the light receiving elements that have received valid data. Therefore, the output read from the data collection unit 24 is also a set of N-bit data and address information.

In the communication system described above, the brightness of the LED is increased or the number of the LEDs is increased in accordance with the distance between the transmitter and the receiver. Is taken as a plurality of pixels. As a result, the optical signal caused by the blinking of the LED can be recognized without any problem.

The present invention is not limited to the above-described embodiment of the present invention, and various modifications and applications are possible without departing from the gist of the present invention. For example, the present invention is not limited to sports such as soccer, baseball, and the like, and may use each horse of a horse race as an object or each car of a car race as an object.

[0040]

According to the present invention, since the systems do not need to be synchronized and there is no need to generate a magnetic field, motion tracking applicable to a wide open area can be realized. Therefore, according to the present invention, the position of each object can be accurately grasped when relaying a sport, a horse race, a car race, or the like. In particular, you can accurately follow the position on the captured image,
Information related to an object such as a player name can be superimposed on a relay video and displayed. If the optical output of the transmitter is sufficient, these effects can be exerted not only on telephoto images but also on images with a wide angle of view. Further, the transmitter according to the present invention is very small and lightweight, and does not hinder the movement of an object such as a player.

[Brief description of the drawings]

FIG. 1 is a schematic diagram for explaining an example in which a motion tracking device according to the present invention is applied to follow the motion of each soccer player.

FIG. 2 is a block diagram schematically showing a configuration of an example of a motion tracking device according to the present invention.

FIG. 3 is a block diagram schematically showing a configuration of another example of the motion tracking device according to the present invention.

FIG. 4 is a schematic diagram used for describing a case where a receiver and a video camera are configured separately.

FIG. 5 is a schematic diagram for explaining an example in which related information obtained by the motion tracking device according to the present invention is displayed so as to be superimposed on an image.

FIG. 6 is a schematic diagram for explaining an example of displaying related information obtained by the motion tracking device according to the present invention.

FIG. 7 is a block diagram schematically showing a configuration for recording coordinate information of an object obtained by the motion tracking device according to the present invention.

FIG. 8 is a schematic diagram showing an image in which coordinate information of an object obtained by the motion tracking device according to the present invention is reproduced and displayed.

FIG. 9 is a block diagram of a communication system used in the present invention.

FIG. 10 is a schematic diagram illustrating an example of a data configuration transmitted in a communication system.

FIG. 11 is a schematic diagram illustrating a configuration of a light receiving surface used in a communication system.

FIG. 12 is a schematic diagram for explaining the operation of an image sensor in a communication system.

[Explanation of symbols]

P1 to P5: players, 10 ₁ to 10 _10: transmitters added to each object, 20: receivers, 31
..Image synthesizing devices, 32 ... databases, N1 to N
5 ... player name, 34 ... coordinate conversion unit, 36 ... recording and reproducing device

Continuation of front page    (72) Inventor Nobuyuki Matsushita             3-14-13 Higashi Gotanda, Shinagawa-ku, Tokyo             Formula company Sony Computer Science Research             Inside (72) Inventor Junichi Koimoto             3-14-13 Higashi Gotanda, Shinagawa-ku, Tokyo             Formula company Sony Computer Science Research             Inside (72) Inventor Shigeru Tajima             3-14-13 Higashi Gotanda, Shinagawa-ku, Tokyo             Formula company Sony Computer Science Research             Inside (72) Inventor, Michimune Kono             3-14-13 Higashi Gotanda, Shinagawa-ku, Tokyo             Formula company Sony Computer Science Research             Inside F-term (reference) 5C022 AB62 AB63 AB65 AC13                 5C054 CA04 CA05 CH01 DA01 EH01                       FE14 GB06 GD01 HA31

Claims (12)

[Claims] 1. A tracking device that tracks the movement of a plurality of objects, wherein the transmitter is configured to transmit, as an optical signal, transmission data added to each of the plurality of objects and capable of identifying the plurality of objects; A receiver that receives an optical signal from a transmitter on a two-dimensional light receiving surface and obtains position data of a light receiving position of the optical signal on the two-dimensional light receiving surface for each of the objects; and a captured image of a plurality of the objects And a synthesizing unit that superimposes information related to the object at a position indicated by the position data in the captured image. 2. The motion tracking device according to claim 1, wherein the captured image is obtained by the two-dimensional light receiving surface of the receiver. 3. The motion tracking device according to claim 1, wherein the transmitter converts the transmission data into an optical signal having a blinking pattern of light or a change in light intensity. 4. A tracking device that tracks the movement of a plurality of objects, wherein each of the plurality of objects and a fixed position that defines an area in which the object moves are added to enable identification of the plurality of objects and the fixed position. A transmitter for transmitting transmission data as an optical signal; an optical signal from a plurality of the transmitters being received by a two-dimensional light receiving surface; and a position data of a light receiving position of the optical signal on the two-dimensional light receiving surface for each of the objects. A motion tracking device, comprising: a receiver for each fixed position; and a conversion unit for obtaining position data of the object in an area where the object moves based on the information on the fixed position. 5. The motion tracking device according to claim 4, wherein the captured image is obtained by the two-dimensional light receiving surface of the receiver. 6. The motion tracking device according to claim 4, wherein the transmitter converts the transmission data into an optical signal having a blinking pattern of light or a change in light intensity. 7. A tracking device that tracks the movement of a plurality of objects, wherein the transmitter is configured to transmit, as an optical signal, transmission data added to each of the plurality of objects and capable of identifying the plurality of objects; A receiver that receives an optical signal from a transmitter on a two-dimensional light receiving surface and obtains position data of a light receiving position of the optical signal on the two-dimensional light receiving surface for each of the objects; and records the position data of the object. A motion tracking device comprising a recording unit. 8. The motion tracking device according to claim 7, wherein the captured image is obtained by the two-dimensional light receiving surface of the receiver. 9. The motion tracking device according to claim 7, wherein the transmitter converts the transmission data into an optical signal having a blinking pattern of light or a change in light intensity. 10. A tracking method for tracking the movement of a plurality of objects, comprising: transmitting, as an optical signal, transmission data that enables the plurality of objects to be identified by a transmitter added to each of the plurality of objects. Receiving a light signal from the plurality of transmitters on a two-dimensional light receiving surface and obtaining position data of a light receiving position of the light signal on the two-dimensional light receiving surface for each of the objects; A motion tracking method, comprising: a step of acquiring a captured image; and a combining step of superimposing information related to the object on a position indicated by the position data in the captured image. 11. A tracking method for following the movement of a plurality of objects, wherein the plurality of objects and the fixed positions are determined by a transmitter added to a plurality of objects and a fixed position defining an area in which the objects move. A transmitting step of transmitting transmission data to be identifiable as an optical signal; optical signals from the plurality of transmitters being received by a two-dimensional light receiving surface; and position data of a light receiving position of the optical signal on the two-dimensional light receiving surface. And a conversion step of obtaining position data of the object in an area where the object moves based on the information of the fixed position. 12. A tracking method for following the movement of a plurality of objects, wherein a transmission step of transmitting transmission data added to each of the plurality of objects and enabling identification of the plurality of objects as an optical signal; A receiving step of receiving an optical signal from a transmitter on a two-dimensional light receiving surface and obtaining position data of a light receiving position of the optical signal on the two-dimensional light receiving surface for each of the objects; and recording the position data of the object. A motion tracking method comprising a recording step.