JP2000020193A - Virtual object control method, virtual object control device, and recording medium - Google Patents

Abstract

(57) [Object] To provide a virtual object control device capable of controlling an object in a virtual space created in a computer by a more realistic operation. An image acquisition unit that acquires an image of a recognition target, an image processing unit that performs image processing for extracting information about the position, movement, and direction of the recognition target from the image acquired by the image acquisition unit; Control means for controlling a control target in a virtual space corresponding to the recognition target based on the information extracted by the image processing means.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a virtual object control for controlling the movement, shape, etc. of a CG model (virtual object) in a virtual space based on information such as the movement and position of a recognition target extracted from a range image. About the method.

[0002] The present invention also provides a method for generating a CG image in a virtual space based on information such as the movement and position of a recognition target extracted from a range image.
The present invention relates to a virtual object control device that controls the movement, shape, and the like of a model (virtual object).

[0003]

2. Description of the Related Art Hitherto, in a playground equipment such as an arcade game machine, many games utilizing human movements such as a boxing game and a shooting game have been developed. In a boxing game, the pressure when the user actually hits the sandbag is physically measured. At this time, only the pressure at the moment when the user's punch hits the sandbag is measured. No information is used as to what route and how fast the punch has been fed from the user. For this reason, it has become a monotonous game that competes only for the strength of a simple punch.

In a shooting game, a light emitter is attached to the tip of a toy gun, and the light of the light emitter is detected on the screen. When the trigger of the gun is fired, it is determined that the object hit by the light is hit by the bullet. Because of this simplicity, rather than competing for the shooting skills, the game has become a game of reflexes that only finds the presented target and shoots it.

[0005] On the other hand, there is a method of recognizing a position, a movement, and the like of a human body by analyzing an image captured by using a CCD camera or the like. In this method, the traveling direction is first photographed using an imaging device such as a CCD camera. In the case of extracting a human from the image, for example, since the color of the face and hands is a flesh color, a pre-processing is performed in which unnecessary parts such as other backgrounds are removed and only an object to be recognized such as an obstacle is cut out. . Then, the shape, movement, and the like of the target object are recognized from the processed image.

[0006] First, a pre-processing portion of extracting the recognition target will be described. In a conventional method, as an operation means for cutting out only a part of an object to be acquired from an image captured by a camera, the object is cut out based on some difference between the object and another part.

As a clue, a method using a change in hue, a method using a difference image, and the like are used. In the case of cutting out using a color, a portion having a large hue difference is extracted, a process such as thinning is performed, and an edge is extracted.
When targeting a human, attention is paid to the skin color of the face and hand parts, and only the hue part is extracted.
However, the method of using the hue difference has a problem that the skin color changes depending on the color and angle of the illumination, and that if the hue of the background is close to the skin color, it is difficult to distinguish it from human skin color. It is difficult to cut out regularly. In addition, in the state where there is no illumination, the captured image becomes entirely dark, and it is difficult for a human to identify an object from a photograph taken in the dark.

There is also a method of calculating a motion vector between frames of a video image and analyzing a moving object. In this case, there is no problem as long as the number of moving objects is small, but there are many moving objects. And the number of motion vectors suddenly increases, the load of calculating the motion vector between frames increases,
Calculation cannot keep up.

[0009] In a game for home use, an input device called a joystick having a force feedback function or a vibration pack having a vibration function has been used in order to increase a sense of realism.

The shape of the joystick is similar to a lever that controls an airplane or the like. For this reason, a game called a flight simulator that simulates the operation of an airplane certainly has an impression. However, in other operations, there is a problem that the usability is different from the actual operation.

Similarly, if the user is used to the vibration pack, he / she can operate a car racing game or the like that simulates driving a car without a steering wheel. However, there is a problem that the operational feeling is greatly different from the actual operation.

[0012] For example, children use various toys such as robots and airplanes to fight each other, or use robots to create and destroy building towns while using real objects. Make a story and play. In the past, it was not possible to fuse such a real thing with a virtual environment created in a computer.

[0013]

As described above, conventionally,
An object in a virtual space existing in a computer could only be controlled by wearing special wear and equipment or using a special device such as a joystick. That is, it has not been possible to integrate a real object and a virtual space without discomfort.

Accordingly, an object of the present invention is to provide a virtual object control method capable of controlling an object in a virtual space created in a computer by a more realistic operation. Also,
An object of the present invention is to provide a virtual object control device that can control an object in a virtual space created in a computer by a more realistic operation method.

[0015]

According to a virtual object control method of the present invention, information relating to the position, movement and direction of a recognition target is extracted from a captured image of the recognition target, and the recognition is performed based on the extracted information. By controlling the control target in the virtual space corresponding to the target, the object in the virtual space created in the computer can be controlled by a more realistic operation.

A virtual object control device according to the present invention comprises: an image acquiring means for acquiring an image of a recognition target; and an image processing for extracting information relating to the position, movement and direction of the recognition target from the image acquired by the image acquiring means. Image processing means for performing, and control means for controlling a control target in a virtual space corresponding to the recognition target based on information extracted by the image processing means,
Is provided, the object in the virtual space created in the computer can be controlled by a more realistic operation.

[0017]

Embodiments of the present invention will be described below with reference to the drawings. (First Embodiment) FIG. 1 schematically shows a configuration of a virtual object control device according to a first embodiment.

As shown in FIG. 1, the virtual object control apparatus according to the present embodiment receives a reflected light and obtains a distance image, for example, a distance image obtaining unit described in Japanese Patent Application No. 9-299648. 1. Analyze the acquired distance image and extract the target information related to the recognition target, such as extraction of the outline and center of gravity of the recognition target, calculation of the distance to the target, calculation of the moving speed of the recognition target, and calculation of the movement vector. An image processing unit 4
The shape of a three-dimensional or two-dimensional virtual space created by CG (computer graphics) and the shape of a three-dimensional or two-dimensional virtual object (for example, a person, playground equipment, etc., hereinafter referred to as a CG model) presented in the virtual space And a virtual space control unit 3 for controlling the shape and movement of the CG model stored in the shape storage unit 2 based on the analysis result of the image processing unit 4. For example, a virtual space as a three-dimensional or two-dimensional CG image is visualized by a display device such as a large liquid crystal panel, and the control target stored in the shape storage unit 2 under the control of the virtual space control unit 3 And a presentation unit 5 for visualizing the shape, movement, and the like of the CG model.

Here, the image acquisition unit 1 and the distance image acquired by the image acquisition unit 1 will be briefly described. FIG. 2 shows the appearance of the image acquisition unit 1. A light receiving unit 103 composed of a circular lens and an area sensor (not shown) at the rear is arranged in the center, and an LED that irradiates light such as infrared rays along the contour around the circular lens. A plurality of (eg, six) light emitting units 101 are arranged at equal intervals.

The light emitted from the light emitting unit 101 is reflected by the object, collected by the lens of the light receiving unit 103, and received by the area sensor at the rear of the lens. The area sensor is, for example, a sensor arranged in a 256 × 256 matrix, and the intensity of the reflected light received by each sensor in the matrix becomes a pixel value. The image acquired in this manner is a distance image as an intensity distribution of reflected light as shown in FIG.

FIG. 3 shows an example of the configuration of the image acquiring unit 1, which mainly comprises a light emitting unit 102, a light receiving unit 103, a reflected light extracting unit 102, and a timing signal generating unit 104. The light emitting section 101 emits light whose intensity varies with time according to the timing signal generated by the timing signal generating section 104. This light is applied to a target object located in front of the light emitting unit.

The light receiving unit 103 detects the amount of light emitted from the light emitting unit 101 and reflected by the target object. The reflected light extraction unit 102 extracts a spatial intensity distribution of the reflected light received by the light receiving unit 103. Since the spatial intensity distribution of the reflected light can be grasped as an image, it is hereinafter referred to as a distance image.

The light receiving section 103 generally receives not only reflected light of the object emitted from the light emitting section 101 but also external light such as illumination light and sunlight. Therefore, the reflected light extracting unit 102 calculates the difference between the amount of light received when the light emitting unit 101 is emitting light and the amount of light received when the light emitting unit 101 is not emitting light, thereby obtaining the light emitting unit 101. Only the reflected light component of the light from the target object is extracted.

The reflected light extraction unit 102 extracts the intensity distribution of the reflected light received by the light receiving unit 103, that is, a distance image as shown in FIG. In FIG. 4, for simplicity, 8 × 8 which is a part of the distance image of 256 × 256 pixels
The case of a distance image of 8 pixels is shown.

The reflected light from the object decreases significantly as the distance of the object increases. When the surface of the object scatters light uniformly, the amount of light received per pixel of the distance image decreases in inverse proportion to the square of the distance to the object.

In FIG. 4, the values (pixel values) of the cells in the matrix indicate the intensity of the acquired reflected light in 256 gradations (8 bits). For example, a cell having a value of “255” is in a state in which the cell is closest to the distance image acquisition unit 1,
A cell having a value of “0” is far from the distance image acquisition unit 1, indicating that the reflected light does not reach the distance image acquisition unit 1.

Each pixel value of the distance image represents the amount of reflected light received by the unit light receiving unit corresponding to the pixel. The reflected light is
The nature of the object (specular reflection, scattering, absorption,
), Object direction, object distance, etc.
When the entire object is an object that scatters light uniformly, the amount of reflected light has a close relationship with the distance to the object. Since a hand or the like has such a property, the distance image when the hand is extended in front of the image acquisition unit 1 reflects the distance to the hand, the inclination of the hand (partially different distance), and the like. A three-dimensional image as shown in FIG. 5 can be obtained.

The image processing unit 4 extracts edges (extracts the outline of the imaged object), extracts the center of gravity, calculates the area, and calculates the distance to the imaged object from the distance image data as shown in FIG. And various kinds of image processing for extracting information such as the shape, motion, and position of the imaging body, such as calculation of motion vectors and motion vectors.

The image acquiring unit 1 captures an image of a user playing a game with a sword 21 as a playground equipment, as shown in FIG. 7, for example. The image acquisition unit 1 acquires a playground equipment as a recognition target, that is, a distance image of a sword (see FIG. 11A), and the image processing unit 4 analyzes the distance image.

The image processing unit 4 extracts the contour, the center of gravity, the three-dimensional position, direction, speed, etc. of the sword from the distance image and passes them to the virtual space control unit 3. The virtual space control unit 3 displays, for example, a virtual space created by three-dimensional or two-dimensional CG (computer graphics) stored in the shape storage unit 2 as shown in FIG. Based on the information extracted in step 4, the user controls the shapes and movements of the CG model 23 of the user and the CG model 22 of the sword stored in the shape storage unit 2 in real time.
It is presented to the presentation unit 5. Thus, the user can play the game as if he / she entered the virtual space.

The image processing unit 4 first obtains the intensity of the reflected light due to the difference in the reflection coefficient of the imaged object and the positional relationship between the recognition target and the other imaged objects as shown in FIG. Extract the outline information of the recognition target in the range image.

For example, if the image acquisition unit 1 is arranged at a position where the user holding the sword can be imaged from the front, the image acquisition unit 1 recognizes the distance image as shown in FIG. The playground equipment as a target, that is, the sword, is located closest to the image acquisition unit 1 than other imaging objects. That is, as the distance from the image acquisition unit 1 is shorter, the reflected light acquired by the image acquisition unit 1 is stronger, so that the pixel value increases (the larger the pixel value, the closer to black). The closest part of the sword tip 31 is the blackest, the part 32 closer to the sword handle, the hand 33 of the user holding the sword gradually decreases in pixel value (white), and the background is farthest away. The value is almost “0”.

The distance value from the image acquisition unit 1 to the object to be imaged is not limited to such four gradations, but can be measured at 256 gradations or more precisely. In calculating the distance d to the object whose contour has been extracted, first, a representative pixel value of an image near the position of the center of gravity of the object is determined. Although there are several representative pixel values such as an average value and a nearest neighbor value, it is assumed here that the nearest neighbor value is used. The intensity of the light reflected from the object decreases in inverse proportion to the square of the distance to the object. That is, the pixel value is P
Assuming that (i, j), the relationship between the pixel value and the distance value w can be expressed as P (i, j) = K / w ² (1). Here, K is determined for each imaging object. For example, when w = 0.5 m, K is adjusted so that the value of the pixel value P (i, j) of the imaging object becomes “255”. Coefficient, which is also called a reflection coefficient. By solving equation (1) for w, the distance value w can be obtained.

Assuming that the reflection coefficient of the sword to be recognized is larger than that of the other image pickup body (for example, even when the sword is behind the hand holding the sword, its pixel value is larger than the pixel value of the image of the hand. Is also large). Then, masking is performed on the distance image using the distance value. Now, if only the image area with the largest value (that is, the closest part) among the four gradations is extracted, only the tip 31 of the sword can be cut out in the case of FIG. This corresponds to setting the pixel value to the lowest value (for example, “0”) except for a pixel having a value equal to or more than a certain fixed value.

In order to extract the contour information, the pixel values of adjacent pixels are compared, and a constant value is entered only when the pixel value is equal to or more than a constant value α, and the continuous image region to which the same constant value is assigned is assigned. Pixels may be extracted.

That is, assuming that the pixel value at the coordinate position (i, j) on the matrix in FIG. 4 is P (i, j) and the pixel value of the outline information is Q (i, j): i, j) -P (i-1, j)}> α, and {P
(I, j) -P (i, j-1)}> α, and {P (i,
j) -P (i + 1, j)}> α, and {P (i, j)-
When P (i, j + 1)｝> α, Q (i, j) = 255. In other cases, by setting Q (i, j) = 0, the contour information of the imaging object as shown in FIG. Obtainable.

As described above, FIG. 11A to FIG.
As shown in (b), the outline information of the tip 31 of the sword can be cut out. The image processing unit 4 calculates the center of gravity Rg of the tip 31 of the sword from the contour information of the tip 31 of the sword.

Further, as shown in FIG. 11B, by obtaining a normal vector of a plane including the position of the center of gravity, the sword direction D is obtained.
g can be calculated. The normal vector is a straight line passing through the center of gravity Rg and perpendicular to a tangent plane to a plane including the center of gravity Rg.

It is possible to determine whether the object is moving by comparing the center of gravity of the sword to be recognized with the center of gravity of the sword to be recognized and extracted from the immediately preceding frame. When it is determined that the subject is moving, a motion vector representing the amount of change in the center of gravity and the direction of change is calculated.

Assuming that the position of the center of gravity of one frame before is Rg ', the moving speed Vg of the center of gravity Rg is represented by (R
g−Rg ′), and if a distance image of 30 frames is acquired in one second, the distance image can be obtained from Vg = (Rg−Rg ′) × 30 (2)

In this manner, the shape and direction of the recognition target, the center of gravity of the recognition target,
Information such as the distance to the center of gravity and the movement can be extracted. Next, referring to the flowchart shown in FIG.
The processing operation of the virtual object control device will be described.

First, when the virtual space control unit 3 is started by turning on the power or instructing the start of the operation, the virtual space control unit 3
The 3D or 2D CG virtual space stored in the display unit 5 is displayed. The CG model (corresponding to the user in this case) of the person stored in the shape storage unit 2 is set in the virtual space displayed on the presentation unit 5 with its center of gravity position r, direction d, and velocity v as initial values. Present within.
The CG model of the playground equipment (corresponding to the sword held by the user in this case) is displayed on the presentation unit 5 with the values of the center of gravity position R, the direction D, and the speed V as initial values. It is presented in the space (step S1).

Thereafter, the image acquiring unit 1 uses the light emitting unit 101 and the light receiving unit 103 as shown in FIG.
A distance image is acquired at a speed of about 30 sheets per second (step S2). In general, perform interactive processing with the user,
That is, the upper limit of the response speed at which the user can recognize that the other side of the computer or the like is responding to his / her action is 0.2 seconds. It is possible for the user to return a natural response.

Next, the image processing unit 4 uses the distance image acquired by the image acquiring unit 1, for example, the distance image data as shown in FIG. Based on the coefficient and the distance value, contour information of the sword (more specifically, the tip of the sword (see FIG. 11A)) 31 is extracted (step S3), and the center of gravity is extracted from the extracted sword contour information. The position Rg, the direction Dg of the sword, and the speed of movement Vg of the position of the center of gravity are calculated (step S4).

The virtual space control unit 3 calculates the sword's center of gravity position Rg, the sword's direction Dg, and the speed of movement of the sword's center of gravity Vg extracted from the distance image in the CG corresponding to the sword in the virtual space.
Model center of gravity position R, direction D, velocity of movement of center of gravity position V
Corresponds to This association converts coordinates, units, and sizes so that the center of gravity position R, the direction D, and the velocity V of the movement of the center of gravity position in the real space extracted from the distance image fit in the virtual space. (Step S5).

Further, the virtual space control unit 3 calculates the center of gravity position R, the direction D, and the speed of movement of the center of gravity of the user's CG model from the center of gravity R, the direction D, and the speed of movement of the center of gravity of the user's CG model. v is calculated (step S6).

For speeding up, the calculation is simplified, and the position of the center of gravity of the CG model of the user is calculated by the following method. FIG. 12 is a diagram for explaining a method of calculating the center of gravity position p and the direction d of the CG model of the person (user) from the center of gravity P and the direction D of the CG model of the playground equipment (sword), and is a virtual space. The control unit 3 refers to, for example, a table storing calculation criteria as shown in FIG. 13 stored in a predetermined memory, and obtains the CG model of the user from the center of gravity position P and the direction D of the CG model of the sword. The center of gravity position p and the direction d are calculated.

The angle between the direction D of the CG model of the sword and the xz plane including the center of gravity R of the CG model of the sword is θ, and the direction D of the sword is
Is the angle between the yz plane containing the center of gravity R of the sword CG model and φ
And Here, φ is conceptually defined as positive when the direction D of the normal vector of the sword is on the right side of the yz plane and negative when it is on the left side.

According to the calculation criteria shown in FIG.
Is smaller (ie, when the sword itself is almost vertical), and when θ is larger than α (ie, when
When the sword itself is nearly horizontal), the estimation method is largely divided into two. In each case, when the sword is on the right side of the yz plane (φ is positive), the sword is y
The judgment is made separately depending on whether it is on the left side of the z plane (φ is negative).

When the sword is nearly vertical, the user bends his wrist and approaches the body to hold the sword. When the sword is nearly horizontal, the user stretches his wrist and lifts his / her body away from the body. Holding a sword.

According to the calculation standard shown in FIG. 13, by estimating the form when the user's CG model picks up the sword from the direction D of the sword CG model, the center of gravity R and the direction D of the sword CG model are obtained. The position of the center of gravity r and the direction (direction of the center of gravity) d of the CG model of the user are calculated as follows.

When the sword is almost vertical and facing right (φ
Is positive), the center of gravity r of the user's CG model
(X3, y3, z3) is the center of gravity position R of the CG model of the sword.
For (x1, y1, z1), X3 = x1-β, y3 = y1-γ, and Z3 = z1 + δ. β, γ, and δ are constants, β is a value proportional to the size of the hand of the user's CG model, and γ is almost the sword C.
Is the size corresponding to the length of the G model, and δ is the size corresponding to １ ／ of the arm length of the CG model of the user.

When the sword is almost vertical and faces to the left (φ
Is negative), the position of the center of gravity r of the CG model of the user
In the x component, left and right are reversed by adding instead of subtracting β. That is, X3 = x1 + β, y3 = y1-γ, and Z3 = z1 + δ.

When the direction of the CG model of the user is substantially perpendicular to the sword, the body is considered to be substantially parallel to the sword. Therefore, the direction D (x2, y2, z2) of the sword is directly changed to the CG of the user. Used as the direction d (X4, Y4, Z4) of the model.

X4 = x2, y4 = y2, Z4 = z2 When the sword is almost horizontal and points to the right (when φ is positive)
, The center of gravity r (x3, y3,
z3) is X3 = x1-ε, Y3 = y1-ζ, and Z3 = z1 + η with respect to the center of gravity position R (x1, y1, z1) of the sword. ε, ζ, and η are constants, and ε is a value proportional to the size of the hand of the CG model of the user, and is a value larger than β. ζ is a value corresponding to the width of the sword, which is considerably smaller than γ. η is a size corresponding to ／ of the arm length of the CG model of the user.

When the sword is almost horizontal and faces to the left (φ
Is negative), the position of the center of gravity r of the CG model of the user
Is added instead of subtracting ε in the x component. That is, X3 = x1 + ε, y3 = y1-ζ, and Z3 = z1 + η.

The direction d of the user's CG model is such that when the sword is substantially horizontal, the body is considered to be rotated by approximately 90 degrees in the vertical plane with respect to the sword. Then, the direction d (X
4, Y4, Z4). That is, X4 = x2, y4 = y2, Z4 = y2 The speed v of the movement of the CG model of the user may be a predetermined value corresponding to the speed V of the movement of the CG model of the playground equipment.

In the shape storage unit 2, as shown in FIG.
The CG model of the person, the CG model of the playground equipment, and the shape of the character in the virtual space are stored. In FIG. 14, for example, data such as the shape of an actual three-dimensional CG model of an object called a person CG model exists after the memory address indicated by the pointer “Pointer1”. The bounding boxes indicate the coordinates of the six vertices of a rectangular parallelepiped circumscribing the three-dimensional CG model. The bounding box checks whether objects in the virtual space are in collision with each other (interference check). If any of the six vertices of this bounding box is inside the other party's bounding box, it is regarded as interfering.

In step S 6, the center of gravity r and the direction d of the user's CG model are obtained from the center of gravity R and the direction D of the CG model of the sword. Read the shape data of the CG model. Then, the position of the center of gravity of the shape data is presented in the virtual space already presented on the presentation unit 5 so that the position becomes r, the direction becomes d, and the speed of movement v becomes (step S7). FIG. 8 shows an example of the display screen of the presenting unit 5 at this time.

The virtual space control unit 3 presents the user's CG model and, at the same time, compares the bounding box stored in the shape storage unit as shown in FIG. An interference check with a certain other object is performed (step S8). if,
If there is interference, it means that the user's CG model has hit another object, so that the presentation unit 5 generates an impact sound (step S15 in FIG. 10). The impact sound may be a specific sound or a different sound for each object. In addition, due to the collision, the user's CG model is rebounded by the collision object. The amount of rebound and the direction of rebound when interfering with the CG model of the user for each of various characters existing in the virtual space are determined in advance according to the position of the interference,
Calculate the degree of rebound from the center of gravity position r of the current user's CG model and its direction d, and calculate the user's CG model.
The center position p and the direction d of the model are updated (step S1).
6).

The center of gravity position p and the direction d of the user's CG model
Is updated, the center of gravity position P and the direction D of the CG model of the sword are calculated again (step S17). If the user's CG model is interfering with the object, after performing the processing of steps S15 to S17, the process returns to step S7, performs an interference check in step S8, and makes a check with the object in another virtual space. When it is determined that there is no interference, the process proceeds to step S9 in FIG.

In step S9, the virtual space control unit 3
The shape data of the CG model of the sword is read from the shape storage unit 2. Then, the center of gravity of the shape data is presented in the virtual space already presented on the presentation unit 5 such that the position is R, the direction is D, and the speed of movement is V (step S9).

The virtual space control unit 3 presents the CG model of the sword and, at the same time, compares the bounding box stored in the shape storage unit as shown in FIG. An interference check with a certain other object is performed (step S10). If there is interference, it means that the CG model of the sword has hit another object, so that a plosive sound corresponding to, for example, the speed V is generated from the presentation unit 5 (step S11). Then, for example, when the object is hard and the sword is rebounded, the position and direction of the center of gravity of the sword change, so the calculation is performed (step S12), and the CG model of the sword is displayed again based on the result. Fix (step S13). Here, if there is a termination instruction, the processing is terminated (step S14).

As described above, according to the first embodiment, the recognition target (for example,
The image processing unit 4 extracts the shape and movement of the recognition target from the captured distance image of the sword as a play tool held by the user, and corresponds to the recognition target based on the extracted shape and movement. By controlling the position, movement, shape, etc. of the control target (CG model of the user and the sword) in the virtual space, the user can wear special clothing and equipment, or intervene with cables or the like extending from the main body. Without anything, you can control the objects in the virtual space created in the computer with more realistic operations. In other words, it is possible to operate realistic objects in the virtual space as if they are in the virtual space, perform realistic simulations, and enjoy the game (game).

When the virtual object control device according to the above-described embodiment is applied to an arcade game, a trader who installs such an arcade game does not need special wear, equipment, cables, and the like. Since it is free from consideration of wasted time when users attach and detach special wear, hygiene considerations such as wear worn by multiple users, consideration of equipment wear due to user movement, etc. The cost for operating the game machine can be greatly reduced.

In the first embodiment, the number of the image acquisition units 1 is one, but the number is not limited to one. A plurality of cameras having the configuration shown in FIG. 3 for acquiring a distance image are installed, and each of the distance images acquired by each camera is subjected to the same image processing as described above. The results may be integrated to calculate the position of the center of gravity of the recognition target in the range image, the direction of the center of gravity, the moving speed, and the like.

Also, by using a plurality of CCD cameras, three-dimensional information (the position of the center of gravity of the object to be recognized, the direction of the center of gravity, the moving speed, etc.) can be obtained from the image of the object to be recognized in the image as in the above embodiment. It is also possible to extract. It is undeniable that the image processing speed in this case is lower than the image processing speed for the distance image. (Second Embodiment) FIG. 15 schematically shows a configuration of a virtual object control device according to a second embodiment of the present invention. In FIG. 15, the same parts as those in FIG. 1 are denoted by the same reference numerals, and different parts will be described. In the first embodiment, when the user moves his / her hand or body, the CG model of the user or the CG model of the sword presented in the screen follows and moves, thereby feeling a sense of presence. . With the impact sound and the popping sound emitted when hitting another object in the virtual space, he felt the collision.

On the other hand, in the second embodiment, by further providing the feedback generating unit 6 and the feedback presenting unit 7, the CG model of the user or the CG model of the sword interferes with another object in the virtual space. Presentation part 5
In addition to generating an impulsive sound and a popping sound in the above, for example, by generating and presenting force (force) feedback, the user can experience a more realistic sense of reality.

The feedback generation unit 6 receives the user's CG
When the CG model of the model or the sword interferes with another object in the virtual space, the magnitude of the force feedback force is calculated.

The feedback presenting unit 7 is set in a playground equipment (in this case, a sword) held by the user, and is provided to the motor in accordance with, for example, the magnitude of the force feedback calculated by the feedback generating unit 6. By generating a rotational torque, an actual force sense is presented. As the motor, for example, a DC motor is used.

The feedback generator 6 calculates the degree of impact S from the weight M of the sword as a playground equipment held by the user in the hand and the moving speed Vg of the center of gravity of the sword extracted from the image.

Impact S = KMV ² (K is a proportional constant) Based on this impact S, the output (current value) of the DC motor constituting the feedback presenting unit 7 embedded in the playground equipment is controlled. When the motor suddenly rotates, a torque in the opposite direction is generated.
Presents force feedback.

The control information may be transmitted to the feedback presenting unit 7 (for example, a DC motor) embedded in the playground equipment by wire communication or IrDA (Infr.
Wireless communication such as ared Data Association may be used.

As described above, according to the second embodiment, the object in the virtual space created in the computer can be controlled by more realistic operations, and the user can control the impact force according to the scene. Because there is feedback, it is possible to have a more realistic experience.

In the configuration of the virtual object control device shown in FIGS. 1 and 15, the processing operation of each component except the image acquisition unit 1 is a program that can be executed by a computer as a floppy disk, hard disk, CD-RO.
It can also be stored and distributed on a recording medium such as M, DVD, or semiconductor memory.

In the first and second embodiments, the case where the virtual object control device of the present invention is applied to a game has been described as an example. However, the scope of the present invention is not limited to this case. Needless to say, the present invention can be applied to practical systems for a wide range of industrial fields, such as dangerous work in which failure is unacceptable, and education and training such as medical simulation.

Further, in the first and second embodiments, the case where the recognition target in the image is a playground equipment (sword) held by the user is described as an example. However, the present invention is not limited to this case. The target may be the user himself, and the control of the virtual object in the virtual space based on the information such as the position, the movement, and the shape extracted from the image of the user can be performed in the same manner as described above.

[0078]

As described above, according to the present invention,
Objects in the virtual space created in the computer can be controlled with more realistic operations.

[Brief description of the drawings]

FIG. 1 is a diagram schematically showing a configuration of a virtual object control device according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of an appearance of a distance image acquisition unit.

FIG. 3 is a diagram illustrating a configuration example of a distance image acquisition unit.

FIG. 4 is a diagram illustrating an example of a distance image in which the intensity of reflected light is a pixel value.

FIG. 5 is a diagram showing a three-dimensional matrix image of a range image as shown in FIG. 4;

FIG. 6 is a diagram showing an example of an outline image of an object extracted from a distance image.

FIG. 7 is a diagram illustrating an example of an imaging state of an image acquisition unit that captures an image of a user playing a game with a sword as a playground equipment.

FIG. 8 is a view showing an example of a virtual space displayed on a display screen of a presentation unit.

FIG. 9 is a flowchart for explaining a processing operation of the virtual object control device in FIG. 1;

FIG. 10 is a flowchart for explaining a processing operation of the virtual object control device in FIG. 1;

11A and 11B are diagrams illustrating an example of a distance image acquired by an image acquiring unit. FIG. 11A illustrates a method of extracting a sword image based on a positional relationship between a recognition target sword and a hand. (B) is a diagram for explaining the normal vector of the sword extracted from the distance image.

FIG. 12 is a diagram for explaining a method of calculating a center of gravity position p and a direction d of a CG model of a person (user) from a center of gravity P and a direction D of a CG model of a playground equipment (sword).

FIG. 13 is a diagram illustrating an example of a table storing calculation criteria for calculating a center of gravity p and a direction d of a CG model of a user from a center of gravity P and a direction D of a CG model of a sword.

FIG. 14 is a diagram illustrating a storage example of data of a virtual object in a shape storage unit;

FIG. 15 is a diagram schematically illustrating a configuration example of a virtual object control device according to a second embodiment of the present invention.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 image acquisition unit 2 shape storage unit 3 virtual space control unit 4 image processing unit 5 presentation unit 6 feedback generation unit 7 feedback presentation unit

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Naoko Umeki 1st in Toshiba Research and Development Center, Komukai Toshiba, Saiwai-ku, Kawasaki City, Kanagawa Prefecture No. 1, Toshiba R & D Center (72) Inventor Yasunori Yamauchi 1 Tokoba Toshiba, Koyuki-ku, Kawasaki-shi, Kanagawa Prefecture Toshiba R & D Center (72) Inventor Isao Mihara, Kawasaki-shi, Kanagawa 1F, Komukai Toshiba-cho, Toshiba R & D Center, F-term (reference) 2C001 BB00 BB10 BC01 BC03 CA00 CA08 CA09 5B050 BA07 BA09 EA05 EA06 EA07 EA08 EA18 EA24 EA28 FA02 FA10 5B057 AA20 BA02 BA13 CA13 CA17 CB13 CB DA16 DB03 DB09 DC06 DC16 DC23

Claims (5)

[Claims] An information on a position, a movement, and a direction of a recognition target is extracted from a captured image of a recognition target, and a control target in a virtual space corresponding to the recognition target is controlled based on the extracted information. A virtual object control method characterized by the following. 2. The virtual object control method according to claim 1, wherein the image is a distance image indicating a spatial intensity distribution of light reflected from the object. 3. An image acquiring means for acquiring an image of a recognition target, and an image processing means for performing image processing for extracting information relating to the position, movement and direction of the recognition target from the image acquired by the image acquiring means. A control unit for controlling a control target in a virtual space corresponding to the recognition target based on the information extracted by the image processing unit. 4. The virtual object control device according to claim 3, wherein the image is a distance image indicating a spatial intensity distribution of light reflected from the object. 5. An image processing means for extracting information relating to the position, movement, and direction of the recognition target from a captured image of the recognition target, and a virtual space corresponding to the recognition target based on the information extracted by the image processing means. And a machine readable recording medium storing a program for executing the control means for controlling a control target in the computer.