JP2006318094A - Information processing method and information processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve operability of a virtual object by checking an interference state between the virtual object and a virtual object in proper timing. <P>SOLUTION: Moving velocity of an indicator is found (S404), and it is decided whether to check the interference state between the operation target virtual object and the fixedly disposed virtual object or not on the basis of the found moving velocity (S405). When interference is generated as the result of the check on the interference state between the operation target virtual object and the fixedly disposed virtual object (S408), the effect of the generation of the interference is notified (S409). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for generating an image of a virtual space.

  The mixed reality system provides a user with a synthesized video obtained by synthesizing a real space video and a virtual space video generated according to the user's viewpoint position, line-of-sight direction, and the like. In the mixed reality system, it is possible to present the viewer with the sensation as if a virtual object actually exists in the real space, more realistic than the conventional virtual reality system (VR system), Observation with an actual size sense becomes possible.

  In order to operate a CG object in the mixed reality space, this 6-degree-of-freedom sensor is used by using a system (referred to herein as a 6-degree-of-freedom sensor) that measures or estimates the 6 degrees of freedom of the position and orientation in the real space. Is associated with the object coordinate system of the CG object. For example, when using the Poltramus Fastrak system, it is possible to measure 6 degrees of freedom of the position and posture of a receiver having a size of several centimeters. By associating this measurement value with the object coordinate system of the CG object, when the receiver is moved / rotated, the CG object is moved / rotated following it.

  By using this receiver as an object operation device, it becomes possible to operate the CG object by the object operation device.

By the way, the mixed reality system is a technology that makes it appear as if a CG object actually exists in the real space. Therefore, for example, when considering the work of arranging a small box in a large box without a lid, a small box CG object (hereinafter referred to as CG object 1) associated with the object manipulation device and the object manipulation device are associated. Measure physical interference with a large CG object (hereinafter referred to as CG object 2) in real time. This is possible by using a method of presenting a force sense that moves the object operation device.
JP 2005-38008 A

  However, in this case, the user is interested in the interference between the inside of the CG object 2 and the CG object 1. In the conventional method, it is necessary to bring the CG object 1 closer from the upper side of the CG object 2 in order to avoid unnecessary interference.

  Since the CG object 1 and the CG object 2 are both CG and are not real objects, physical interference does not occur in the real space. Therefore, if desired, it is desired to move the CG object 1 into the CG object 2 by punching out the wall of the CG object 2 and then check the interference between the CG object 1 and the CG object 2. Then, the user is not forced to perform an unnecessary operation of bringing the CG object 1 closer to the CG object 2 from above.

  In addition, the process of performing the interference check between CG objects requires a relatively complicated process. In some cases, the interference check is performed, so that the display speed of the video presented to the user through the video observation apparatus decreases. It may end up. As a result, the operation itself of arranging the CG object 1 in the CG object 2 becomes difficult.

  The present invention has been made in view of the above problems, and an object of the present invention is to improve the operability of a virtual object by checking the interference state between virtual objects at a suitable timing.

  In order to achieve the object of the present invention, for example, an information processing method of the present invention comprises the following arrangement.

That is, a first acquisition step of acquiring the position and orientation of the observer's viewpoint;
A second acquisition step of acquiring the position and orientation of the pointing tool held by the observer in hand;
An arrangement step of arranging the first virtual object in the virtual space with the position and orientation acquired in the second acquisition step;
An information processing method comprising: a generation step of generating an image that is visible when the virtual space including the first virtual object is viewed from a viewpoint having the position and orientation acquired in the first acquisition step;
A calculation step for obtaining a moving speed of the indicator;
A determination step of determining whether or not to check an interference state between the first virtual object and the second virtual object arranged in the virtual space, based on the moving speed of the pointing tool;
As a result of checking the interference state between the first virtual object and the second virtual object, if the first virtual object and the second virtual object interfere with each other, the fact is notified. And a notification step.

  In order to achieve the object of the present invention, for example, an information processing method of the present invention comprises the following arrangement.

That is, a first acquisition step of acquiring the position and orientation of the observer's viewpoint;
A second acquisition step of acquiring the position and orientation of the pointing tool held by the observer in hand;
An arrangement step of arranging the first virtual object in the virtual space with the position and orientation acquired in the second acquisition step;
An information processing method comprising: a generation step of generating an image that is visible when the virtual space including the first virtual object is viewed from a viewpoint having the position and orientation acquired in the first acquisition step;
Arranged in the first virtual object and the virtual space based on the moving direction of the pointing tool and the positional relationship between the first virtual object and the second virtual object arranged in the virtual space A determining step for determining whether or not to check the interference state with the second virtual object being performed;
As a result of checking the interference state between the first virtual object and the second virtual object, if the first virtual object and the second virtual object interfere with each other, the fact is notified. And a notification step.

  In order to achieve the object of the present invention, for example, an information processing method of the present invention comprises the following arrangement.

That is, a first acquisition step of acquiring the position and orientation of the observer's viewpoint;
A second acquisition step of acquiring the position and orientation of the pointing tool held by the observer in hand;
An arrangement step of arranging the first virtual object in the virtual space with the position and orientation acquired in the second acquisition step;
An information processing method comprising: a generation step of generating an image that is visible when the virtual space including the first virtual object is viewed from a viewpoint having the position and orientation acquired in the first acquisition step;
An estimation step for estimating a load of processing for checking an interference state between the first virtual object and the second virtual object arranged in the virtual space;
A determination step for determining whether to perform the check according to the load estimated in the estimation step;
As a result of checking the interference state between the first virtual object and the second virtual object, if the first virtual object and the second virtual object interfere with each other, the fact is notified. And a notification step.

  In order to achieve the object of the present invention, for example, an information processing apparatus of the present invention comprises the following arrangement.

That is, first acquisition means for acquiring the position and orientation of the viewpoint of the observer;
Second acquisition means for acquiring the position and orientation of the pointing tool held by the observer in his hand;
Arrangement means for arranging the first virtual object in the virtual space with the position and orientation acquired by the second acquisition means;
An information processing apparatus comprising: a generation unit configured to generate an image that is visible when a virtual space including the first virtual object is viewed from a viewpoint having the position and orientation acquired by the first acquisition unit;
A calculation means for obtaining a moving speed of the indicator;
Determining means for determining whether or not to check an interference state between the first virtual object and the second virtual object arranged in the virtual space, based on a moving speed of the pointing tool;
As a result of checking the interference state between the first virtual object and the second virtual object, if the first virtual object and the second virtual object interfere with each other, the fact is notified. And an informing means.

  In order to achieve the object of the present invention, for example, an information processing apparatus of the present invention comprises the following arrangement.

That is, first acquisition means for acquiring the position and orientation of the viewpoint of the observer;
Second acquisition means for acquiring the position and orientation of the pointing tool held by the observer in his hand;
Arrangement means for arranging the first virtual object in the virtual space with the position and orientation acquired by the second acquisition means;
An information processing apparatus comprising: a generation unit configured to generate an image that is visible when a virtual space including the first virtual object is viewed from a viewpoint having the position and orientation acquired by the first acquisition unit;
Arranged in the first virtual object and the virtual space based on the moving direction of the pointing tool and the positional relationship between the first virtual object and the second virtual object arranged in the virtual space Determining means for determining whether or not to check the interference state with the second virtual object being performed;
As a result of checking the interference state between the first virtual object and the second virtual object, if the first virtual object and the second virtual object interfere with each other, the fact is notified. And an informing means.

  In order to achieve the object of the present invention, for example, an information processing apparatus of the present invention comprises the following arrangement.

That is, first acquisition means for acquiring the position and orientation of the viewpoint of the observer;
Second acquisition means for acquiring the position and orientation of the pointing tool held by the observer in his hand;
Arrangement means for arranging the first virtual object in the virtual space with the position and orientation acquired by the second acquisition means;
An information processing apparatus comprising: a generation unit configured to generate an image that is visible when a virtual space including the first virtual object is viewed from a viewpoint having the position and orientation acquired by the first acquisition unit;
Estimating means for estimating a load of processing for checking an interference state between the first virtual object and the second virtual object arranged in the virtual space;
Determining means for determining whether to perform the check according to the load estimated by the estimating means;
As a result of checking the interference state between the first virtual object and the second virtual object, if the first virtual object and the second virtual object interfere with each other, the fact is notified. And an informing means.

  In order to achieve the object of the present invention, for example, an information processing method of the present invention comprises the following arrangement.

That is, an acquisition step of acquiring the position of the pointing tool operated by the observer;
An arrangement step of arranging the first virtual object according to the position of the pointing tool;
An information processing method comprising: a notification step of determining an interference state between the first virtual object and a second virtual object different from the first virtual object, and performing notification according to the determination result. ,
The notification step is controlled according to the moving speed of the pointing tool.

  With the configuration of the present invention, the operability of the virtual object can be improved by checking the interference state between the virtual objects at a suitable timing.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment]
FIG. 1 is a diagram illustrating an appearance of a system for providing an observer (user) with a mixed reality space in which a virtual space is superimposed on a real space.

  In the figure, reference numeral 200 denotes a transmitter which generates a magnetic field. Reference numeral 100 denotes a head-mounted display device (hereinafter referred to as HMD: Head) that is attached to the observer's head and provides an image of a space (mixed reality space) obtained by combining the real space and the virtual space in front of the observer's eyes. The camera 102a and 102b, the display devices 101a and 101b, and the magnetic receiver 201 are referred to as “Mounted Display”.

  The cameras 102a and 102b each capture a moving image of a real space that can be seen from the right and left eyes of an observer wearing the HMD 100 on the head, and the captured images of each frame are output to the computer 400 at the subsequent stage.

  The display devices 101a and 101b are each mounted on the HMD 100 so that the observer is positioned in front of the right eye and the left eye when the HMD 100 is mounted on the head. Display the based image. Accordingly, an image generated by the computer 400 is provided in front of the viewer's right eye and left eye.

  The magnetic receiver 201 detects a change in the magnetic field generated by the transmitter 200 and outputs a detection result signal to the subsequent position / orientation measurement device 205. The detected signal is a magnetic signal in a coordinate system (hereinafter referred to as a sensor coordinate system) in which the position of the transmitter 200 is an origin, and three axes orthogonal to each other at the position of the origin are x, y, and z axes. It is a signal indicating a change in magnetic field detected according to the position and orientation of the receiver 201.

  Based on this signal, the position / orientation measurement apparatus 205 obtains the position / orientation of the magnetic receiver 201 in the sensor coordinate system, and data indicating the obtained position / orientation is output to the computer 400 at the subsequent stage.

  FIG. 2 is a diagram illustrating a specific configuration of the HMD 100.

  Reference numeral 101 denotes a video display device which is composed of a small liquid crystal display device of about 0.5 to several inches. Reference numeral 103 denotes a free-form surface prism that serves as a lens for enlarging the image of the image display device 101. With such a configuration, the video displayed on the video display device 101 is presented to the observer as a video equivalent to 90 inches, for example, 2 m ahead.

  Reference numeral 102 denotes a video input device, which includes an imaging device such as a CCD camera or a CMOS camera. An imaging system prism 104 serves as a lens for converging light in the real space to the video input device 102. The imaging system prism 104 is arranged outside the free-form surface prism 103 so that the optical axes coincide with each other, thereby eliminating the parallax between the video input by the video input device 102 and the video displayed on the video display device 101, and in real space. Can be reproduced without a sense of incongruity.

  Returning to FIG. 1, reference numeral 300 denotes an indicator held in the observer's hand, which is a magnetic receiver itself or a magnetic receiver incorporated therein. In the following, for the sake of simplicity, it is assumed that the indicator 300 is a magnetic receiver itself. However, even if the magnetic receiver is incorporated in the indicator 300, the essence of the following description does not change.

  Like the magnetic receiver 201, the pointing device 300 (magnetic receiver) detects a change in the magnetic field according to its position and orientation in the sensor coordinate system, and indicates a signal indicating the detection result (in other words, the pointing device 300 in the sensor coordinate system). A signal indicating its own position / posture) is output to the position / posture measuring apparatus 205. Accordingly, the position / orientation measuring apparatus 205 can obtain the position / orientation of the pointing tool 300 in the sensor coordinate system based on this signal, and data indicating the obtained position / orientation is output to the computer 400 at the subsequent stage. Therefore, even if the observer holds the pointing device 300 in his / her hand and changes its position and posture, the position / posture of the pointing device 300 is always measured by the pointing device 300 itself, and the measured position / posture data is stored in the computer 400. Entered.

  301 is a stand.

  A computer 400 generates an image signal to be output to the display devices 101a and 101b of the HMD 100, receives data from the position / orientation measurement device 205, and performs processing such as management. This computer is generally composed of, for example, a PC (personal computer), WS (workstation), or the like. FIG. 8 is a block diagram illustrating a hardware configuration of the computer 400.

  A CPU 2201 controls the entire computer 400 using programs and data stored in the RAM 2202 and the ROM 2203 and controls data communication with an external device connected to the I / F 2207. Also, each process described later performed by the computer 400 is executed.

  Reference numeral 2202 denotes a RAM, an area for temporarily storing programs and data loaded from the external storage device 2205, an area for temporarily storing data received via the I / F 2207, and the CPU 2201 Various areas such as a work area necessary for executing this process can be provided as appropriate.

  A ROM 2203 stores a boot program, setting data of the computer 400, and the like.

  An operation unit 2204 includes a keyboard, a mouse, a joystick, and the like, and various instructions can be input to the CPU 2201 when operated by an operator of the computer 400.

  Reference numeral 2205 denotes an external storage device that functions as a large-capacity information storage device such as a hard disk drive device. Programs and data for causing the CPU 2201 to execute each process described below performed by the OS (Operating System) and the computer 400 here. Are stored in the RAM 2202 under the control of the CPU 2201. In addition, what is described as known data (information) in the following description (or data that should be necessary for the processing described below) is also stored in the external storage device 2205, and is controlled by the CPU 2201 as necessary. Thus, the data is loaded into the RAM 2202.

  A display unit 2206 includes a CRT, a liquid crystal screen, and the like, and can display a processing result by the CPU 2201 using an image, text, or the like.

  Reference numeral 2207 denotes an I / F, to which the position / orientation measuring apparatus 205, the HMD 100, and the like are connected. The computer 400 performs data communication with the position / orientation measuring apparatus 205, the HMD 100, and the like via the I / F 2207. Can do.

  Reference numeral 2208 denotes a bus connecting the above-described units.

  The computer 400 having the above configuration captures real space images obtained from the cameras 102a and 102b, and generates virtual object images that can be seen from the cameras 102a and 102b based on the position and orientation obtained from the magnetic receiver 201. . Then, the generated image is superimposed on the previously captured real space image, and the superimposed image is output to the display devices 101a and 101b. As a result, a mixed reality space image corresponding to the position and orientation of each eye is displayed in front of the right and left eyes of the observer wearing the HMD 100 on the head.

  FIG. 3 is a block diagram showing a functional configuration of the computer 400.

  Reference numerals 401R and 401L denote video capture units, which capture images input from the cameras 102a and 102b, respectively, as data.

  Reference numeral 404 denotes a position / orientation information input unit that captures data output from the position / orientation measurement apparatus 205. This data includes data indicating the position and orientation of the magnetic receiver 201 and the pointing tool 300 in the sensor coordinate system.

  Reference numeral 406 denotes 3DCG drawing data, which is data for generating an image of a virtual object constituting the virtual space. In this embodiment, since the virtual object is composed of polygons, the 3DCG drawing data includes polygon normal vector data, coordinate value data of each vertex constituting the polygon, polygon color data, and texture mapping. When performing, texture image data and the like are included. Note that what constitutes a virtual object is not limited to a polygon.

  Reference numeral 405 denotes a position / orientation calculation unit, which obtains the position / orientation of the cameras 102a and 102b in the sensor coordinate system using data indicating the position / orientation in the sensor coordinate system of the magnetic receiver 201 input from the position / orientation information input unit 404. If the position and orientation relationship between the magnetic receiver 201 and the cameras 102a and 102b is obtained in advance as a bias, the position and orientation of the cameras 102a and 102b can be obtained by adding this bias to the position and orientation measured by the magnetic receiver 201. ). Hereinafter, the cameras 102a and 102b may be collectively referred to as “viewpoint”.

  A CG rendering unit 407 generates a virtual space image that can be seen according to the position and orientation of the cameras 102a and 102b. In this virtual space, a virtual object to be superimposed on the pointing device 300 (hereinafter referred to as “operation target virtual object”) and one or more virtual objects (hereinafter referred to as “fixed placement virtual”) fixedly placed in the virtual space. (Referred to as "object").

  Accordingly, the CG rendering unit 407 arranges the operation target virtual object in the virtual space with the position and orientation of the pointing tool 300 in the sensor coordinate system, and places the fixed placement virtual object in the virtual space with the predetermined position and orientation. An image (virtual space image) that is visible when the virtual space where the respective virtual objects are arranged is viewed according to the position and orientation of the cameras 102a and 102b calculated by the position and orientation calculation unit 405 is generated.

  Note that processing for generating an image of a virtual space that can be seen from a viewpoint having a predetermined position and orientation is a well-known technique, and thus detailed description thereof will be omitted.

  Reference numerals 402R and 402L denote video composition units, which are generated by the CG rendering unit 407 on the real space images input from the video capture units 401R and 401L, respectively, and “virtual space images that can be seen according to the position and orientation of the camera 102a” and “ The “virtual space images that can be seen according to the position and orientation of the camera 102b” are superimposed and output to the video generation units 403R and 403L, respectively. Accordingly, it is possible to generate a mixed reality space image that can be seen according to the position and orientation of the camera 102a, and a mixed reality space image that can be seen according to the position and orientation of the camera 102b.

  The video generation units 403R and 403L output the mixed reality space images output from the video synthesis units 402R and 402L to the display devices 101a and 101b, respectively, as video signals. Thereby, an image of the mixed reality space corresponding to each eye is displayed in front of the right eye and the left eye of the observer wearing the HMD 100 on the head.

  Reference numeral 408 denotes a CG object speed measuring unit, which refers to “position information of the pointing tool 300” sequentially obtained from the position / orientation information input unit 404 and obtains the current moving speed of the pointing tool 300. There are various methods for obtaining the moving speed. For example, the position information obtained from the position / orientation calculation unit 405 at time t and the position information obtained from the position / orientation calculation unit 405 at time (t−1). May be obtained as a moving speed of the pointing device 300 at time t.

  Reference numeral 409 denotes an interference check unit that performs a process of checking an interference state between the operation target virtual object and the fixedly arranged virtual object (whether or not they collide with each other). Such a check process is not always performed, and is determined by the moving speed of the pointing tool 300 obtained by the CG object speed measuring unit 408.

  That is, when the moving speed of the pointing device 300 is equal to or higher than a predetermined value, it is considered that the observer is moving the pointing device 300 to a desired position. At this time, the interference state check process is performed. In doing so, the observer moves the pointing tool 300 while paying attention to the interference between the operation target virtual object and the fixed-placement virtual object, so that the movement of the pointing tool 300 becomes troublesome. Therefore, in such a case, the interference state check process is not performed.

  On the other hand, when the moving speed of the pointing tool 300 is equal to or less than a predetermined value, it is considered that the observer moves the pointing tool 300 to a vicinity of a desired position and finely adjusts the position and orientation of the operation target virtual object. . For example, when the operation target virtual object is arranged at a predetermined position and orientation in the fixed arrangement virtual object, after the operation target virtual object is moved into the fixed arrangement virtual object, the operation target virtual object should be in a desired position and orientation. The observer finely moves the pointing tool 300 to adjust the position and orientation of the operation target virtual object. If the interference state check process is not performed here, the observer cannot know whether the operation target virtual object is properly contained in the fixedly arranged virtual object. In such a case, an interference state check process is performed.

  Therefore, the CG object speed measurement unit 408 notifies the interference check unit 409 every time the moving speed of the pointing tool 300 is obtained. The interference check unit 409 performs interference check processing if the notified speed is equal to or lower than a predetermined value, and conversely does not perform interference check processing if the notified speed is equal to or higher than the predetermined value.

  In addition, the process of checking the interference state between virtual objects is performed, for example, by checking whether or not each polygon constituting each virtual object collides. As described above, the technology for checking the interference state between virtual objects and identifying which part is interfering with each other is widely used in the field of video games. Therefore, a detailed description thereof will be omitted.

  When the interference check unit 409 determines that the operation target virtual object and the fixed placement virtual object are interfering with each other, the interference check unit 409 notifies the CG rendering unit 407 of each interference portion (polygon). Change the drawing of the interference part differently from the other parts so that each interference part is visually easy to understand, such as changing the polygon color of each interference part of the virtual object to be manipulated and the virtual object to be fixed or blinking To.

  FIG. 4 is a flowchart of a series of processing until a mixed reality space image for one frame is generated and output to the display devices 101a and 101b. Note that programs and data for causing the CPU 2201 of the computer 400 to execute the processing according to the flowchart of FIG. 10 are stored in the external storage device 2205, which is loaded into the RAM 2202 according to the control of the CPU 2201, and is used by the CPU 2201. Thus, the computer 400 executes each process described below.

  First, since an image of the real space captured by the cameras 102a and 102b is input into the computer 400 via the I / F 2207, the CPU 2201 stores it in the RAM 2202 (step S401).

  On the other hand, data indicating the position and orientation of the magnetic receiver 201 in the sensor coordinate system and data indicating the position and orientation of the pointing device 300 in the sensor coordinate system are stored in the computer 400 via the I / F 2207 from the position and orientation measurement apparatus 205. Since these are input, the CPU 2201 stores them in the RAM 2202 or the external storage device 2205 (step S402).

  In addition, since each process in the said step S401, S402 is performed independently, it does not limit to performing the process in each step in this order.

  Next, the CPU 2201 obtains the position and orientation of the cameras 102a and 102b in the sensor coordinate system by adding the bias to the position and orientation of the magnetic receiver 201 in the sensor coordinate system. The indicated data is temporarily stored in the RAM 2202 (step S403).

  Next, the CPU 2201 obtains the current moving speed of the pointing tool 300 using the position information of the pointing tool 300 acquired in step S402 and the position information of the pointing tool 300 acquired in the past (step S404). The method for obtaining is not particularly limited as described above.

  Then, it is determined whether or not the obtained moving speed is equal to or less than a predetermined value (step S405). If it is equal to or less than the predetermined value, the process proceeds to step S406, and the interference state between the operation target virtual object and the fixedly arranged virtual object Is checked (step S406).

  Then, after the process in step S406, or when the moving speed of the pointing tool 300 is equal to or higher than the predetermined value, the process proceeds to step S407, and the operation target virtual object and the fixed arrangement virtual object are arranged in the virtual space (step S407). . That is, the operation target virtual object is arranged in the virtual space with the position and orientation in the sensor coordinate system of the pointing device 300 acquired in step S402, and the fixed arrangement virtual object is arranged in the virtual space with the predetermined position and orientation. To do. The arrangement position / posture of the fixed arrangement virtual object is not limited to being fixed, and may be dynamically changed.

  Next, it is determined whether or not interference has been determined in step S406 (step S408). If it is determined that interference has occurred, the process proceeds to step S409 to notify the interference portion. (Step S409). In this embodiment, as a notification method, the interference part is notified by display, such as drawing the interference part in red or blinking.

  Then, an image (virtual space image) that is seen when the virtual space in which each virtual object is arranged is viewed from the viewpoint having the position and orientation obtained in step S403 is generated on the RAM 2202 (step S410). That is, an image of the virtual space that can be seen from the position and orientation of the cameras 102a and 102b is generated.

  Here, since the real space image obtained from the camera 102a and the real space image obtained from the camera 102b in step S401 are already stored on the RAM 2202, the virtual space image and camera viewed from the camera 102a are stored on the respective images. By generating an image of the virtual space viewed from 102b, a mixed reality space image viewed from the camera 102a and a mixed reality space image viewed from the camera 102b are generated on the RAM 2202 as a result.

  Therefore, the CPU 2201 outputs each mixed reality space image to the display devices 101a and 101b via the I / F 2207 (step S411).

  As described above, according to the present embodiment, when the observer moves the operation target virtual object to a desired position and orientation, this check process is performed when moving, that is, when it is not desired to check the interference state with the fixedly arranged virtual object. If the position / orientation of the operation target virtual object is adjusted, that is, if you want to check the interference state with the fixedly arranged virtual object, this check process is performed. An operation for moving the operation target virtual object to a desired position and orientation in the fixedly arranged virtual object can be easily performed.

  In this embodiment, when virtual objects interfere with each other, the display form of the interference part of each virtual object is made different from the other parts, but there are various notification methods. Can be considered. For example, if virtual objects interfere with each other, the fact that the interference has occurred may be notified by voice, or the fact that the interference has occurred may be displayed as a text sentence on the display devices 101a and 101b. In the case of notifying interference by sound, it is necessary to add to the computer 400 a sound data, a converter that converts the sound data into an analog signal, and a speaker that outputs sound based on the analog signal.

  As another notification method, a force sense presentation device may be used to present a force sense so as to move the pointing tool 300 in a direction that does not interfere.

[Second Embodiment]
In the first embodiment, only “the moving speed of the pointing tool 300” is used as a criterion for determining whether or not to check the interference state between virtual objects. In the present embodiment, “the moving direction of the pointing tool 300” and “the positional relationship between the operation target virtual object and the fixed placement virtual object” are used as the determination criteria.

  The system configuration according to this embodiment is the same as that of the first embodiment, but the functional configuration of the computer 400 is different from that of the first embodiment.

  FIG. 5 is a block diagram illustrating a functional configuration of the computer 400 according to the present embodiment. Note that the same reference numerals in FIG. 3 denote the same parts as in FIG. 3, and a description thereof will be omitted. The configuration shown in FIG. 5 is obtained by adding a CG object positional relationship measuring unit 410 to the configuration shown in FIG. The points other than those described below are the same as those in the first embodiment.

  The CG object speed measuring unit 408 obtains the moving speed of the pointing device 300 as in the first embodiment, but here, paying attention to the moving direction.

  Further, the CG object positional relationship measuring unit 410 obtains the positional relationship between the operation target virtual object and the fixed placement virtual object.

  FIG. 6 is a flowchart of a series of processing until a mixed reality space image for one frame is generated and output to the display devices 101a and 101b. Note that programs and data for causing the CPU 2201 of the computer 400 to execute the processing according to the flowchart of FIG. 10 are stored in the external storage device 2205, which is loaded into the RAM 2202 according to the control of the CPU 2201, and is used by the CPU 2201. Thus, the computer 400 executes each process described below.

  The processing in steps S601 to S604 is the same as that in steps S401 to S404, but the moving speed obtained in step S604 is obtained as a speed vector. For example, for example, a difference between position information obtained from the position / orientation calculation unit 405 at time t and position information obtained from the position / orientation calculation unit 405 at time (t−1), that is, a three-dimensional vector is obtained as a velocity vector. .

  Next, the CPU 2201 functions as the CG object positional relationship measuring unit 410 and determines the positional relationship between the operation target virtual object and the fixed placement virtual object (step S605). In the present embodiment, it is determined whether or not the operation target virtual object is located inside the fixed placement virtual object. Since the process for determining whether one virtual object is positioned on the other virtual object is well known, a description thereof will be omitted.

  If the operation target virtual object is located inside the fixed arrangement virtual object, the process proceeds to step S607, and the interference state check process between the operation target virtual object and the fixed arrangement virtual object is performed (step S406). S607). That is, when the operation target virtual object is arranged at a predetermined position and orientation in the fixed arrangement virtual object, after the operation target virtual object is moved into the fixed arrangement virtual object, the operation target virtual object has the desired position and orientation. As much as possible, the observer moves the pointing tool 300 finely to adjust the position and orientation of the operation target virtual object. Therefore, if the interference state check process is not performed, the observer will properly place the operation target virtual object in the fixedly arranged virtual object. I do n’t know if it ’s in place. In such a case, an interference state check process is performed. Then, when the process in step S607 is completed, the process proceeds to step S608.

  On the other hand, if the operation target virtual object is located outside the fixed arrangement virtual object, the process proceeds to step S606, and it is determined whether or not the operation target virtual object is moving toward the inside of the fixed arrangement virtual object. Therefore, the three-dimensional vector (vector A indicating the moving direction of the pointing tool 300) obtained in step S604 and the vector B heading from the position of the operation target virtual object (position of the pointing tool 300) to the placement position of the fixed placement virtual object. Is calculated, and it is determined whether or not the calculated inner product value is positive (step S606).

  Here, when the operation target virtual object has not moved toward the inside of the fixed-placement virtual object, the inner product value indicates a negative value. In such a case, the process proceeds to step S607 and interference occurs. Check the status.

  On the other hand, when the operation target virtual object is moving toward the inside of the fixed placement virtual object, the inner product value indicates a positive value. In such a case, the interference state check process in step S607 is performed. Without it, the process proceeds to step S608. That is, when the operation target virtual object is placed at a predetermined position and orientation in the fixed placement virtual object, the pointing tool 300 is located outside the fixed placement virtual object, and the pointing tool 300 faces the fixed placement virtual object. The pointing device 300 is considered to be in the process of being moved to a desired position, and at this point, when the interference state check process is performed, the observer can operate the operation target virtual object and the fixed arrangement virtual object. Since the pointing tool 300 is moved while considering the interference with the object, the movement of the pointing tool 300 becomes troublesome. Therefore, in such a case, the interference state check process is not performed.

  Then, the processing in each step after step S608 is performed in the same manner as in each step after S407.

  In this embodiment, as an example of the process of determining the positional relationship between the operation target virtual object and the fixed placement virtual object, it is determined whether or not the operation target virtual object is located inside the fixed placement virtual object. More generally, an embodiment may be adopted in which it is determined whether or not “the positional relationship between the operation target virtual object and the fixed placement virtual object satisfies a predetermined positional relationship”.

[Third Embodiment]
In the first embodiment, only “the moving speed of the pointing tool 300” is used as a criterion for determining whether or not to check the interference state between virtual objects. In the present embodiment, “estimated load given to the computer 400 for performing the interference state check process” is used as the determination criterion.

  The system configuration according to this embodiment is the same as that of the first embodiment, but the functional configuration of the computer 400 is different from that of the first embodiment.

  FIG. 7 is a block diagram showing a functional configuration of the computer 400 according to the present embodiment. Note that the same reference numerals in FIG. 3 denote the same parts as in FIG. 3, and a description thereof will be omitted. The configuration shown in FIG. 7 is obtained by replacing the CG object speed measurement unit 408 with the interference check load prediction unit 411 in the configuration shown in FIG. The points other than those described below are the same as those in the first embodiment.

  The interference check load prediction unit 411 performs a process of estimating the processing load when the CPU 2201 performs a check process of an interference state between the operation target virtual object and the fixed placement virtual object. When this processing load is large, for example, the image generation processing in the mixed reality space is affected, and the image drawing rate and frame rate are reduced.

  Therefore, the interference check load prediction unit 411 estimates the processing load, and when the estimated processing load is equal to or greater than a predetermined value, that is, processing that affects other processing (including mixed reality space image generation processing). If the load is greater than or equal to the load, this is notified to the interference check unit 409. The interference check unit 409 does not perform the interference state check process when the estimated processing load is equal to or greater than the predetermined value, and conversely performs the interference state check process when the estimated processing load is equal to or smaller than the predetermined value. Since there is no effect on the other processes, an interference state check process is performed.

  Note that the flowchart of a series of processing from the generation of the mixed reality space image for one frame to the output to the display devices 101a and 101b according to the present embodiment is as follows in the flowchart of FIG. Become.

  That is, in step S404, the CPU 2201 estimates the processing load of the check process of the interference state between the operation target virtual object and the fixed placement virtual object. In step S405, it is determined whether or not the estimated processing load is equal to or greater than a predetermined value. If the estimated processing load is equal to or greater than the predetermined value, the process proceeds to step S407. Proceed to

[Fourth Embodiment]
In each of the above embodiments, the interference state check process is not performed depending on a predetermined state. However, the interference state check process is not completely performed. If there is a simple check method, it is used. You may do it.

  In each of the above embodiments, the video see-through type HMD 100 is used, but an optical see-through type may be used.

  Further, the “index used for determining whether or not to perform interference state check processing” used in each of the above embodiments may be used in appropriate combination.

  In each of the above embodiments, the position and orientation of the observer's viewpoint is acquired using a magnetic sensor. However, other types of sensors such as an optical sensor and an ultrasonic sensor are used. It goes without saying that it is also good.

  Further, the means for obtaining the position and orientation of the viewpoint is not limited to using a sensor. For example, a marker is arranged in the real space, and the arrangement position and the marker coordinates on the image captured by the camera are used. A general method of acquiring the position and orientation of the HMD 100 using the position may be used. Therefore, as long as the position and orientation of the viewpoint can be obtained, the use of the sensor is not limited.

[Other Embodiments]
Also, an object of the present invention is to supply a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and the computer (or CPU or Needless to say, this can also be achieved when the MPU) reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a figure which shows the external appearance of the system for providing the observer (user) with the mixed reality space which superimposed the virtual space on the real space. It is a figure which shows the specific structure of HMD100. 2 is a block diagram showing a functional configuration of a computer 400. FIG. It is a flowchart of a series of processes until a mixed reality space image for one frame is generated and output to the display devices 101a and 101b. It is a block diagram which shows the function structure of the computer 400 which concerns on the 2nd Embodiment of this invention. It is a flowchart of a series of processes until a mixed reality space image for one frame is generated and output to the display devices 101a and 101b. It is a block diagram which shows the function structure of the computer 400 which concerns on the 3rd Embodiment of this invention. 2 is a block diagram showing a hardware configuration of a computer 400. FIG.

Claims (18)

A first acquisition step of acquiring the position and orientation of the observer's viewpoint;
A second acquisition step of acquiring the position and orientation of the pointing tool held by the observer in hand;
An arrangement step of arranging the first virtual object in the virtual space with the position and orientation acquired in the second acquisition step;
An information processing method comprising: a generation step of generating an image that is visible when the virtual space including the first virtual object is viewed from a viewpoint having the position and orientation acquired in the first acquisition step;
A calculation step for obtaining a moving speed of the indicator;
A determination step of determining whether or not to check an interference state between the first virtual object and the second virtual object arranged in the virtual space, based on the moving speed of the pointing tool;
As a result of checking the interference state between the first virtual object and the second virtual object, if the first virtual object and the second virtual object interfere with each other, the fact is notified. An information processing method comprising: a notification step.   In the calculation step, a difference vector between the position acquired in the second acquisition step at time t and the position acquired in the second acquisition step at time (t−1) is obtained, and the magnitude of the obtained difference vector is calculated. The information processing method according to claim 1, wherein the information is obtained as a moving speed of the pointing tool at time t.   In the determining step, when the moving speed of the pointing tool is equal to or lower than a predetermined value, it is determined that the interference state between the first virtual object and the second virtual object is checked. The information processing method according to claim 1.   In the determining step, when the moving speed of the pointing tool is equal to or higher than a predetermined value, it is determined that the interference state between the first virtual object and the second virtual object is not checked. The information processing method according to claim 1. A first acquisition step of acquiring the position and orientation of the observer's viewpoint;
A second acquisition step of acquiring the position and orientation of the pointing tool held by the observer in hand;
An arrangement step of arranging the first virtual object in the virtual space with the position and orientation acquired in the second acquisition step;
An information processing method comprising: a generation step of generating an image that is visible when the virtual space including the first virtual object is viewed from a viewpoint having the position and orientation acquired in the first acquisition step;
Arranged in the first virtual object and the virtual space based on the moving direction of the pointing tool and the positional relationship between the first virtual object and the second virtual object arranged in the virtual space A determining step for determining whether or not to check the interference state with the second virtual object being performed;
As a result of checking the interference state between the first virtual object and the second virtual object, if the first virtual object and the second virtual object interfere with each other, the fact is notified. An information processing method comprising: a notification step.   In the determining step, when the first virtual object and the second virtual object have a predetermined positional relationship, an interference state between the first virtual object and the second virtual object is determined. 6. The information processing method according to claim 5, wherein it is determined to perform a check.   In the determining step, when the first virtual object and the second virtual object are not in a predetermined positional relationship, a vector directed from the position of the pointing tool to the position of the second virtual object; When the inner product value with the moving direction vector of the pointing tool is calculated and the calculated inner product value is negative, the interference state between the first virtual object and the second virtual object is checked. 6. The information processing method according to claim 5, wherein the information processing method is determined.   In the determining step, when the first virtual object and the second virtual object are not in a predetermined positional relationship, a vector directed from the position of the pointing tool to the position of the second virtual object; When the inner product value with the moving direction vector of the pointing tool is calculated and the calculated inner product value is positive, the interference state between the first virtual object and the second virtual object is not checked. The information processing method according to claim 5, wherein the information processing method is determined. A first acquisition step of acquiring the position and orientation of the observer's viewpoint;
A second acquisition step of acquiring the position and orientation of the pointing tool held by the observer in hand;
An arrangement step of arranging the first virtual object in the virtual space with the position and orientation acquired in the second acquisition step;
An information processing method comprising: a generation step of generating an image that is visible when the virtual space including the first virtual object is viewed from a viewpoint having the position and orientation acquired in the first acquisition step;
An estimation step for estimating a load of processing for checking an interference state between the first virtual object and the second virtual object arranged in the virtual space;
A determination step for determining whether to perform the check according to the load estimated in the estimation step;
As a result of checking the interference state between the first virtual object and the second virtual object, if the first virtual object and the second virtual object interfere with each other, the fact is notified. An information processing method comprising: a notification step.   In the determining step, when the load estimated in the estimating step is a predetermined value or more, it is determined that the interference state between the first virtual object and the second virtual object is not checked. The information processing method according to claim 9, wherein:   In the determining step, when the load estimated in the estimating step is equal to or less than a predetermined value, it is determined that the interference state between the first virtual object and the second virtual object is checked. The information processing method according to claim 9. Furthermore,
A third acquisition step of acquiring an image of a real space visible from the viewpoint;
The output step of superimposing the image generated in the generation step on the image acquired in the third acquisition step and then outputting the image to the outside is provided. The information processing method described. First acquisition means for acquiring the position and orientation of the viewpoint of the observer;
Second acquisition means for acquiring the position and orientation of the pointing tool held by the observer in his hand;
Arrangement means for arranging the first virtual object in the virtual space with the position and orientation acquired by the second acquisition means;
An information processing apparatus comprising: a generation unit configured to generate an image that is visible when a virtual space including the first virtual object is viewed from a viewpoint having the position and orientation acquired by the first acquisition unit;
A calculation means for obtaining a moving speed of the indicator;
Determining means for determining whether or not to check an interference state between the first virtual object and the second virtual object arranged in the virtual space, based on a moving speed of the pointing tool;
As a result of checking the interference state between the first virtual object and the second virtual object, if the first virtual object and the second virtual object interfere with each other, the fact is notified. An information processing apparatus comprising: a notification unit. First acquisition means for acquiring the position and orientation of the viewpoint of the observer;
Second acquisition means for acquiring the position and orientation of the pointing tool held by the observer in his hand;
Arrangement means for arranging the first virtual object in the virtual space with the position and orientation acquired by the second acquisition means;
An information processing apparatus comprising: a generation unit configured to generate an image that is visible when a virtual space including the first virtual object is viewed from a viewpoint having the position and orientation acquired by the first acquisition unit;
Arranged in the first virtual object and the virtual space based on the moving direction of the pointing tool and the positional relationship between the first virtual object and the second virtual object arranged in the virtual space Determining means for determining whether or not to check the interference state with the second virtual object being performed;
As a result of checking the interference state between the first virtual object and the second virtual object, if the first virtual object and the second virtual object interfere with each other, the fact is notified. An information processing apparatus comprising: a notification unit. First acquisition means for acquiring the position and orientation of the viewpoint of the observer;
Second acquisition means for acquiring the position and orientation of the pointing tool held by the observer in his hand;
Arrangement means for arranging the first virtual object in the virtual space with the position and orientation acquired by the second acquisition means;
An information processing apparatus comprising: a generation unit configured to generate an image that is visible when the virtual space including the first virtual object is viewed from a viewpoint having the position and orientation acquired by the first acquisition unit;
Estimating means for estimating a load of processing for checking an interference state between the first virtual object and the second virtual object arranged in the virtual space;
Determining means for determining whether to perform the check according to the load estimated by the estimating means;
As a result of checking the interference state between the first virtual object and the second virtual object, if the first virtual object and the second virtual object interfere with each other, the fact is notified. An information processing apparatus comprising: a notification unit.   A program that causes a computer to execute the information processing method according to any one of claims 1 to 12.   A computer-readable storage medium storing the program according to claim 16. An acquisition step of acquiring the position of the pointing tool operated by the observer;
An arrangement step of arranging the first virtual object according to the position of the pointing tool;
An information processing method comprising: a notification step of determining an interference state between the first virtual object and a second virtual object different from the first virtual object, and performing notification according to the determination result. ,
The information processing method characterized by controlling the said alerting | reporting process according to the moving speed of the said indicator.

Images