JP7188900B2 - Information processing system and program - Google Patents

Description

The present invention relates to an information processing system and program.

Presently, instead of virtual reality (VR) or augmented reality (AR), it means a combination of a real space (real space) and a space created virtually using a computer (virtual space). A technology called Mixed Reality (MR) is attracting attention. In a space where mixed reality is realized (mixed reality space), an object in the real space and an object in the virtual space overlap the shape information of the two three-dimensional spaces of the real space and the virtual space, and influence in real time. Matching experiences are possible.
For example, in Patent Document 1, when a real object is positioned behind a virtual object (when the user cannot see the real object), the user is notified in advance of the existence of the real object approaching. technique is described. Specifically, when the distance between the real object and the user is within a predetermined distance, the display of the virtual object positioned in front is controlled to be semi-transparent or outline display, and the real object positioned in the background is displayed. describes a technique that allows for the visibility of objects in

JP 2016-4493 A

On the other hand, in the conventional technology, when a virtual object (virtual object) is positioned behind a real object (real object), only the shape information of the real object is used as a criterion for rendering, so the virtual object is uniformly drawn. A non-rendering method is used. Therefore, even if the real object is transparent, the virtual object disappears from the space as soon as it is hidden behind it. However, such a phenomenon that goes against the laws of nature gives an unnatural impression to the user who is experiencing the mixed reality.

SUMMARY OF THE INVENTION An object of the present invention is to render a virtual object located behind a transparent real object so that the virtual object can be experienced as if it were real.

According to the first aspect of the invention, when at least part of a virtual object is hidden behind a transmissive real object, information about a first striped pattern formed on the surface of the real object and the An acquisition means for acquiring information on a second striped pattern formed on the surface of a virtual object, and a relationship between an angle formed by the information on the first striped pattern and the information on the second striped pattern is an interference fringe pattern. It is determined whether a generation condition is satisfied, and if the generation condition is satisfied, a grayscale image generated by the first striped pattern and the second striped pattern is hidden behind the real object among the virtual objects. The information processing system includes generating means for generating interference fringes that reproduce how an area appears in a real space, and rendering means for rendering the virtual object including the generated interference fringes .
The invention according to claim 2 provides a computer with a first striped pattern formed on the surface of a transmissive real object when at least part of the virtual object is hidden behind the real object. Interference between a function of acquiring information and information of a second striped pattern formed on the surface of the virtual object, and an angle formed by the information of the first striped pattern and the information of the second striped pattern A determination is made as to whether or not a stripe occurrence condition is met, and if the occurrence condition is met, a grayscale image generated by the first stripe pattern and the second stripe pattern is displayed behind the real object among the virtual objects. A program for realizing a function of generating interference fringes that reproduce how a region hidden in the space looks like in the real space, and a function of drawing the virtual object including the generated interference fringes .

According to the first aspect of the invention, it is possible to draw a virtual object located behind a transparent real object and to experience as if the virtual object actually exists.
According to the second aspect of the invention, it is possible to draw a virtual object positioned behind a translucent real object, thereby making it possible to experience as if the virtual object actually exists.

FIG. 2 is a diagram illustrating the principle of a mixed reality experience experienced by a user wearing a glasses-type terminal capable of transparently viewing the outside world. It is a figure which shows an example of the hardware constitutions of a glasses type terminal. 1 is a diagram illustrating an example of a functional configuration of a glasses-type terminal; FIG. FIG. 10 is a flowchart illustrating an example of processing operations performed when displaying a virtual object on a glasses-type terminal; FIG. FIG. 4 is a diagram for explaining a region of a virtual object that is hidden behind a real object; (A) shows the positional relationship between the physical object and the virtual object, and (B) shows the portion of the virtual object that is hidden behind the physical object. FIG. 10 is a diagram for explaining the difference between drawing a virtual object according to a conventional technique and drawing a virtual object according to the present embodiment; (A) is an example of drawing a virtual object according to a conventional technique, and (B) is an example of drawing a virtual object according to the present embodiment. FIG. 10 is a diagram for explaining the effect of a difference in transmittance of a real object on rendering of a virtual object; (A) is an example of drawing a virtual object when the transmittance of the physical object is high, and (B) is an example of drawing the virtual object when the transmittance of the physical object is low. FIG. 10 is a diagram for explaining the effect of the difference in color of the physical object on the rendering of the virtual object; (A) is a rendering example of the virtual object when the real object is colored light blue, and (B) is a rendering example of the virtual object when the physical object is colored light red. FIG. 10 is a diagram for explaining how different patterns applied to a physical object affect drawing of a virtual object; (A) is a drawing example of a virtual object when oblique lines extending in a diagonal direction are formed on the surface of the physical object, and (B) is a drawing example of a virtual object when a mesh pattern is formed on the surface of the physical object. It is a drawing example of an object. FIG. 10 is a diagram for explaining the influence of a difference in refractive index between physical objects on rendering of a virtual object; (A) is an example of drawing a virtual object when the refractive index of the physical object is small, and (B) is an example of drawing a virtual object when the refractive index of the physical object is large. FIG. 10 is a diagram for explaining how a difference in the degree of polarization of a physical object affects rendering of a virtual object; (A) is an example of drawing a virtual object when the degree of polarization of the physical object is large, and (B) is an example of drawing a virtual object when the degree of polarization of the physical object is small. FIG. 10 is a diagram illustrating a case where interference fringes (moiré) are generated and a case where they are not generated due to the relationship between the surface pattern of a real object and the surface pattern of a virtual object; (A) is a drawing example of a virtual object when interference fringes do not occur, and (B) is a drawing example of a virtual object when interference fringes occur. FIG. 10 is a diagram for explaining an example of drawing a virtual object when there are a plurality of physical objects; (A) shows a mixed reality perceived by a user, and (B) is a diagram for explaining drawing processing of a virtual object. FIG. 10 is a diagram for explaining the principle of a mixed reality experience experienced by a user wearing a display device that displays an image obtained by synthesizing a virtual object with an image of the external world captured in real time. It is a figure showing an example of functional composition of a display.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<Embodiment 1>
In this embodiment, a case will be described where a glasses-type terminal capable of transparently viewing the outside world is used to experience mixed reality.
FIG. 1 is a diagram for explaining the principle of a mixed reality experience experienced by a user wearing a glasses-type terminal 1 capable of transmissively viewing the outside world.

The hardware portion of this type of terminal 1 has already been put into practical use by a plurality of manufacturers. For example, Microsoft's HoloLens (trademark), Sony's SmartEyeglass (trademark), and Konica Minolta's Wearable Communicator (trademark). This type of terminal 1 is also called a transmissive device, a retinal projection device, or the like.
A glasses-type terminal 1 shown in FIG. 1 has a highly transparent light guide plate 2, a small display unit 3 for displaying images, and a virtual object drawing unit 4 for drawing a virtual object (virtual object 11). ing.
The glasses-type terminal 1 here is an example of an information processing apparatus and an example of an information processing system.

The light guide plate 2 is made of, for example, a member having a transparency of 85% or more, and a visible light transmissive diffraction grating (not shown) is arranged therein. A holographic diffraction grating, for example, is used as the visible light transmission diffraction grating.
The visible light transmissive diffraction grating acts to linearly transmit the external light B1 incident from the front of the light guide plate 2 and guide it to the user's eyeball 5 . On the other hand, the visible light transmissive diffraction grating refracts the display light B2 incident on the light guide plate 2 from the display unit 3 to propagate inside the light guide plate 2, and then refracts the display light B2 in the direction of the eyeball 5. acts like
The external light B1 and the display light B2 are synthesized within the eyeball 5 . As a result, the user wearing the terminal 1 perceives a mixed reality scene in which the virtual object (virtual object 11) is combined with the real object (real object 12). Incidentally, in the example of FIG. 1, the virtual object 11 is located closer to the user than the real object 12 is.

<Hardware configuration of glasses-type terminal 1>
FIG. 2 is a diagram showing an example of the hardware configuration of the glasses-type terminal 1. As shown in FIG.
The terminal 1 shown in FIG. 2 includes a CPU (Central Processing Unit) 21 that controls the entire device through the execution of programs (including basic software), and a ROM 22 that stores programs such as BIOS (Basic Input Output System) and basic software. , and a RAM (Random Access Memory) 23 used as a program execution area.
The ROM 22 is composed of, for example, an electrically rewritable non-volatile semiconductor memory.
The CPU 21 , ROM 22 and RAM 23 function as the computer 20 .

The computer 20 includes display units 3L and 3R that display virtual objects, cameras 24L and 24R that capture images of the outside world, inertial measurement sensors 25 that measure inertial information such as angles, angular velocities, and accelerations, as well as real objects. , an illuminance sensor 27 for detecting ambient brightness, and a wireless communication unit 28 for communicating with the outside are connected.
An image for the left eye is displayed on the display section 3L for the left eye, and an image for the right eye is displayed on the display section 3R for the right eye. Parallax is reproduced between the image for the left eye and the image for the right eye. Therefore, the user wearing the terminal 1 can view the virtual object 11 stereoscopically.

The camera 24L is placed on the left eye side of the user and the camera 24R is placed on the right eye side of the user. Stereo images of the surroundings of the terminal 1 are taken by the cameras 24L and 24R. The images captured by the cameras 24L and 24R are used for recognizing the real object and measuring the distance to the surface of the real object. Note that the camera used for measuring the distance to the real object and the camera used for recognizing the real object may be prepared separately.
The inertial measurement sensor 25 is used to measure the position and orientation of the head, and is used for line-of-sight tracking and the like.
The depth sensor 26 uses infrared rays and ultrasonic waves to measure the distance to objects existing in the real space.

<Functional configuration of glasses-type terminal 1>
FIG. 3 is a diagram showing an example of the functional configuration of the glasses-type terminal 1. As shown in FIG.
The functional configuration shown in FIG. 3 is realized through the execution of programs by the CPU 21 .
The functional configuration shown in FIG. 3 represents, among various functions realized through execution of a program, a function that allows the user to perceive a mixed reality space in which virtual objects are placed behind real objects.

In the case of FIG. 3, the CPU 21 includes a real space information acquisition unit 31 that acquires real space information from images captured by the cameras 24L and 24R, and a real object transmission unit 31 that acquires transmission information of the physical object 12 (see FIG. 1). An information acquisition unit 32, and a virtual object transmission area determination unit 33 that determines an area hidden behind the transparent physical object 12 based on the position of the eyeball 5 (see FIG. 1) of the virtual object 11 (see FIG. 1). and a virtual object drawing unit 4 for drawing an image of the virtual object 11 on the display units 3L and 3R (see FIG. 2).

The physical space information acquisition unit 31 acquires various pieces of information about the physical space from the captured image, and stores the acquired information in the RAM 23 as physical space information 41 .
The type of information stored as the physical space information 41 varies depending on the occasion and purpose of using the spectacles-type terminal 1 .
However, by increasing the types of information, the experience in the mixed reality space can be brought closer to the experience in the real space.
In the case of this embodiment, the physical space information 41 includes information on the physical object 12 that is given or acquired in advance in addition to information on the physical object 12 that is added in real time.

Information about the physical object 12 may be estimated (calculated) from the captured image, or may be stored in the non-volatile area of the RAM 23 as known information for each physical object 12 .
Information estimated from a captured image includes information such as color information that can be directly obtained from the captured image, and information estimated using a method described later.
The non-volatile area of the RAM 23 stores, for example, information applied to all transparent portions of the physical object 12 (including a representative value of the transparency information and an equation for calculating the transparency information). Note that the non-volatile area of the RAM 23 may store information for each transparent portion.
The physical space information acquisition unit 31 in the present embodiment acquires from the RAM 23 information on each physical object 12 specified by image recognition.

The information stored in the RAM 23 may also include information on several types of filters that reproduce the appearance of a certain physical object 12 when viewed through another physical object 12 . Individual filters are given by a combination of multiple items such as transmittance, refractive index, and degree of polarization.
The physical space information acquisition unit 31 in the present embodiment may be provided with a function of acquiring a filter that realizes the same appearance as an image obtained by capturing a transparent portion of the physical object 12 . The filter here is an example of transparency information.

Information about the physical object 12 includes, for example, information about individual objects (including people), information about the physical space where the user is located, information about the distance from the user's position to each position in the image, information about the light source, and information about imaging. including information.
Here, the information of individual objects includes, for example, shape, color tone, material, transmission information, and information specifying the position in the physical space. Existing technology is used for object recognition. For example, a method of detecting edges and color regions as feature amounts is used. Artificial intelligence may be used for object recognition.
Information related to imaging includes information on the positions of the cameras 24L and 24R in the real space, directions of movement of the cameras 24L and 24R in the real space, information on directions in which the cameras 24L and 24R take images in the real space, and the like. is included. The images captured by the cameras 24L and 24R are accompanied by information on the date and time of capturing.

Transmittance information that provides various information about transmittance includes, for example, information on portions with and without transmittance, transmittance information on portions with transmittance, refractive index of portions with transmittance, and information on transmittance. It includes the color tone of the part, the degree of polarization of the transmissive part, the pattern of the transmissive part, and the like. Incidentally, the transmittance of the non-transmissive portion is 0 (zero).
Information such as transmittance may be estimated through image processing or may be given in advance. Techniques for estimating transparency include a method of comparing multiple images taken at multiple times, a method of acquiring transmission information corresponding to an object identified by artificial intelligence from a database, and the like. The database may be stored in a server (not shown) on a cloud network, for example. Note that if the database does not contain the transmission information corresponding to the specified object, the artificial intelligence may estimate the transmission information corresponding to the specified object based on the information of similar items existing in the database. .
The texture of the object changes depending on the combination of individual elements included in the transmission information.
Note that the physical space information 41 may be stored, for example, in a server (not shown) on a cloud network.

The physical space information acquisition unit 31 in the present embodiment also has a function of generating or updating a three-dimensional model simulating the physical space (that is, a function of virtualizing the physical space).
The physical space information acquisition unit 31 consistently integrates multiple pieces of information acquired from the physical space in the virtual space to generate or update a three-dimensional model. The three-dimensional model here is stored in the RAM 23 as the physical space virtualization information 42 .
A mixed reality space is a space in which the virtual object 11 is arranged in a space obtained by virtualizing the real space.

The physical object transmission information acquisition unit 32 in this embodiment acquires transmission information from the physical space information 41 of the physical object 12 .
In the case of the present embodiment, the physical object transmission information acquisition unit 32 acquires transmission information of the physical object 12 positioned in front of the virtual object 11 with respect to the position of the eyeball 5 of the user wearing the terminal 1. subject to
Here, information such as the position where the virtual object 11 is arranged (the position within the three-dimensional model), shape, color tone, material, etc. is stored as virtual object information 43 .
Note that the position of the user's eyeball 5 is given in relation to the terminal 1 rather than being actually measured.
The physical object transmission information acquisition unit 32 in the present embodiment is an example of acquisition means.

The virtual object transmission region determination unit 33 in the present embodiment determines the region of the virtual object 11 that is hidden behind the transparent portion of the physical object 12 .
A region of the virtual object 11 determined to be hidden behind a transparent portion of the physical object 12 is associated with the transparency information of the physical object 12 located on the near side.
In this embodiment, the transmission information of the physical object 12 associated with the virtual object 11 is called virtual object drawing information 44 . The contents of the virtual object drawing information 44 also change depending on the movement of the user wearing the terminal 1, the movement of the object in the physical space, and the position where the virtual object 11 is arranged.

The virtual object drawing unit 4 uses the real space virtualization information 42, the virtual object information 43, and the virtual object drawing information 44 to draw the image of the virtual object 11 for the display unit 3L (see FIG. 2) and the display unit 3R (see FIG. 2). ) draws the image of the virtual object 11 for
The virtual object drawing unit 4 in the present embodiment includes a region of the virtual object 11 that is hidden behind the real object 12 as a drawing target. That is, the virtual object 11 as a whole is to be drawn.
By displaying the virtual object 11 at a position hidden by the transparent physical object 12 in this manner, the user can perceive the virtual object 11 as a whole. As a result, it is possible to enhance the sense of reality in mixed reality as compared with the conventional technology.

Furthermore, the virtual object rendering unit 4 in the present embodiment makes the appearance of the virtual object 11 at a position hidden by the transparent physical object 12 more realistic. Transmittance, refractive index, color tone, pattern, etc. are reflected in rendering of the virtual object 11 .
For example, the virtual object rendering unit 4 prepares a filter to which the transmittance, refractive index, color tone, pattern, etc. of the physical object 12 are applied, and applies the prepared filter to the target area of the virtual object 11. to draw. Filters here also include information for rendering interference fringes and other interference effects perceived by superimposing multiple patterns.
That is, the virtual object rendering unit 4 creates an image that reproduces the appearance when another real object 12 is placed behind the transmissive real object 12, using the transmittance, refractive index, and color tone of the real object 12. , pattern, etc.
When a filter is used, the virtual object information 43 does not need to be changed, and the amount of calculation can be reduced. Therefore, even when the area hidden behind the physical object 12 changes rapidly, the transmission information of the physical object 12 can be reflected in the drawing of the virtual object 11 in real time.
The virtual object drawing unit 4 in the present embodiment is an example of generating means and drawing means.

<Processing operation performed by glasses-type terminal 1>
FIG. 4 is a flowchart for explaining an example of processing operations performed when the virtual object 11 is drawn on the glasses-type terminal 1. As shown in FIG.
The processing operations shown in FIG. 4 are realized through the execution of the program by the CPU 21 . In the drawing, the step is represented by the symbol S.
First, the CPU 21 acquires information on the physical space (step 1). Through this process, the CPU 21 recognizes the physical object 12 that the user wearing the terminal 1 is visually recognizing through the light guide plate 2 .
Next, the CPU 21 acquires transmission information of the recognized physical object 12 (step 2). In the case of this embodiment, it is important to recognize which part of the physical object 12 has transparency.

Next, the CPU 21 selects one of the one or more virtual objects 11 to be drawn (step 3).
The CPU 21 determines whether or not there is an area hidden behind the physical object 12 with the selected virtual object 11 as the processing target (step 4).
Here, the CPU 21 determines whether or not the virtual object 11 to be processed is hidden behind the real object 12 based on the position of the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 .
Whether or not the terminal 1 is hidden behind is determined based on the position of the eyeball 5 of the user wearing the terminal 1 . In step 4, it is determined whether or not the virtual object 11 is included within the range obtained by extending the virtual straight line connecting the eyeball 5 and the outer edge of the physical object 12 .
For example, if the physical object 12 does not exist between the virtual object 11 and the terminal 1 , the CPU 21 obtains a negative result and proceeds to step 7 .

FIG. 5 is a diagram for explaining a region 15 hidden behind the real object 12 in the virtual object 11. As shown in FIG. (A) shows the positional relationship between the physical object 12 and the virtual object 11 , and (B) shows the area 15 hidden behind the physical object 12 in the virtual object 11 .
In FIG. 5, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 is positioned on the normal line extending forward from the plane of the paper. That is, the physical object 12 is located closer to the user than the virtual object 11 is.
When the position of the eyeball 5 assumed in FIG. 5 is used as a reference, the area 15 of the virtual object 11 hidden behind the real object 12 is the shaded range.
Returning to the description of FIG.

If a positive result is obtained in step 4, the CPU 21 identifies the area where the physical object 12 is hidden behind the transparent part and the area where the physical object 12 is hidden behind the non-transparent (non-transparent) part. (Step 5).
In the case of step 5, the CPU 21 places the area of the virtual object 11 within the range obtained by extending the imaginary straight line connecting the eyeball 5 and the outer edge of the transmissive portion behind the transmissive portion of the real object 12. identified as a region hidden in
In the case of the present embodiment, the CPU 21 identifies areas other than the area identified as the area where the physical object 12 is hidden behind the transparent portion as the area hidden behind the non-transparent (non-transparent) portion. do.
Incidentally, in the case of FIG. 5, if the entire physical object 12 is a transparent portion, the region hidden behind the transparent portion matches the hatched region 15 .

Note that when there are a plurality of physical objects 12 positioned to hide the virtual object 11 behind, the CPU 21 identifies the region of the virtual object 11 hidden behind the transparent portion of each physical object 12 .
Also, when one physical object 12 has a plurality of transparent parts, the CPU 21 identifies the area of the virtual object 11 hidden behind each transparent part.
Therefore, the number of areas specified for one virtual object 11 is not limited to one. Note that when multiple areas are specified, those areas do not always match.

Next, the CPU 21 associates and stores the transmission information of the physical object 12 for each area specified in step 5 (step 6).
That is, the CPU 21 stores the virtual object drawing information 44 in association with the virtual object 11 to be processed. In this embodiment, it is assumed that the transmission information of the physical object 12 is already known.
After that, the CPU 21 determines whether or not all the virtual objects 11 have been selected (step 7).
If a negative result is obtained in step 7, the CPU 21 returns to step 3. At step 3, one of the unselected virtual objects 11 is selected as a processing target.
On the other hand, if a positive result is obtained in step 7, the CPU 21 draws all parts of all virtual objects 11 using the associated transparency information (step 8).

<Drawing example>
A drawing example of the virtual object 11 according to the present embodiment will be described below using a specific example.
<Drawing example 1>
FIG. 6 is a diagram for explaining the difference between drawing the virtual object 11 according to the conventional technique and drawing the virtual object 11 according to the present embodiment. (A) is a rendering example of the virtual object 11 according to the conventional technique, and (B) is a rendering example of the virtual object 11 according to the present embodiment.
In FIG. 6, a drawing example according to the conventional technique is indicated as a comparative example.
In the case of FIG. 6 as well, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 is positioned on the normal line extending forward from the paper surface.

As shown in (A), in the conventional technology, when the flat plate-shaped physical object 12 is closer to the user than the cylindrical virtual object 11 (that is, a part of the virtual object 11 is behind the physical object 12) hidden), even if the real object 12 is wholly translucent, the virtual object 11 in the area hidden behind it is excluded from the rendering target.
Therefore, the user does not perceive the virtual object 11 inside the translucent real object 12 , and instead perceives the real object located behind the translucent real object 12 .

On the other hand, outside the translucent physical object 12, the user perceives the cylindrical virtual object 11; Gone.
In this way, the scenery perceived by the user unnaturally connects between the inside and the outside of the transparent physical object 12 .
Further, in the conventional technology, even if the virtual object 11 is entirely hidden behind the physical object 12, the user cannot know the existence of the virtual object 11. FIG.

On the other hand, in the case of the present embodiment, as shown in (B), even if a part of the virtual object 11 is hidden behind the translucent real object 12, the user can see the entire cylindrical virtual object 11. can be perceived by
As described above, by using the technique according to the present embodiment, the scenery does not unnaturally connect between the inside and the outside of the real object 12 having transparency, and the experience as if the virtual object 11 actually exists can be realized. be possible.

Moreover, if the technology according to the present embodiment is used, when the user moves in the real space, the virtual object 11 that was perceived just before disappears as soon as it is hidden behind the translucent real object 12. It is possible to eliminate phenomena and phenomena in which the virtual object 11, which was not perceived until just before, suddenly appears from behind the translucent real object 12. - 特許庁
That is, even if the user moves around the translucent physical object 12, the virtual object 11 can be continuously perceived.
Therefore, it is possible for the user to perceive the virtual object 11 without distinguishing it from a real object.

The same applies when only the translucent physical object 12 moves without the user and the virtual object 11 moving, or when only the virtual object 11 moves without the user and the translucent physical object 12 moving. Since unnatural phenomena are eliminated, the sense of reality of the virtual object 11 can be enhanced.
Also, it becomes possible to make the user aware of the virtual object 11 that is entirely hidden behind the real object 12, which could not be noticed by the conventional technology.

<Drawing example 2>
7A and 7B are diagrams for explaining the influence of the difference in transmittance of the physical object 12 on drawing of the virtual object 11. FIG. (A) is a drawing example of the virtual object 11 when the transmittance of the physical object 12 is high, and (B) is a drawing example of the virtual object 11 when the transmittance of the physical object 12 is low.
In the case of FIG. 7 as well, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 (see FIG. 1) is positioned on the normal extending forward from the plane of the paper.
In the case of FIG. 7, the relatively high transmittance of 1 (case (A)) is higher than the relatively low transmittance of 2 (case (B)). It is possible to clearly perceive the shape of the virtual object 11 .
Therefore, the user can experience as if the virtual object 11 actually exists.

<Drawing example 3>
8A and 8B are diagrams for explaining the influence of the difference in color of the physical object 12 on drawing of the virtual object 11. FIG. (A) is a rendering example of the virtual object 11 when the physical object 12 is colored light blue, and (B) is a rendering example of the virtual object 11 when the physical object 12 is colored light red. be.
In the case of FIG. 8 as well, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 (see FIG. 1) is positioned on the normal extending forward from the plane of the paper.
Due to drawing restrictions, light blue and light red cannot be expressed in FIG. It becomes possible to approximate the appearance of other objects in the perceived real space.
Therefore, the user can experience as if the virtual object 11 actually exists.

<Drawing example 4>
9A and 9B are diagrams for explaining the effect of different patterns applied to the physical object 12 on drawing of the virtual object 11. FIG. (A) is a drawing example of the virtual object 11 when oblique lines (pattern 1) extending diagonally are formed on the surface of the physical object 12, and (B) is a mesh pattern (pattern 1) on the surface of the physical object 12. This is a drawing example of the virtual object 11 when the pattern 2) is formed.
In the case of FIG. 9 as well, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 (see FIG. 1) is positioned on the normal line extending forward from the paper surface.
As shown in FIG. 9, by reflecting the pattern formed on the surface of the physical object 12 in the drawing of the virtual object 11, the appearance of the virtual object 11 perceived through the physical object 12 can be It is possible to approximate the appearance of other objects in the real space perceived through the .
Therefore, the user can experience as if the virtual object 11 actually exists.

<Drawing example 5>
FIG. 10 is a diagram for explaining how the difference in the refractive index of the physical object 12 affects the drawing of the virtual object 11. As shown in FIG. (A) is a drawing example of the virtual object 11 when the physical object 12 has a small refractive index, and (B) is a drawing example of the virtual object 11 when the physical object 12 has a large refractive index.
In the case of FIG. 10 as well, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 (see FIG. 1) is positioned on the normal extending forward from the plane of the paper.
In the case of FIG. 10, due to the influence of the refractive index of the physical object 12, the outer edge of the virtual object 11 drawn at a position hidden behind the physical object 12 and the outer edge of the virtual object 11 drawn outside the physical object 12 are different. are discontinuous. That is, there is a shift in the drawing position of the outer edge of the virtual object 11 with the outer edge of the physical object 12 as a boundary.
Also, the amount of deviation in drawing is small at a refractive index of 1 (case A), which is relatively small, and is large at a refractive index of 2 (case of B), which is relatively large.
Therefore, the user can experience as if the virtual object 11 actually exists.

<Drawing example 6>
11A and 11B are diagrams for explaining the effect of the difference in the polarization information of the physical object 12 on the drawing of the virtual object 11. FIG. (A) is an example of drawing the virtual object 11 when the degree of polarization of the physical object 12 is large, and (B) is an example of drawing the virtual object 11 when the degree of polarization of the physical object 12 is small.
In the case of FIG. 11 as well, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 (see FIG. 1) is positioned on the normal extending forward from the plane of the paper.
In the example of (A), since the degree of polarization is large (the rate of removing reflected light and miscellaneous light is large), the proportion of the light component arriving from the virtual object 11 in the light component passing through the physical object 12 is relatively in a state of abundance. Therefore, the visibility of the virtual object 11 is good.
On the other hand, in the example of (B), since the degree of polarization is relatively small (the rate of removing reflected light and random light is small), the light component arriving from the virtual object 11 among the light components passing through the real object 12 is The ratio is relatively small. Therefore, the visibility of the virtual object 11 is degraded.
Therefore, the user can experience as if the virtual object 11 actually exists.
In addition, the effect due to the difference in information on polarization may include the influence due to the difference in optical rotation. By reflecting the difference in optical rotation in drawing, it is possible to express the difference in appearance of optical isomers.

<Drawing example 7>
12A and 12B are diagrams for explaining cases in which interference fringes (moire) occur and do not occur due to the relationship between the surface pattern of the real object 12 and the surface pattern of the virtual object 11. FIG. (A) is a drawing example of the virtual object 11 when no interference fringes are generated, and (B) is a drawing example of the virtual object 11 when interference fringes are generated.
An interference fringe is a pattern that is perceived due to a shift in the period when a plurality of regularly repeated patterns are superimposed.

In the case of FIG. 12 as well, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 (see FIG. 1) is positioned on the normal line extending forward from the paper surface.
In the example of (A), diagonal lines are uniformly formed on the surface of the physical object 12 and diagonal lines are uniformly formed on the surface of the virtual object 11 . The angle between the oblique lines here does not satisfy the conditions for generating interference fringes. For this reason, in the area where the real object 12 and the virtual object 11 having transparency overlap, a pattern in which the patterns of the two objects are simply superimposed is drawn.

Interference fringes occur, for example, when two fringes with the same width intersect at a slight angle, or when two fringes with slightly different widths overlap in parallel and the spacing between fringes changes periodically. . Needless to say, interference fringes are generated when patterns having periodicity (for example, parallel lines and concentric circles) meet specific conditions.
The example of (B) shows interference fringes that occur when the striped pattern of the real object 12 and the striped pattern of the virtual object 11 have the same width and are arranged so that they intersect at a slight angle. represent. The interference fringes appear as a pattern of pseudo grayscale.

In the case of this embodiment, when the CPU 21 determines that the striped pattern of the real object 12 recognized by the image processing and the striped pattern of the virtual object 11 to be drawn satisfy the conditions for generating interference fringes, the interference fringes are generated. to draw.
Therefore, the user can experience as if the virtual object 11 actually exists.
Although FIG. 12 exemplifies interference fringes as an example of a phenomenon perceived by light interference, effects not accompanied by interference fringes, such as Morpho butterfly scales and Majora paint, may be reproduced. Here, the effect of morpho butterfly scales is the effect of reflecting only blue wavelength light through interference, and the effect of majora coating is that the surface of an object changes color depending on the viewing angle and how the light hits it. say the effect.

<Drawing example 8>
FIG. 13 is a diagram for explaining an example of drawing the virtual object 11 when there are a plurality of physical objects 12. As shown in FIG. (A) shows a mixed reality perceived by a user, and (B) is a diagram for explaining drawing processing of a virtual object 11. FIG.
In the case of FIG. 13 as well, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 (see FIG. 1) is positioned on the normal line extending forward from the plane of the paper.
In the case of FIG. 13, the physical object 12A is a frame-shaped member that does not have transparency, and the physical object 12B is a member that has transparency. The physical object 12A is, for example, a windowpane frame, and the physical object 12B is, for example, a glass plate of the windowpane.

In the case of FIG. 13, part of the virtual object 11 is hidden behind the physical object 12A, and the remaining part of the virtual object 11 is hidden behind the physical object 12B.
That is, the entire virtual object 11 is hidden behind the physical objects 12A and 12B.
Therefore, with the conventional technology, the virtual object 11 is not rendered. As a result, the user cannot notice the existence of the virtual object 11 .

On the other hand, in the present embodiment, transparency information of the physical object 12B is associated with the region 11B of the virtual object 11 that is hidden behind the real object 12B having transparency, and The transmission information of the physical object 12A is associated with the region 11A hidden behind the object 12A.
Therefore, the area 11B of the virtual object 11 becomes a drawing area, and the area 11A becomes a non-drawing area.
In addition, the user wearing the terminal 1 perceives a part of another real object behind the area 11B. is behind the virtual object 11 can be understood.
As described above, the user can experience as if the virtual object 11 actually exists.

<Embodiment 2>
In this embodiment, a case of using a head-mounted display device for experiencing mixed reality will be described.
FIG. 14 is a diagram illustrating the principle of experiencing mixed reality by a user wearing a display device 100 that displays an image obtained by synthesizing a virtual object with an image of the external world captured in real time. .

In FIG. 14, parts corresponding to those in FIGS. 1 and 2 are denoted by corresponding reference numerals.
The display device 100 displays in front of the user's eyeball 5 an image obtained by synthesizing the image of the external world captured by the cameras 24L and 24R and the image of the virtual object 11 drawn by the virtual object drawing unit 4 by the image synthesizing unit 101. It is displayed on the arranged display units 3L and 3R.
The display device 100 here is an example of an information processing device and an example of an information processing system.
The hardware configuration of the display device 100 is the same as that of the glasses-type terminal 1 (see FIG. 2). Therefore, description of the hardware configuration of the display device 100 is omitted.

FIG. 15 is a diagram showing an example of the functional configuration of the display device 100. As shown in FIG.
In FIG. 15, parts corresponding to those in FIG. 3 are shown with reference numerals.
The basic functional configuration of the display device 100 is the same as that of the glasses-type terminal 1 (see FIG. 2). A functional configuration unique to the display device 100 is an image synthesizing unit 101 .
The image synthesizing unit 101 has a function of synthesizing two images so that the image drawn by the virtual object drawing unit 4 and the image of the external world picked up by the cameras 24L and 24R are matched.
For example, the image synthesizing unit 101 compares the three-dimensional model stored as the physical space virtualization information 42 with the images of the external world captured by the cameras 24L and 24R, and selects an area for synthesizing the image of the virtual object 11. decide.
As described above, although the system of this embodiment for perceiving mixed reality is different from that of the first embodiment, the fact that the sense of reality of mixed reality perceived by the user is higher than that of the conventional technique is the same as that of the embodiment. Same as 1.

<Other embodiments>
Although the embodiments of the present invention have been described above, the technical scope of the present invention is not limited to the scope described in the above-described embodiments. It is clear from the scope of claims that the technical scope of the present invention includes various modifications and improvements to the above-described embodiment.
For example, in the above embodiment, the display units 3L and 3R for both the left and right eyes are used, but the number of display units may be one. For example, in the case of a glasses-type terminal 1 (see FIG. 1), one display unit may be arranged in front of either the left or the right. Further, for example, in the case of the display device 100 (see FIG. 12), one display section may be arranged in front of both eyes.
In the above-described embodiment, the virtual object drawing unit 4 is implemented as one of the functions of the glasses-type terminal 1 (see FIG. 1) or the display device 100 (see FIG. 12). The function of the virtual object drawing unit 4 may be executed in an information processing device such as a server connected to a cloud network. The glasses-type terminal 1 and the server on the external network that executes the functions of the virtual object drawing unit 4 are an example of an information processing system.
In the above-described embodiment, the functions of the virtual object rendering unit 4 are realized using the CPU 21, which is a general-purpose arithmetic unit. Graphics Processing Unit) may be used.

REFERENCE SIGNS LIST 1 glasses-type terminal 2 light guide plate 3, 3L, 3R display unit 4 virtual object rendering unit 11 virtual object 11A area hidden behind real object 12A 11B of real object 12B Area hidden behind 15 Area hidden behind real object 12 in virtual object 11 12, 12A, 12B Real object 31 Real space information acquisition unit 32 Real object transmission information acquisition unit 33 Virtual Object transmission region determination unit 41 Real space information 42 Real space virtualization information 43 Virtual object information 44 Virtual object drawing information 100 Display device 101 Image synthesizing unit B1 Outside light B2 …Display light

Claims (2)

When at least part of a virtual object is hidden behind a transmissive real object, information of a first striped pattern formed on the surface of the real object and formed on the surface of the virtual object Acquisition means for acquiring information of the second striped pattern;
It is determined whether or not the relationship between the angle formed by the information on the first stripe pattern and the information on the second stripe pattern satisfies the interference fringe generation condition. generating means for generating a grayscale image generated by the second striped pattern as interference fringes that reproduce how a region of the virtual object hidden behind the real object appears in real space ;
drawing means for drawing the virtual object including the generated interference fringes ;
An information processing system having to the computer,
When at least part of a virtual object is hidden behind a transmissive real object, information of a first striped pattern formed on the surface of the real object and formed on the surface of the virtual object a function of acquiring information of the second striped pattern;
It is determined whether or not the relationship between the angle formed by the information on the first stripe pattern and the information on the second stripe pattern satisfies the interference fringe generation condition. a function of generating a grayscale image generated by the second fringe pattern as an interference fringe that reproduces how a region of the virtual object hidden behind the real object appears in real space ;
and a function of drawing the virtual object including the generated interference fringes .

Images