JP2022035607A - Presentation system, server, second terminal and program - Google Patents

Abstract

To provide a presentation system capable of efficiently performing avatar drawing.SOLUTION: A presentation system includes: a first recognition part 11 for acquiring first recognition information by recognizing a first user state; a second positioning part 22 for acquiring second positioning information by positioning a position attitude of a second user; a third drawing part 33 for acquiring third drawing information obtained by drawing an avatar of the first user, by reflecting the first recognition information at a virtual camera viewpoint arranged at the second positioning information; a fourth drawing part 34 for acquiring fourth drawing information obtained by drawing the avatar in quality higher than in a drawing mode of the third drawing part; an extraction part 35 for extracting difference between the third drawing information and the fourth drawing information as second extraction information; a second drawing part 26 for acquiring second drawing information obtained by drawing the avatar in the same quality as in the drawing mode of the third drawing part; a second integration part 27 for acquiring second integration information being the same avatar obtained by simulating the fourth drawing information, by reflecting the second extraction information on the second drawing information; and a second presentation part 28 for displaying the second integration information to the second user.SELECTED DRAWING: Figure 2

Description

The present invention relates to a presentation system, a server, a terminal and a program for drawing an avatar, which can be used for remote communication and the like.

Regarding the video communication technology between remote locations that can be used for remote communication, Non-Patent Document 1 transmits the user's point group information measured by a depth sensor as 3D (three-dimensional) video transmission to the communication partner, and the other party's point group information is transmitted. It discloses the efforts to draw on the device. Further, Patent Document 1 discloses a method in which a plurality of images taken by a terminal are transmitted to a server, skeleton information estimated by the server is transmitted to a communication partner, and then the terminal applies the skeleton information to an avatar.

Japanese Unexamined Patent Publication No. 2020-6229

Ben Cutler, 2 others, "holoportation", [online], September 4, 2018 [Search on July 17, 2018], Internet <URL: https://www.microsoft.com/en-us / research / project / holoportation-3 ＞

However, in the prior art, it has not been possible to realize high-quality information presentation under the condition that the computational resources of the terminal used on the user side, the communication band, and the like cannot always be abundantly used.

Non-Patent Document 1 has a problem that it cannot be realized if the communication band is narrow because a huge amount of point cloud information is transmitted. Further, in order to acquire the user's point cloud information from all directions, it is necessary to arrange a plurality of depth sensors, which causes a problem that the device becomes large-scale. Further, since the resolution of the depth sensor is limited, there is a problem that the separation between the user and the background is not sufficient and the background is mixed in the user's point cloud and the quality is low. Moreover, there is a problem that a blind spot is generated depending on the posture of the user and the point cloud of the relevant portion cannot be acquired.

Patent Document 1 has a problem that it cannot be realized if the communication band is narrow because a plurality of images from different viewpoints are transmitted from the terminal to the server. In addition, since the avatar is drawn on a terminal with limited computational resources, there is a problem that the quality is lower than that of drawing on a server with abundant computational resources when trying to draw in real time.

In view of the above problems of the prior art, it is an object of the present invention to provide a presentation system, a server, a second terminal and a program capable of efficiently performing avatar drawing.

In order to achieve the above object, the present invention is a presentation system for positioning the position and orientation of a first recognition unit that recognizes a state related to communication of a first user and obtains first recognition information, and a second user. In the second positioning unit that obtains the second positioning information and the virtual camera viewpoint arranged in the second positioning information, the third drawing information that reflects the first recognition information and draws the avatar of the first user is obtained. In the third drawing unit to be obtained and the virtual camera viewpoint arranged in the second positioning information, the avatar of the first user is drawn with higher quality than the drawing mode of the third drawing unit, reflecting the first recognition information. The fourth drawing unit that obtains the fourth drawing information, the extraction unit that extracts the difference between the third drawing information and the fourth drawing information as the second extraction information, and the virtual camera viewpoint arranged in the second positioning information. In the second drawing unit, which reflects the first recognition information and obtains the second drawing information in which the avatar of the first user is drawn with the same quality as the drawing mode of the third drawing unit, and the second drawing information. The second integrated unit that obtains the second integrated information that is the avatar of the first user as imitating the fourth drawing information by reflecting the second extracted information, and the second integrated information. It is characterized by including a second presentation unit to be displayed to two users. Further, the present invention is characterized in that the server includes the third drawing unit, the fourth drawing unit, and the extraction unit. Further, the present invention is characterized in that it is a second terminal including the two positioning units, the second drawing unit, the second integrated unit, and the second presenting unit. Further, the present invention is characterized in that it is a program that causes a computer to function as the server or the second terminal.

According to the present invention, as the difference between two common avatars drawn with different qualities, the second is extracted at high speed even when a line having a narrow communication band is used by being extracted in the form of the second extraction information. The second integration as a high-quality drawing result equivalent to that of the 4th drawing unit without transmitting the extracted information to the 2nd user side and performing high-quality drawing by the 4th drawing unit on the 2nd user side. Since it is possible to obtain information and display it to the second user as an avatar of the first user, it is possible to efficiently draw the avatar.

It is a block diagram of the presentation system which concerns on one Embodiment. It is a functional block diagram of the presentation system which concerns on one Embodiment in the case of N = 2. It is a sequence diagram of the operation of the presentation system which concerns on one Embodiment. It is a figure which shows the schematic example of the remote communication in real time by avatar. As a mock example of the first recognition information, it is a figure which shows the feature point distribution about a facial expression extracted from a facial image in the case of a facial expression recognition. It is a figure which shows the schematic example which minimized the quantization error by limiting the quantization step qi to 1 or 4. It is a figure which shows the schematic example of each drawing information and the 2nd extraction information. It is a functional block diagram of the presentation system which concerns on one Embodiment when the avatar display processing is performed bidirectionally. It is a figure which shows the example of the hardware configuration in a general computer.

FIG. 1 is a configuration diagram of a presentation system 100 according to an embodiment, in which the presentation system 100 is configured to be able to communicate with each other via a network NW such as the Internet, and N (N ≧ 2) terminals 10, It is equipped with 20, ..., N0, and server 30. The users who use the terminals 10, 20, ..., N0 are the users U1, U2, ..., UN, respectively, and these N users U1, U2, ..., UN are their own terminals 10, 20, ..., N0 (for example). By using a smartphone terminal, a head-mounted display terminal, etc.), it is possible to perform remote communication using an avatar (communication partner user's avatar) by the presentation system 100 while being present at each remote location.

Hereinafter, for the sake of explanation, N = 2 is set, and remote communication is performed between the first user U1 using the first terminal 10 and the second user U2 using the second terminal 20 in the presentation system 100 via the server 30. Is taken as an example. Even in the case of N ≧ 3, remote communication between any two users out of N users is realized in the same way as in the case of two users U1 and U2, so that remote communication with N people can be performed in exactly the same way. It is possible to do.

FIG. 2 is a functional block diagram of the presentation system 100 according to the embodiment in the case of N = 2. The presentation system 100 includes a first terminal 10 used by the first user U1, a second terminal 20 used by the second user U2, and a server 30. The first terminal 10 and the second terminal 20 acquire information necessary for remote communication from users U1 and U2, respectively, and transmit the acquired information to the other terminal. The server 30 performs a process of relaying the transmitted information and transmitting it to the terminal of the other party, but after performing a predetermined drawing process or the like using the information transmitted at the time of the relay, the other party By transmitting to the terminal of No. 1, high-quality remote communication between users U1 and U2 can be realized even in the situation where the first terminal 10 and the second terminal 20 have restrictions on computational resources and the like. ..

As shown in FIG. 2, the first terminal 10 includes a first recognition unit 11 and a first positioning unit 12, and the second terminal 20 has a second positioning unit 22, a second drawing unit 26, a second integrated unit 27, and a second terminal 20. 2 The presentation unit 28 is provided, and the server 30 includes a third drawing unit 33, a fourth drawing unit 34, and an extraction unit 35.

In FIG. 2, the third drawing unit 33 and the fourth drawing unit 34 of the server 30 are collectively shown as the functional unit 31, which represents the following information transmission / reception. That is, the first recognition information and the first positioning information acquired by the first recognition unit 11 and the first positioning unit 12 of the first terminal 10, and the second positioning information acquired by the second positioning unit 22 of the second terminal 20 respectively. Is transmitted to the server 30 side, and is collectively shown as a functional unit 31 in order to express that the third drawing unit 33 and the fourth drawing unit 34 use these information.

FIG. 3 is a sequence diagram of the operation of the presentation system 100 according to the embodiment, and is presented by performing the entire operation of FIG. 3 at each time t = 1, 2, 3, ... At a predetermined processing rate. The system 100 enables remote communication using avatars in real time between the first user U1 who uses the first terminal 10 and the second user U2 who uses the second terminal 20.

As shown in FIGS. 2 and 3, the outline of real-time processing at each time t is as follows. (Note that only the outline will be explained first from the viewpoint of the processing of each functional unit and the flow of exchanging processing information between each functional unit, and the details of the individual processing of each functional unit will be described later.)

In the first terminal 10, the first recognition unit 11 recognizes the facial expression of the user U1 and obtains the first recognition information R1 (t) at time t, and uses this first recognition information R1 (t) on the server 30. It is transmitted to the third drawing unit 33 and the fourth drawing unit 34 (steps S111 and S112). In the first terminal 10, the first positioning unit 12 positions the position and orientation of the first user U1 at time t to obtain the first positioning information P1 (t), and uses this first positioning information P1 (t) as a server. It is transmitted to the third drawing unit 33 and the fourth drawing unit 34 of 30 (steps S121 and S122).

In the second terminal 20, the second positioning unit 22 positions the position and orientation of the second user U2 at time t to obtain the second positioning information P2 (t), and uses this second positioning information P2 (t) as a server. It is transmitted to the third drawing unit 33 and the fourth drawing unit 34 of 30 (steps S221 and S222), and is output to the second drawing unit 26 in the second terminal 20 (step S223).

The third drawing unit 33 of the server 30 draws the avatar of the first user U1 at time t in standard quality in the position and orientation determined by the first positioning information P1 (t) and the second positioning information P2 (t). As a result, the third drawing information G3 (t) is obtained, and the third drawing information G3 (t) is output to the extraction unit 35 (step S331). The fourth drawing unit 34 draws the avatar of the first user U1 at time t in high quality in the position and orientation determined by the first positioning information P1 (t) and the second positioning information P2 (t). The drawing information G4 (t) is obtained, and the fourth drawing information G4 (t) is output to the extraction unit 35 (step S341).

Here, the fourth drawing unit 34 draws with higher quality than the drawing quality of the third drawing unit 33. Further, the drawing quality of the third drawing unit 33 is the same as the drawing quality of the second drawing unit 26 of the second terminal 20, and the third drawing unit 33 and the second drawing unit 26 perform the same drawing. In the second drawing unit 26, the third drawing unit 33, and the fourth drawing unit 34, in the common position and posture determined by the first positioning information P1 (t) and the second positioning information P2 (t), the time is determined by the drawing quality of each. Draw the avatar of the first user U1 at t.

In the server 30, the extraction unit 35 further extracts the difference (difference) between the third drawing information G3 (t) and the fourth drawing information G4 (t) as the second extraction information E2 (t) at time t. This second extraction information E2 (t) is transmitted to the second integration unit 27 of the second terminal 20 (step S352).

Here, instead of the third drawing information G3 (t) and the fourth drawing information G4 (t) which are the direct drawing results of the avatar, the second extraction information E2 (t) as the difference in which the amount of information is reduced is used. By transmitting from the server 20 to the second terminal 20, it is possible to suppress the pressure on the communication band of the network NW.

The server 30 also performs the first recognition of the first user at the time t obtained from the first recognition unit 11 and the first positioning unit 12 as a relay process of information transmission from the first terminal 10 to the second terminal 20. The information R1 (t) and the first positioning information P1 (t) are transmitted as they are to the second drawing unit 26 of the second terminal 20 (step S351). (In addition, in FIGS. 2 and 3, for convenience, the relay transmission source of the first recognition information R1 (t) and the first positioning information P1 (t) is drawn as the extraction unit 35, but the extraction unit 35 particularly draws the first. Processing such as further processing of the recognition information R1 (t) and the first positioning information P1 (t) is not performed.)

The second drawing unit 26 of the second terminal 20 draws the avatar of the first user U1 at time t in standard quality in the position and orientation determined by the first positioning information P1 (t) and the second positioning information P2 (t). As a result, the second drawing information G2 (t) is obtained, and the second drawing information G2 (t) is output to the second integration unit 27 (step S261).

As described above, the second drawing unit 26 of the second terminal 20 has the same quality as the third drawing unit 33 of the server 30, and is common to be determined by the first positioning information P1 (t) and the second positioning information P2 (t). At the position and orientation of, the avatar of the first user U1 at time t is drawn. That is, the second drawing information G2 (t) obtained by the second drawing unit 26 is the same as the third drawing information G3 (t) obtained by the third drawing unit 33.

The second integration unit 27 adds the second extraction information E2 (t) obtained from the extraction unit 35 of the server 30 to the second drawing information G2 (t), thereby causing the second integration at time t. The information G2S (t) is obtained, and the second integrated information G2S (t) is output to the second presentation unit 28 (step S271). The second presentation unit 28 is configured as a display and displays the second integrated information G2S (t) to the second user U2.

Here, the second extraction information E2 (t) is the difference between the fourth drawing information G4 (t) drawn with high quality on the server 30 and the third drawing information G3 (t) drawn with standard quality (“E2). (t) = G4 (t) -G3 (t) "), and the second drawing information G2 (t), which is the same information as the third drawing information G3 (t), is the second of the second terminal 20. Obtained by the drawing unit 26. Therefore, the second integrated information G2S (t) obtained by adding in the second integrated unit 27 becomes the same information as the fourth drawing information G4 (t) drawn with high quality in the server 30, and is the fourth information of the server 30. The avatar of the first user U1 as a result of being drawn with high quality in the drawing unit 34 is restored by addition as the second integrated information G2S (t) without directly drawing on the second terminal 20 itself, and is restored by the second user. It will be possible to present to U2.

As will be described later, when the second extraction information E2 (t) is obtained as a difference (“E2 (t) = G4 (t) -G3 (t)”) and then lossy compression by quantization is further applied. However, in this case as well, the second integrated information G2S (t) obtained by adding in the second integrated unit 27 is completely different from the fourth drawing information G4 (t) drawn with high quality on the server 30. Is not the same, but it will be obtained as an imitation of this as much as possible.

FIG. 4 is a diagram showing a schematic example of real-time remote communication by the avatar realized by the above configurations of FIGS. 2 and 3. As shown in Example EX11, the first user U1 who uses the first terminal 10 exists in the environment E1 (room, etc.), and the second user U2 who uses the second terminal 20 is in the environment E2 at a remote location. Exists in (room, etc.). For remote environments E1 and E2 such as Example EX11, Example EX12 shows avatar communication by augmented reality display realized by the presentation system 100. The first user U1 who uses the first terminal 10 displays the avatar A2 of the second user U2 who is the communication partner in the virtual space V1 in which the augmented reality display (superimposed display of the avatar A2) is added to the environment E1. By communicating with the avatar A2, it becomes possible to communicate with the second user U2 that exists remotely. Similarly, the second user U2 who uses the second terminal 20 has the avatar A1 of the first user U1 who is the communication partner in the virtual space V2 in which the augmented reality display (superimposed display of the avatar A1) is added to the environment E2. By displaying and communicating with the avatar A1, it becomes possible to communicate with the remote first user U1.

The above configurations of FIGS. 2 and 3 realize the right side of the example EX12 of FIG. (That is, the second integrated information G2S (t) is the avatar A1 of the first user U1 and is provided to the second user U2 as an augmented reality display.) The first terminal 10 and the second. By exchanging the roles of the terminal 20, the left side of the example EX12 in FIG. 4 can be realized in exactly the same way.

Here, the avatar A1 reflects the first positioning information P1 (t), which is the position and orientation of the first user U1, and the first recognition information R1 (t), which is a facial expression, etc., in the form of the second integrated information G2S (t). Since it is drawn in real time, the real-time behavior of the first user U1 is reflected as it is as an avatar, and augmented reality is displayed in the virtual space V2 for the second user U2. That is, if the first user U1 changes the position and posture, the avatar A1 also follows and the three-dimensional position and posture in the virtual space V2 changes, and if the first user U1 changes the facial expression and the like, the avatar A1 also follows. The facial expression etc. changes. (The opposite, the relationship between the avatar A2 and the first user U1 in the virtual space V1 is exactly the same.)

Further, the avatar A1 is in the form of the second integrated information G2S (t) as seen from the second positioning information P2 (t) which is the position and orientation of the viewpoint of the second user U2 (virtual for drawing on the image plane). Since it is drawn in real time (using the position and orientation of the second positioning information P2 (t) as the position and orientation of the camera), the second user U2 moves and wraps around, for example, to the side or back of the avatar A1. , It is also possible to see Avata A1.

In this way, the avatar A1 drawn as the second integrated information G2S (t) at each time t is drawn as a two-dimensional region on the image plane when the time t is fixed, but the avatar of the three-dimensional model Since it is drawn two-dimensionally as a drawing result, it is displayed while changing as a three-dimensional shape according to the movement of users U1 and U2 with the progress of time t.

The bidirectional avatar communication shown in Example EX12 of FIG. 4 is realized by the configuration of FIG. 8 described later, but the configuration of FIG. 8 is a rewrite of the configuration of FIGS. 2 and 3 as bidirectional. It is nothing more than a configuration in which the roles of the first terminal 10 and the second terminal 20 are exchanged with respect to the configurations of FIGS. 2 and 3, which corresponds to the configuration of FIG.

Hereinafter, details of each functional unit of FIG. 2 that operates in real time at each time t = 1, 2, 3, ... As shown in FIG. 3 will be described.

The first recognition unit 11 recognizes, for example, a posture (pose) and / or a facial expression as state information related to the communication of the first user U1, and obtains the first recognition information R1 (t) at time t. FIG. 5 is a diagram showing a landmark coordinate distribution relating to a facial expression extracted from a facial expression in the case of facial expression recognition as a mock example of the first recognition information. For facial expression recognition, existing techniques for estimating the coordinates of facial landmarks as in Non-Patent Document 2 below can be used. For posture recognition, existing technologies for estimating skeletal information, such as tracking technology using sensors attached to each part of the body and image recognition technology using a camera, can be used.
[Non-Patent Document 2] A. Bulat and et al. `` How far are we from solving the 2D & 3D Face Alignment problem ?,'' International Conference on Computer Vision, 2017

The first positioning unit 12 and the second positioning unit 22 obtain the first positioning information P1 (t) and the second positioning information P2 (t) at time t as the positions and postures of the first user U1 and the second user U2, respectively. The processing of the first positioning unit 12 and the second positioning unit 22 is common, and Visual SLAM (image-based self-position estimation and environment) is an arbitrary existing method for positioning the position and orientation (information corresponding to the external parameters of the camera). Existing technologies such as (simultaneous execution of map creation) and 6DOF (6 degrees of freedom) sensor can be used. If the depth information of the environment in which the user exists can be obtained at the time of positioning, it can be included as a part of the positioning information.

In order to capture and capture the first user U1 and the second user U2 when the first recognition unit 11, the first positioning unit 12, and the second positioning unit 22 are realized by any existing method as described above. When using a camera, a dedicated sensor, etc. as the hardware of the device, the hardware is fixedly provided in the first terminal 10 or the second terminal 20 (for example, the mobile device when each terminal is a mobile terminal such as a smartphone). It may be the built-in camera of the terminal), or it may be installed and provided in the environments E1 and E2 in which the first user U1 and the second user U2 exist.

In the server 30, the third drawing unit 33 and the fourth drawing unit 34 both have three-dimensional coordinates determined by the first positioning information P1 (t) and the second positioning information P2 (t) (the virtual space V2 of the second user U2). By arranging the avatar A1 of the first user U1 on the (three-dimensional camera coordinates) and drawing this avatar A1 reflecting the first recognition information R1 (t), the third drawing information G3 (t) and the third drawing information G3 (t) and the third are drawn, respectively. 4 Obtain drawing information G4 (t). As described above, the third drawing information G3 (t) and the fourth drawing information G4 (t) are the same three-dimensional avatars drawn in the same state and in the same arrangement, but only the drawing quality is different. Therefore, what distinguishes the drawing quality is the setting related to rendering such as the setting of the light source model and the surface reflection model in 3D computer graphics.

Here, regarding the avatar A1 of the first user U1 to be drawn, a predetermined three-dimensional structure is configured so that the first recognition information R1 (t) can be reflected in drawing as a parameter (a parameter for determining a facial expression or a pose). You can prepare the model in advance. Any existing 3D computer graphics technique may be used for drawing according to facial expressions and poses.

The second positioning information P2 (t) represents the position and orientation of the viewpoint of the second user U2 who sees the avatar A1 of the first user U1, and is the three-dimensional world coordinates of the environment E2 in which the second user U2 exists. Predetermined conversion (translation and rotation conversion) T for the first positioning information P1 (t) of the first user U1 that is acquired as the internal position and orientation (external parameter of the camera) and is drawn as the avatar A1. The first positioning information T / P1 (t) converted into the three-dimensional world coordinates of the environment E2 is used as the position / orientation of the avatar A1, and this is performed from the virtual camera position determined by the second positioning information P2 (t). Just draw the avatar A1.

For example, the position (translational component) in the converted first positioning information T / P1 (t) may be drawn as the position of the avatar's face, and the posture (rotational component) may be drawn as the direction of the avatar's face.

In the server 30, the extraction unit 35 is obtained in a state where the third drawing information G3 (t) and the fourth drawing information G4 (t) (both of which, the avatar A1 is drawn as a mask image only in the same region in the image plane). ), The second extraction information E2 (t) = G4 (t) -G3 (t) is obtained.

Here, in order to suppress the transmission amount of the second extraction information, "a · G3 (t) + b" obtained by performing a primary conversion on the third drawing information G3 (t) and the fourth drawing information G4 (t) The second extraction information E2 (t) may be obtained as the difference as follows. The coefficients a and b of the linear transformation may be obtained at each time t by the least squares method and transmitted to the second drawing unit 26 of the second terminal 20 as information accompanying the second extraction information E2 (t). .. With a = 1, the average pixel value of "G3 (t) + b" obtained by adding the value b to the third drawing information G3 (t) (uniformly added to each pixel position) and the fourth drawing information G4 (t) The value of b may be obtained so that the average pixel values match.
E2 (t) = G4 (t)-a ・ G3 (t) -b

In addition, when suppressing the difference between the third drawing information G3 (t) and the fourth drawing information G4 (t) by the linear conversion by the above coefficients a and b, the primary conversion is changed to the third drawing information G3 (t). It is applied as "a ・ G4 (t) + b" to the 4th drawing information G4 (t) instead of the above, and the 2nd extraction information E2 (t) as a difference is obtained as follows. May be good.
E2 (t) = a ・ G4 (t) + b-G3 (t)

Further, the above coefficients a and b are obtained by obtaining the third drawing information G3 (t) and the fourth drawing information G4 (t) as common values in the entire mask image as a partial region in the image plane. Alternatively, the entire mask image may be divided into a plurality of block areas and then obtained as a value for each block area.

Here, when the third drawing information G3 (t) and the fourth drawing information G4 (t) as the original image are composed of, for example, 8 bits and a pixel value in the range of 0 to 255, they are obtained as a difference image. The second extraction information E2 (t) can usually be composed of pixel values in the range of -255 to +255, which is wider than the initial number of bits. Therefore, in order to quantize the difference value so that it fits in the initial color depth B bpp (bit per pixel) (B is, for example, 8 bits / pixel), the extraction unit 35 suppresses the quantization error. May be quantized, and then the quantized second extraction information E2 (t) _{[quantization]} may be transmitted to the second integration unit 27. Specifically, as shown in the following equation, the quantization step qi that minimizes the quantization error is obtained for the value Pi of the histogram of the difference value (frequency Pi as the number of pixels corresponding to the difference value). Quantization may be performed by (obtaining by an arbitrary existing method such as the greedy method).

In the above equation, int () represents the integerization function and N represents the number of bins in the histogram. Here, further, in order to speed up the calculation of the solution by reducing the degree of freedom of the solution, a constraint limiting the quantization step may be imposed, and the information of the quantization step is the quantized first. 2 Extraction information E2 (t) May be transmitted to the second integration unit 27 as information accompanying _{[quantization]} .

FIG. 6 is a diagram (B = 8) showing a schematic example in which the candidate of the value of the quantization step qi is limited to 1 or 4 (q1 = 1, q2 = 4) as a predetermined combination to minimize the quantization error. (Bits), and in the histogram shown on the upper side, the difference values are distributed in the range of min to max, not the entire range of -255 to +255. The range indicated by the horizontal double-sided arrow on the upper side is the upper q1 * {2 ^B * q2- (max-min)} / (q2-q1) ranges (upper range) of the frequency of the histogram of the difference value. As shown in the graph showing the correspondence between the 8-bit quantization value and the difference value (range -255 to +255), the result of quantization with the quantization error minimized on the lower side is shown in this upper range. The quantization step is finely set to 1, and the quantization step is roughly set to 4 except for the upper range. (Note that the significance of the upper number is as follows. When you want to map pixel values that do not fit in 8-bit 0 to 255 (existing from min to max) to 8-bit 0 to 255, A When the pixel value of is quantized by q1 = 1 and the remaining 255-A pixel values are quantized by q2 = 4, A is maximized in order to minimize the error. Therefore, assuming that the histogram is monotonically decreasing and the pixel value from 0 to max-min is mapped from 0 to 255, y = (1 / q1) * x and y = (1 / q2) * x + 255- (Max-min) / q2 intersections are the maximum A.)

In the second terminal 20, the process of obtaining the second drawing information G2 (t) by the second drawing unit 26 is the process of obtaining the third drawing information G3 (t) by the third drawing unit 33 of the server 30 as described above. Since it is the same as (the drawing quality is also the same), the duplicate description will be omitted.

In the second integration unit 27, the second extraction information E2 (t) is added to the second drawing information G2 (t) which is the same as the third drawing information G3 (t), so that the drawing is performed with high quality. The second integrated information G2S (t) is obtained as the same as the fourth drawing information G4 (t) or as a copy of the fourth drawing information G4 (t). The processing of the second integration unit 27 corresponds to the reverse of the processing of the extraction unit 35 of the server 30.

In the second integration unit 27, when the second extraction information E2 (t) is extracted by using the linear transformation by the coefficients a and b described above, the second extraction information E2 (t) is similarly used in the second. The integrated information G2S (t) should be obtained. Further, when the second extraction information E2 (t) is quantized in the above-mentioned quantization step, the second integration unit 27 determines the corresponding difference value from the quantized value by dequantization, and then the difference. The second extracted information E2 (t) as a value distribution may be obtained, and the second integrated information G2S (t) may be obtained.

The second presentation unit 28 is composed of a display as hardware, and provides the second user with the second integrated information G2S (t) as a drawing of the avatar A1 of the first user U1 obtained by the second integration unit 27. Display against. When the display constituting the second presentation unit 28 is an optical see-through type, it is sufficient to display only the second integrated information G2S (t) which is the drawing result of the avatar, and when the user U2 wears this optical see-through type display. The optical see-through display may be arranged so that the position and orientation of the viewpoint coincide with the position and orientation of the second positioning information P2 (t) to be positioned by the second positioning unit 22. (That is, the second positioning unit 22 sets the position / orientation of the arranged optical see-through display (which matches the position / orientation of the viewpoint of the second user U2 when worn by the second user U2) in the second positioning. Information P2 (t) (position and orientation of the virtual camera for drawing the virtual space V2 of the second user U2) may be positioned.) Similarly, the display constituting the second presentation unit 28 is a video. In the case of the see-through type, the second integrated information G2S (t), which is the drawing result of the avatar, may be superimposed on the background image and displayed, and the background image displayed on this video see-through type display is the second positioning unit 22. It suffices to use a camera taken at the current time t with a camera that matches the position and orientation of the second positioning information P2 (t) to be positioned. (That is, the second positioning unit 22 may position the position and orientation of the camera that captures the background image as the second positioning information P2 (t). The second positioning unit 22 takes an image and takes a second image from this image. When the positioning information P2 (t) is positioned, the image taken by the camera that captures this image may be used as the background image to be displayed on this video see-through display.)

FIG. 7 is a diagram showing a schematic example of each drawing information and the second extraction information, and is a diagram showing the third drawing information G3 (t) and the second drawing information G2 (t) drawn as the same with standard quality. The fourth drawing information G4 (t) drawn with higher quality than these, and the second extraction information E2 (t) as the difference between the third drawing information G3 (t) and the fourth drawing information G4 (t). An example of, is shown. Each drawing information is an example in which only the face part is drawn as the avatar of the first user U1, but the avatar including the body part may be drawn. In the fourth drawing information G4 (t), a directional light source is arranged, and rendering is performed in consideration of reflection and shadow on the surface of the avatar, so that the third drawing information G3 (t) and the second drawing information do not take these into consideration. It is drawn with higher quality than G2 (t).

As described above, according to the presentation system 100 of the present embodiment, in the remote communication using the avatar, the three-dimensional avatar which is the same as or almost the same as the three-dimensional avatar drawn with high quality by using the abundant computational resources of the server 30. Is displayed on the user terminal without drawing directly, enabling remote communication with a sense of reality using a high-quality 3D avatar, and transmitting only the difference obtained from the drawing result of the server 30. It is also possible to suppress the amount of communication between the server 30 and the user terminal.

Hereinafter, various supplements and the like for each embodiment will be described.

(1) As described in the schematic description, in the presentation system 100, information at each time t = 1, 2, 3, ... At a predetermined processing rate is synchronously processed in real time, but the first terminal 10, the second terminal 10, the second. By synchronizing the clock (timekeeping function) in advance in the terminal 20 and the server 30 by an existing method such as a network time protocol, processing can be performed at each common time t. The current time when the second integrated information G2S (t) is finally presented by the second presenting unit 28 is in the future than the time t associated with the second integrated information G2S (t) due to transmission delay or processing delay. Time t + Δt (Δt> 0) may be set.

The first terminal 10, the second terminal 20, and the server 30 synchronize the time t when each information (first recognition information R1 (t), first positioning information P1 (t), second positioning information P2 (t)) is acquired. Then, based on this, the time t is linked as a time stamp, and the second, third, and fourth drawing information G2 (t), G3 (t), G4 (t) and the second extraction information E2 (t), The second integrated information G2S (t) is obtained. After synchronizing the time t in this way, the processing rates may be different from each other in all or a part of the first terminal 10, the second terminal 20, and the server 30.

(2) When the second positioning information P2 (t) including the depth information is obtained in the positioning of the second positioning unit 22, the first user in the second drawing unit 26, the third drawing unit 33, and the fourth drawing unit 34. When drawing the 2nd, 3rd, and 4th drawing information G2 (t), G3 (t), and G4 (t) as the 3D avatar of U1, it is deeper than the depth information in the whole 3D avatar. If there is a part located on the side (far side when viewed from the virtual camera), the part located on the back side may not be drawn. Since the part located on the back side is a part that is shielded by some real object in the environment E2 where the second user U2 exists, by not drawing, a natural drawing result is reflected by the occlusion by the real object. May be obtained. (In addition, depending on the positional relationship, the avatar drawn only partially may be drawn as if it is buried in a real object (for example, inside a wall).)

(3) When displaying the avatar A1 of the first user U1 in the fixed position posture in the virtual space V2 provided to the second user U2 in the remote communication usage setting, each time is displayed in the first positioning unit 12. The process of obtaining the first positioning information P1 (t) in real time with t may be omitted. In this case, assuming that the real-time first positioning information P1 (t) is a constant value (predetermined value given in advance) regardless of the time t, the third drawing unit 33, the fourth drawing unit 34, and the server 30 The processing of the extraction unit 35 and the processing of the second drawing unit 26, the second integration unit 27, and the second presentation unit 28 of the second terminal 20 may be performed in the same manner. (The fixed position and orientation in the virtual space V2 are determined by the constant value and the above-mentioned predetermined conversion T.)

(4) In the above description with reference to FIGS. 2 and 3, as described above in the schematic description, the process of displaying the avatar A1 of the first user U1 to the second user U2 in the virtual space V2 (“first avatar”). Although it was related to "display processing"), the roles of the first terminal 10 and the second terminal 20 were exchanged, and the avatar A2 of the second user U2 was changed to the virtual space V1 for the first user U1 in exactly the same manner. It is also possible to perform a process of displaying within (referred to as "second avatar display process").

FIG. 8 is a functional block diagram of the presentation system 100 according to an embodiment in which the first avatar display process and the second avatar display process are performed in both directions. Since the configuration for performing the first avatar display process in FIG. 8 is the same as that in FIG. 2, duplicated description will be omitted. In FIG. 8, the first terminal 10 includes a first positioning unit 12, a first drawing unit 16, a first integrated unit 17, and a first presenting unit 18 as a configuration for performing a second avatar display process, and is a second terminal. 20 includes a second recognition unit 21 and a second positioning unit 22, and the operation when each of these units performs the second avatar display processing is the second positioning unit 22 in the second terminal 20 during the first avatar display processing, respectively. , The second drawing unit 26, the second integration unit 27 and the second presentation unit 28, and the first recognition unit 11 and the first positioning unit 12 in the first terminal 10 are the same (of the first user to be processed). The information and the information of the second user are exchanged and are exactly the same), so duplicate explanations will be omitted. Since the processing on the server 30 is also the same as the second avatar display processing and the first avatar display processing (the information of the first user to be processed and the information of the second user are exchanged and are exactly the same), they are duplicated. The explanation given is omitted.

(5) When remote communication is performed between the users U1 and U2 using the avatar by the presentation system 100, the voice may be recorded in real time and played back by the other user. When the movement of the mouth of the user U1 is reflected in the first recognition information P1 (t), the avatar A1 of the user U1 speaks in conjunction with the movement of the mouth spoken by the user U1 to the other user U2. Will be displayed, and the spoken content will also be played back as voice.

(6) FIG. 9 is a diagram showing an example of a hardware configuration in a general computer device 70. The first terminal 10, the second terminal 20, and the server 30 in the presentation system 100 can each be realized as one or more computer devices 70 having such a configuration. When each of the first terminal 10, the second terminal 20, and the server 30 is realized by two or more computer devices 70, information necessary for processing may be transmitted and received via the network NW. The computer device 70 is a CPU (central processing unit) 71 that executes a predetermined instruction, and a GPU (graphics calculation device) as a dedicated processor that executes a part or all of the execution instructions of the CPU 71 on behalf of the CPU 71 or in cooperation with the CPU 71. ) 72, RAM73 as the main storage device that provides the work area to the CPU71 (and GPU72), ROM74 as the auxiliary storage device, communication interface 75, display 76, input interface 77 that accepts user input by mouse, keyboard, touch panel, etc. It includes one or more types of sensors 79 that perform sensing and measurement using other than image imaging, such as a camera 78 that captures the environment and the user, and a LiDAR sensor, and a bus BS for exchanging data between them.

Each functional unit of the first terminal 10, the second terminal 20, and the server 30 can be realized by a CPU 71 and / or a GPU 72 that reads a predetermined program corresponding to the function of each unit from the ROM 74 and executes the program. Both CPU71 and GPU72 are a kind of arithmetic unit (processor). Here, when the display-related processing is performed, the display 76 further operates in conjunction with the display 76, and when the communication-related processing related to data transmission / reception is performed, the communication interface 75 further operates in conjunction with the display 76. The first presentation unit 18 and the second presentation unit 28 may output an augmented reality display by realizing the display 76.

100 ... presentation system, 10 ... first terminal, 20 ... second terminal, 30 ... server
11 ... 1st recognition unit, 12 ... 1st positioning unit
22 ... 2nd positioning unit, 26 ... 2nd drawing unit, 27 ... 2nd integrated unit, 28 ... 2nd presentation unit
33 ... 3rd drawing unit, 34 ... 4th drawing unit, 35 ... extraction unit

Claims (13)

The first recognition unit that recognizes the state related to the communication of the first user and obtains the first recognition information,
The second positioning unit that obtains the second positioning information by positioning the position and posture of the second user,
A third drawing unit that reflects the first recognition information and obtains a third drawing information that draws the avatar of the first user from the viewpoint of the virtual camera arranged in the second positioning information.
From the viewpoint of the virtual camera arranged in the second positioning information, the fourth drawing information in which the avatar of the first user is drawn with higher quality than the drawing mode of the third drawing unit is obtained by reflecting the first recognition information. 4th drawing part and
An extraction unit that extracts the difference between the third drawing information and the fourth drawing information as the second extraction information,
From the viewpoint of the virtual camera arranged in the second positioning information, the second drawing information in which the avatar of the first user is drawn with the same quality as the drawing mode of the third drawing unit is obtained by reflecting the first recognition information. 2 drawing part and
By reflecting the second extraction information in the second drawing information, a second integration unit that obtains the second integrated information that is an avatar of the first user as imitating the fourth drawing information, and a second integration unit.
A presentation system including a second presentation unit that displays the second integrated information to the second user. The first recognition unit is provided in the first terminal used by the first user.
The two positioning units, the second drawing unit, the second integration unit, and the second presentation unit are provided in the second terminal used by the second user.
The presentation system according to claim 1, wherein the third drawing unit, the fourth drawing unit, and the extraction unit are provided in a server. The presentation system according to claim 2, wherein the first terminal, the second terminal, and the server are configured to be capable of communicating with each other via a network. The first recognition unit is described in any one of claims 1 to 3, wherein the first recognition unit recognizes a facial expression and / or a pose as a state related to the communication of the first user and obtains the first recognition information. Presentation system. The extraction unit has one of the third drawing information and the fourth drawing information subjected to conversion processing so as to suppress the difference between the third drawing information and the fourth drawing information, and the other. The presentation system according to any one of claims 1 to 4, wherein the difference between the above and the second is extracted as the second extraction information together with the information of the conversion process. The presentation system according to claim 5, wherein the conversion process is a linear conversion or a constant addition. The second extraction unit assumes that each difference value in the pixel difference value map calculated as the difference between the third drawing information and the fourth drawing information is quantized so as to suppress the quantization error. The presentation system according to any one of claims 1 to 6, wherein the extraction information is extracted. The presentation system according to claim 7, wherein the extraction unit limits the quantization step at the time of quantization. When the second positioning unit positions the position and posture of the second user to obtain the second positioning information, the second positioning unit also acquires depth information in the environment in which the second user exists.
In the second drawing unit, the third drawing unit, and the fourth drawing unit, when the avatar of the first user is drawn as the second drawing information, the third drawing information, and the fourth drawing information, respectively. The presentation system according to any one of claims 1 to 8, wherein the portion shielded by the depth information is not drawn. Further, a first positioning unit for positioning the position and posture of the first user and obtaining the first positioning information is provided.
In the second drawing unit, the third drawing unit, and the fourth drawing unit, the avatar of the first user is arranged in a position and posture corresponding to the first positioning information, and the second drawing information, the third drawing unit. The presentation system according to any one of claims 1 to 9, wherein the drawing information and the fourth drawing information are drawn respectively. The first terminal used by the first user and equipped with the first recognition unit,
A second terminal used by a second user and having a second positioning unit, a second drawing unit, a second integrated unit, and a second presentation unit.
A server in a presentation system including a third drawing unit, a fourth drawing unit, and a server including an extraction unit.
The first recognition unit recognizes a state related to the communication of the first user and obtains the first recognition information.
The second positioning unit positions the position and posture of the second user to obtain the second positioning information, and then obtains the second positioning information.
The third drawing unit obtains the third drawing information in which the avatar of the first user is drawn by reflecting the first recognition information in the virtual camera viewpoint arranged in the second positioning information.
The fourth drawing unit draws the avatar of the first user with higher quality than the drawing mode of the third drawing unit, reflecting the first recognition information, from the viewpoint of the virtual camera arranged in the second positioning information. Obtaining the 4th drawing information
The extraction unit extracts the difference between the third drawing information and the fourth drawing information as the second extraction information.
The second drawing unit draws the avatar of the first user with the same quality as the drawing mode of the third drawing unit, reflecting the first recognition information, from the viewpoint of the virtual camera arranged in the second positioning information. Obtaining the second drawing information,
By reflecting the second extraction information in the second drawing information, the second integrated unit obtains the second integrated information which is the avatar of the first user as imitating the fourth drawing information. ,
The second presenting unit is a server characterized in that the second integrated information is displayed to the second user. The first terminal used by the first user and equipped with the first recognition unit,
A second terminal used by a second user and having a second positioning unit, a second drawing unit, a second integrated unit, and a second presentation unit.
A second terminal in a presentation system including a third drawing unit, a fourth drawing unit, and a server including an extraction unit.
The first recognition unit recognizes a state related to the communication of the first user and obtains the first recognition information.
The second positioning unit positions the position and posture of the second user to obtain the second positioning information, and then obtains the second positioning information.
The third drawing unit obtains the third drawing information in which the avatar of the first user is drawn by reflecting the first recognition information in the virtual camera viewpoint arranged in the second positioning information.
The fourth drawing unit draws the avatar of the first user with higher quality than the drawing mode of the third drawing unit, reflecting the first recognition information, from the viewpoint of the virtual camera arranged in the second positioning information. Obtaining the 4th drawing information
The extraction unit extracts the difference between the third drawing information and the fourth drawing information as the second extraction information.
The second drawing unit draws the avatar of the first user with the same quality as the drawing mode of the third drawing unit, reflecting the first recognition information, from the viewpoint of the virtual camera arranged in the second positioning information. Obtaining the second drawing information,
By reflecting the second extraction information in the second drawing information, the second integrated unit obtains the second integrated information which is the avatar of the first user as imitating the fourth drawing information. ,
The second presenting unit is a second terminal characterized in that the second integrated information is displayed to the second user. A program comprising the computer functioning as the server according to claim 11 or the second terminal according to claim 12.

Images