WO2018186169A1 - Video generation device, video generation method, and video generation program - Google Patents

Abstract

A video generation method that generates a video by having a camera in a three-dimensional (3D) pseudo space perform a perspective projection of an object in the 3D pseudo space, and displays the video on a video display device, said video generation method comprising: a field of view setting step in which the field of view of the camera is set on the basis of field of view information that is inputted from the video display device; an interpupillary distance setting step in which an interpupillary distance is set; a video rendering step in which a reference video is rendered on the basis of the field of view and a positional relationship between the object and the camera; and a video shifting step in which two videos are generated by shifting the reference video in a sideways direction on the basis of the field of view, the interpupillary distance, and the positional relationship between the object and the camera.

Description

VIDEO GENERATION DEVICE, VIDEO GENERATION METHOD, AND VIDEO GENERATION PROGRAM

The present invention relates to a video generation device, a video generation method, and a video generation program.
The present application claims priority on Japanese Patent Application No. 2017-075213 filed in Japan on April 5, 2017, the contents of which are incorporated herein by reference.

Recently, in order to realize augmented reality (AR) and mixed reality (MR), wearable devices such as eyeglass-type terminals and smart glasses are attracting attention. Patent Documents 1 and 2 describe a display device that realizes a glasses-type terminal. AR and MR are realized by displaying different images on the left-eye display and right-eye display included in the glasses-type terminal to obtain the effects of convergence and binocular parallax.

JP 2016-1449587 WO2016 / 113951

However, when two images for the left eye and right eye are generated in real time, the load on the processor such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), etc. will increase and processing will be lost. There is a problem that occurs.

An aspect of the present invention has been made in view of such circumstances, and an object thereof is to provide a video generation method and a video generation program capable of realizing AR and MR while suppressing the load on the processor.

In order to solve the above-described problems, a video generation method and a video generation program according to an aspect of the present invention are configured as follows.

A video generation method according to an aspect of the present invention is a video generation method in which an object in a 3D (Three Dimensions) pseudo space is projected by a camera in the 3D pseudo space to generate a video and displayed on a video display device. A visual field setting process for setting the visual field of the camera based on visual field information input from the video display device, an interpupillary distance setting process for setting an interpupillary distance, the visual field, the object, and the camera. By shifting the reference image in the horizontal direction based on the image rendering process for rendering the reference image based on the positional relationship, the visual field, the inter-pupil distance, and the positional relationship between the object and the camera, A video shift process for generating the video.

Also, in the video generation method of the present invention, the video rendering process measures the load of the processor, and when the load falls below a threshold value, the camera is located at a position away from the camera by the interpupillary distance in the 3D pseudo space. Is added, and the video is generated by performing perspective projection with each of the cameras, and the video shift process is skipped.

In the video generation method of the present invention, when the number of objects in the 3D pseudo space is two or more, the video rendering process adds a camera at a location separated from the camera by the interpupillary distance in the 3D pseudo space. Then, the video is generated by performing perspective projection with each of the cameras, and the processing of the video shift process is skipped.

The video generation program according to an aspect of the present invention causes a computer to execute the above-described video generation method.

An image generation apparatus according to an aspect of the present invention is an image generation apparatus that generates an image by perspectively projecting an object in a 3D (Three) pseudo space by a camera in the 3D pseudo space, and displays the image on a video display device. A visual field setting unit that sets a visual field of the camera based on visual field information input from the video display device, an interpupillary distance setting unit that sets an interpupillary distance, the visual field, the object, and the camera. A video rendering unit that renders a reference video based on a positional relationship; and, by shifting the reference video in the horizontal direction based on the visual field, the inter-pupil distance, and the positional relationship between the object and the camera, A video shift unit for generating the video.

According to one aspect of the present invention, the video generation method can generate a video that can be AR displayed while reducing the load on the processor.

It is a schematic block diagram which shows the structure of the video generation apparatus which concerns on 1st Embodiment. It is a figure which shows an example of 3D pseudo | simulation space in 1st Embodiment. It is a figure which shows an example of the relationship between the image | video and the eyes which the image | video production | generation method in 1st Embodiment outputs. It is a figure which shows an example of 3D pseudo | simulation space in 2nd Embodiment. It is a figure which shows an example of 3D pseudo | simulation space in 2nd Embodiment.

(First embodiment)
FIG. 1 is a diagram illustrating an example of a video generation apparatus 10 according to the present embodiment. FIG. 1 also shows the video display unit 11. The video display unit 11 is provided in a video display device such as a glasses-type terminal, a smart glass, or a head-mounted display. The video generation device 10 may be provided in the video display device, or may be provided in a terminal device that can be connected to the video display device such as a smartphone. As shown in FIG. 1, a video generation device 10 (video generation method) in this embodiment includes a visual field setting unit (visual field setting process) 101, an interpupillary distance setting unit (interpupillary distance setting process) 102, and a video rendering unit ( A video rendering process (103) 103 and a video shift unit (video shift process) 104 are included.

The visual field setting unit 101 sets the size of the visual field (FoV: “Field Of View”) used in the video rendering unit 103 based on the visual field information input from the video display unit 11. The field of view is, for example, an angle in the vertical direction when the camera in the 3D pseudo space performs perspective projection. The visual field setting unit 101 outputs the set visual field value to the video rendering unit 103. The visual field information is, for example, the visual field value itself. The visual field value may be a vertical value, a diagonal value, or a horizontal value. Further, the visual field value may be one or more values among a vertical value, a diagonal value, and a horizontal value. The visual field information is, for example, an ID such as a vendor ID or a product ID of the video display device provided with the video display unit 11. The video generation apparatus 10 can store the ID and the field-of-view value in association with each other. The video generation apparatus 10 can have a table (database) that associates IDs and field-of-view values. The video generation visual field setting unit 101 can determine the visual field value based on this ID. The field-of-view setting unit 101 can determine the value of the field of view by, for example, comparing this ID with its own database. The field-of-view setting unit 101 can, for example, inquire a server on the Internet for this ID and receive a field-of-view value. For example, the viewing angle setting unit 101 can use the table (database) on the server.

The interpupillary distance setting unit 102 sets the interpupillary distance (PD) and outputs it to the video rendering unit 103 and the video shift unit 104. The interpupillary distance may be a fixed value or may be manually input by the user. Alternatively, the interpupillary distance setting unit 102 may set the interpupillary distance measured by a sensor provided in the video display unit 11 or a wearable sensor.

The video rendering unit 103 renders video by the camera shooting an object in the 3D pseudo space. At this time, the visual field of the camera can be set to the visual field value input from the visual field setting unit 101.

The video shift unit 104 receives display information from the video display unit 11. This display information may be obtained from the Internet. The video shift unit 104 shifts the video input from the video rendering unit 103 based on the interpupillary distance input from the interpupillary distance setting unit 102 and the display information, and converts the video into two videos. The video shift unit 104 outputs the two generated videos to the video display unit 11. Note that the video before the shift can be referred to as a reference video.

FIG. 2 shows an example of the 3D pseudo space used by the video rendering unit 103. 20 is a view of the 3D pseudo space viewed from the height direction. 21 is a view of the 3D pseudo space as viewed from the side. For example, the north direction of the 3D pseudo space can be the z axis, the height direction can be the y axis, and the east direction can be the x axis. In this case, 20 is a diagram of the 3D pseudo space viewed from the plus direction of the y axis, and 21 is a diagram of the 3D pseudo space viewed from the plus direction of the x axis. The camera 201 can generate an image by perspective projection based on the positional relationship between itself and the object 202 in the 3D pseudo space and the field of view set in the field of view setting unit 101. In this example, the object 202 is a rectangular parallelepiped. This perspective projection can be performed based on the field of view 211. The visual field 211 can be the visual field set in the visual field setting unit 101. In this example, the vertical visual field is set, but the horizontal visual field can be set from the display size and the aspect ratio of the video display unit 11. Reference numeral 203 denotes the distance between the camera 201 and the object 202.

FIG. 3 is a diagram illustrating an example in which the video shift unit 104 shifts the image generated by the video rendering unit 103. Reference numerals 301 and 311 denote displays of the video display unit 11. Reference numeral 302 denotes a horizontal center line of the display 301, and reference numeral 312 denotes a center line of the display 311. 303 represents the user's left eye and 313 represents the user's right eye. 304 represents the center line of the left eye 303, and 314 represents the center line of the right eye 313. Reference numeral 305 denotes an object corresponding to 202 in FIG. 2 rendered by the video rendering unit 103. In the case of 20 in FIG. 2, an image in which the object 305 is displayed on the center line 302 is generated on the display 301, and the image shift unit 104 applies a rightward shift thereto. Similarly, the video shift unit 104 adds a leftward shift to the video displayed on the display 311. Thereby, the user can obtain the effect of congestion so that the object 306 exists at a position away from the user by the distance 203. The shift amount can be performed based on the visual field set by the visual field setting unit 101, the inter-pupil distance set by the inter-pupil distance setting unit 102, and the distance between the object in the 3D pseudo space rendered by the video rendering process 103 and the camera.

FIG. 4 is a diagram in which the setting of the 3D pseudo space in FIG. 2 is changed. The object 402 is obtained by moving the object 202 in the horizontal direction. Reference numeral 403 denotes the distance between the camera 201 and the object 402. Reference numeral 404 denotes a line in the same direction as the horizontal direction of the camera 201 and passes through the object 402. Reference numeral 405 denotes the length of a perpendicular line taken from the camera 201 to the line 404. If the value of the length 405 is x, for example, the shift amount can be determined using a polynomial related to x. Specifically, the shift amount y can be y = a _n (xx ₀ ) ⁿ +... A ₁ (xx ₀ ) + a ₀ . x ₀ and the coefficient a _i is a value determined by the image display device 11 can be known from the output obtained by inputting the information of the image display device 11 in the database. The database may be in the video generation device 10. The database may be on the server.

As described above, according to the present embodiment, the video rendering unit 103 performs rendering once, and generates two videos that can obtain the effect of congestion based on the copy, thereby reducing the load on the processor. can do.

Note that the video shift performed by the video shift unit 104 may be performed when the processor load exceeds a threshold. When the load does not exceed the threshold, the video rendering process 103 can prepare two cameras for the left eye and the right eye in the 3D pseudo space, and render two videos. An image generated by the left-eye camera can be displayed on the display 301 of FIG. 3, and an image generated by the right-eye camera can be displayed on the display 311 of FIG. In this case, the processing of the video shift unit 104 is skipped. The video rendering unit 103 may calculate the processor load. (Second Embodiment)
In the first embodiment, the video generation device 10 performs rendering using one camera in the 3D pseudo space, and generates a video with congestion by shifting the rendered video in the horizontal direction. In the present embodiment, a case will be described in which a plurality of target objects exist in the 3D pseudo space.

FIG. 5 is an example in which an object 501 further exists in the 3D pseudo space of FIG. In such a case, if a video is generated by a lateral shift based only on the positional relationship between the camera 201 and the object 202, the sense of distance between the camera 201 and the object 501 is not correct. In order to avoid this, for example, when the video rendering unit 103 in FIG. 1 detects that the number of objects in the 3D pseudo space is two or more, it sets not the camera 201 but also a left eye camera and a right eye camera. Based on these, a left-eye video and a right-eye video can be generated. By doing so, the sense of distance of the object 501 can be correctly expressed.

Alternatively, the object 202 and the object 501 can include depth on information. Specifically, the video rendering unit 103 in FIG. 1 can determine whether to use the left-eye camera and the right-eye camera based on whether the number of objects for which depth-on information is set is one. For example, in FIG. 5, when depth on information is set for the object 202 and depth on information is not set for the object 501, the video rendering unit 103 renders a video based on the camera 201, and The shift unit 104 shifts the video, thereby generating a left-eye video and a right-eye video. By doing so, the load on the processor can be reduced.

Note that a program that operates in the video generation device, the video generation method, and the video generation program according to one aspect of the present invention is a program that controls a CPU or the like so as to realize the functions of the above-described embodiments according to one aspect of the present invention. (A program that causes a computer to function). Information handled by these devices is temporarily stored in the RAM at the time of processing, then stored in various ROMs and HDDs, read out by the CPU, and corrected and written as necessary. As a recording medium for storing the program, a semiconductor medium (for example, ROM, nonvolatile memory card, etc.), an optical recording medium (for example, DVD, MO, MD, CD, BD, etc.), a magnetic recording medium (for example, magnetic tape, Any of a flexible disk etc. may be sufficient. In addition, by executing the loaded program, not only the functions of the above-described embodiment are realized, but also based on the instructions of the program, the processing is performed in cooperation with the operating system or other application programs. The functions of the invention may be realized.

Also, when distributing to the market, the program can be stored and distributed on a portable recording medium, or transferred to a server computer connected via a network such as the Internet. In this case, the storage device of the server computer is also included in one embodiment of the present invention. In addition, part or all of the video generation method and the video generation program in the above-described embodiments may be realized as an LSI that is typically an integrated circuit. Each functional block of the receiving apparatus may be individually formed as a chip, or a part or all of them may be integrated into a chip. When each functional block is integrated, an integrated circuit controller for controlling them is added.

Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to progress in semiconductor technology, an integrated circuit based on the technology can also be used.

Note that the present invention is not limited to the above-described embodiment. It goes without saying that the video generation method of the present invention is not limited to application to eyeglass-type terminals, but can be applied to portable devices, wearable devices, and the like.

The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and the design and the like within the scope not departing from the gist of the present invention are also claimed. Included in the range.

One embodiment of the present invention is suitable for use in a video generation method and a video generation program. One embodiment of the present invention is used in, for example, a communication system, a communication device (for example, a mobile phone device, a base station device, a wireless LAN device, or a sensor device), an integrated circuit (for example, a communication chip), a program, or the like. be able to.

DESCRIPTION OF SYMBOLS 10 Image | video production | generation part 11 Image | video display part 101 Visual field setting part 102 Interpupillary distance setting part 103 Image | video rendering part 104 Image | video shift part 20 3D pseudo space (height direction)
21 3D pseudo space (horizontal direction)
201 Camera 202 Object 203 Distance 301, 311 Display 302, 312 Display center line 303, 313 eyes 304, 314 Center lines 305, 306 Object 402 Object 403 Distance 404 Line 405 passing through the object in the same direction as the horizontal direction of the camera Distance 501 object

Claims (6)

An image generating method for generating an image in a 3D (Three Dimensions) pseudo space by a perspective projection of a camera in the 3D pseudo space and displaying the image on a video display device,
A visual field setting process for setting the visual field of the camera based on visual field information input from the video display device;
An interpupillary distance setting process for setting the interpupillary distance;
A video rendering process for rendering a reference video based on the visual field and a positional relationship between the object and the camera;
Based on the visual field, the interpupillary distance, and the positional relationship between the object and the camera, a video shift process for generating two images by shifting the reference image in a horizontal direction;
A video generation method comprising: The video rendering process measures the processor load,
If the load is below a threshold, add a camera in the 3D pseudospace at a distance from the camera by the interpupillary distance;
Generate the video by performing perspective projection with each of the cameras,
Skip the video shift process;
The video generation method according to claim 1. In the video rendering process, when the number of objects in the 3D pseudo space is 2 or more, a camera is added to the 3D pseudo space at a location separated from the camera by the interpupillary distance,
Generate the video by performing perspective projection with each of the cameras,
Skip the video shift process;
The video generation method according to claim 1. When the video rendering process is an object in the 3D pseudo space and the number of objects for which depth-on information is set is two or more, the camera is placed at a location separated from the camera by the interpupillary distance in the 3D pseudo space. Add
Generate the video by performing perspective projection with each of the cameras,
Skip the video shift process;
The video generation method according to claim 1. A video generation program for causing a computer to execute the video generation method according to claim 1. A video generation apparatus that generates an image by causing a camera in the 3D pseudo space to project an object in a 3D (Three Dimensions) pseudo space, and displays the image on a video display device.
A visual field setting unit for setting the visual field of the camera based on visual field information input from the video display device;
An interpupillary distance setting unit for setting an interpupillary distance;
A video rendering unit for rendering a reference video based on the visual field and the positional relationship between the object and the camera;
Based on the visual field, the interpupillary distance, and the positional relationship between the object and the camera, a video shift unit that generates two images by shifting the reference image in the horizontal direction;
A video generation apparatus comprising:

Images