JP2017098921A - Transmission device, transmission method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the amount of data for a display image to be transferred according to the movement of an observer. SOLUTION: A transmitting device for transmitting image data according to a line of sight of an observer to a receiving device having a display unit, an acquiring unit for acquiring a line of sight of the observer, and luminance information from all directions. Holding means for holding an environment map consisting of: extracting means for extracting partial image data of the visual field range from the environment map with reference to a visual field range corresponding to a line-of-sight direction; A projecting unit that performs projective transformation according to the unit and generates display image data, and compares a data amount of the partial image data with a data amount of the display image data, and compares the partial image data with the display image data. Determining means for determining which of the data is to be transmitted to the receiving device; and one of the partial image data and the display image data based on a result determined by the determining means. And transmitting means for transmitting. [Selection] Fig. 6

Description

  The present invention relates to a technique for efficiently transferring image data to be displayed on a head mounted display or the like.

  A technique for displaying an image in accordance with the viewing direction of the observer using a head-mounted display (hereinafter, HMD) fixed to the observer's head is known. The image data displayed on the HMD is an image of an area obtained by cutting out the video in the entire peripheral direction each time, and the original image is also called an environment map. The environment map can be acquired by, for example, photographing using a fisheye lens or a method of capturing light reflected by a mirror ball with a camera. When a high-resolution environment map is acquired, the amount of data increases.

  It is known that an environment map is held in a server terminal, and only a necessary area of the environment map requested by a user is cut out and transmitted to a client terminal to which an HMD is connected. In order to display a display image naturally in accordance with the viewing direction of the observer, it is desired to prevent a delay in transferring data.

  Patent Document 1 discloses a method of changing the compression rate for compressing data in accordance with the degree of distortion when acquiring an environment map in consideration of the camera characteristics specific to the fisheye lens that acquired the environment map. Thereby, the data amount of the environment map can be compressed with high image quality.

JP 2011-55366 A

  However, in the method of Patent Document 1, compression is performed at a high compression ratio in a region with a large distortion, whereas in an environment map taken with a camera with a small distortion and a lens with good lens performance, the data amount is reduced. Can not.

  Therefore, an object of the present invention is to reduce the amount of data for a display image to be transferred according to the movement of an observer.

In order to solve the above problems, the present invention is a transmission device that transmits image data corresponding to the line-of-sight direction of an observer to a reception device having a display unit, and an acquisition unit that acquires the line-of-sight direction of the observer;
A holding means for holding an environment map composed of luminance information from all directions, an extraction means for extracting partial image data of the visual field range from the environmental map with reference to a visual field range corresponding to the line-of-sight direction, and the part The image data is subjected to projective transformation in accordance with the display unit to generate display image data, the data amount of the partial image data is compared with the data amount of the display image data, and the partial data A determination unit that determines which of image data and display image data is to be transmitted to the receiving device, and one of the partial image data and the display image data based on a result determined by the determination unit Transmitting means for transmitting.

  In the present invention, it is possible to reduce the amount of data for a display image to be transferred according to the movement of the observer.

The block diagram which shows the structure of the image processing system which displays an image on HMD. The figure which shows the structure of an environment map. The figure which shows the structure of HMD. The figure explaining the relationship between an environmental map and a visual field. The figure which shows notionally the weekly report which displays an image on HMD. The flowchart which shows the delivery process of an environment map. The flowchart which shows the delivery process of an environment map. The block diagram which shows the detailed structure of a server terminal and a client terminal. The block diagram which shows the detailed structure of a server terminal and a client terminal.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

<First Embodiment>
In this embodiment, image data is efficiently transferred between a server terminal that is a transmission side that stores an environment map and transmits an image, and a client terminal that receives and displays a part of the environment map according to the line-of-sight direction. A distribution method will be described. FIG. 1 is a block diagram showing a system configuration of an image processing apparatus applicable to the first embodiment. A server terminal (transmission device) that transmits partial image data cut out from the environment map includes a CPU 101, a main memory 102, a storage device 103, a GPU 104, a communication device 105, and a bus 106. First, the CPU 101 executes arithmetic processing and various programs. The main memory 102 provides the CPU 101 with programs, data, work areas, and the like necessary for processing. The storage device 103 is a device that stores an image processing program and a plurality of pieces of image data. For example, a hard disk is used. The GPU 104 is a device for displaying image data and a GUI, and is normally connected to a display. The communication device 105 is a device connected to various networks, and distributes partial image data generated by cutting out from the environment map to the client terminal via the communication device 105. The bus 106 is a bus that connects the above-described units.

  A client terminal (receiving device) that receives part of the environment map data from the server terminal has the same basic configuration as the server terminal, but an HMD (head mounted display) 107 is connected as an image display means. The client terminal transmits the line-of-sight direction of the observer detected by the HMD to the server terminal, and receives partial image data of the environment map corresponding to the line-of-sight direction from the server terminal. The client terminal performs image processing on the received partial image data as necessary, and displays a display image.

  FIG. 2 is a diagram for explaining the environment map. As shown in FIG. 2A, the line-of-sight direction from the observer's viewpoint position 201 can be expressed using polar coordinates (θ, φ). Image data obtained by quantizing the polar coordinates and mapping the luminance at each polar coordinate is an environment map. Each pixel constituting the environment map holds a luminance vector, the magnitude of the luminance vector represents a luminance value, and the direction of the luminance vector always faces the viewpoint position 201. In other words, the environment map is luminance information from the entire periphery that enters the observer. The environmental map holding method includes an equirectangular cylinder map method shown in FIG. 2B and a spherical map method shown in FIG. The equirectangular map method is a method in which θ is a horizontal axis and φ is a vertical axis and is developed on a plane. However, distortion occurs because spherical data is projected onto the plane. The spherical map method is data in which the axis extending from the center of the circle to the arc is φ and the rotation direction from the center of the circle is θ, but in this case as well, distortion occurs because the sphere is projected in two dimensions. For this reason, in both methods, when the HMD is displayed, it is necessary to perform projective conversion on the partial image of the cut-out area.

  FIG. 3 is a diagram for explaining the configuration of the HMD 107. As shown in FIG. 3A, the HMD 107 is equipped with a display unit 301 for displaying an image and a three-axis acceleration sensor 303 for detecting the viewing direction of the observer. In addition, there is a fixing portion 302 for fixing to the head, and the observer can attach the HMD 107 to the head by putting the fixing portion 302 on the ear. As shown in FIG. 3B, the display unit 301 displays an image obtained by viewing the virtual visual field 307 from the observer's eye 304 on the liquid crystal display 306 included in the display unit 301. At this time, the focal position is adjusted by passing the lens 305. By displaying the display image generated based on the partial image of the region cut out from the environment map on the liquid crystal display 306, it is possible to reproduce the video with a sense of presence as if it were at the viewpoint position 201 of the environment map.

FIG. 4 is a diagram for explaining the relationship between the virtual visual field and the environment map. As shown in FIG. 4A, the line-of-sight direction viewed from the viewpoint position 401 is defined as a line-of-sight vector V (θ, φ). At this time, an area (field range 402) covering the virtual visual field 307 that is assumed to be viewed from the viewpoint position 401 around the line-of-sight vector V (θ, φ) is represented by four vectors V ₀ (θ, φ), It is defined by V ₁ (θ, φ), V ₂ (θ, φ), and V ₃ (θ, φ). FIG. 4B shows the relationship between the environment map and the four vectors that define the field-of-view range. FIG. 4B illustrates the case of two different viewing directions. Assume that the visual field range defined by the four vectors centered on the line-of-sight vector V (θ, φ) is a rectangular region as shown in FIG. 4B. This field-of-view range is an area to be displayed on the display unit 301 of the HMD 107 according to the line-of-sight direction. Further, the rectangular area that becomes the visual field range varies depending on the direction of the line-of-sight vector V (θ, φ). When the origin in the environment map is (θ, φ) = (0, 0), the size of the visual field range is minimized when the angle information φ = 0, and the rectangular area increases as φ increases. The size S _{env of the} rectangular area can be calculated using the formula shown in FIG. Here, Bit is the number of bits of each pixel in the environment map. A display image can be obtained by executing projective transformation processing in consideration of the optical characteristics of the eyepiece on the rectangular area. The size of the display image is represented by the formula shown in FIG. The size of the display image is S _display obtained by multiplying the number of pixels of the liquid crystal display 306 (W _LCD × H _LCD ) by the number of bits that can be displayed, and is always constant. Therefore, when distributing the partial image data to the client terminal, the server terminal distributes the smaller amount of the partial image data and the display image data in the environment map, thereby reducing the data transfer amount as much as possible. it can.

FIG. 5 is a diagram for explaining a flow of displaying an environment map using the HMD. The server terminal reads the environment map from the storage device 103. The client terminal acquires angle information (θ, φ) 520 from an acceleration sensor built in the HMD 107. This angle information is the line-of-sight vector V (θ, φ) described with reference to FIG. The client terminal uses the angle information 520 and a field of view range corresponding to the angle information (four vectors V ₀ (θ, φ), V ₁ (θ, φ), V ₂ (θ, φ), V ₃ (θ, φ)). Is transmitted to the server terminal. Based on the information received from the client terminal, the server terminal extracts data necessary for display from the environment map developed in the main memory 102 and transfers the data to the client terminal.

  FIG. 8 is a block diagram showing a detailed logical configuration of each of the server terminal and the client terminal. In the server terminal, from the holding unit 901 that holds the environment map, the image cutout unit 802 cuts out partial image data of an area corresponding to the acquired angle information and field-of-view range. The determination unit 803 determines whether the cut partial image data is transferred to the client terminal as it is, or is converted into a display image and then transferred to the client terminal. The partial image output 804 outputs the cut partial image data to the client terminal as it is. On the other hand, the projective conversion unit 805 performs projective conversion on the extracted partial image data in accordance with the liquid crystal display 306 of the HMD 107. Information necessary for projective transformation is acquired from the client terminal in advance.

  Here, the client terminal also has a projective transformation unit 809. Similar to the projection conversion unit 805, the shielding conversion unit 809 performs projection conversion that matches the spray 306 with the liquid crystal of the HMD 107. The projective transformation unit 809 performs projective transformation only when the partial image data cut out from the server terminal is received as it is. The display control unit 810 outputs the display image obtained by the conversion by the projection conversion unit 809 in the client terminal or the display image acquired from the server terminal to the liquid crystal display 306 of the HMD 107 for display.

  FIG. 6 is a flowchart showing a flow of data transfer between the server terminal and the client terminal in the first embodiment. The server terminal and the client terminal implement each process by the CPU in each terminal executing a computer program that implements the process shown in the flowchart.

  In step S <b> 601, the image holding unit 801 reads the environment map viewed by the observer from the storage device 103 and develops it in the main memory 102. Here, when a plurality of environment maps are held in the storage device 103, the observer may select a mode he / she wants to view from among the modes associated with the environment map.

  In step S <b> 602, the angle information acquisition unit 807 acquires the information in which the acceleration sensor 303 built in the HMD 107 detects the direction in which the viewer's head is facing, and determines the viewer's line-of-sight direction as the angle information (θ , Φ).

In step S603, the visual field range calculation unit 808 of the client terminal specifies the visual field range based on the acquired angle information. Here, the visual field range is a rectangular area centered on the position indicated by the angle information as shown in FIG. Further, the rectangular area specified as the visual field range varies in size according to the size of φ. In this way, the four vectors V ₀ (θ, φ), V ₁ (θ, φ), V ₂ (θ, φ), and V ₃ (θ, φ) that specify the visual field range are transferred to the server terminal.

In step S604, the determination unit 803 determines the data amount S _env of the partial image data cut out from the environment map based on the angle information that is the line-of-sight vector from the client terminal and the visual field range specified by the four vectors, as illustrated in FIG. It calculates according to the formula shown in.

In step S605, the image cutout unit 802 extracts partial image data of the visual field range from the environment map based on the four vectors that specify the visual field range. Step decision unit 803 in S606 compares the data amount _{S display} of the display image to be displayed on the liquid crystal display 306 of the data amount _{S env} and HMD107 of clipped partial image data. Note that the determination unit 803 acquires the number of pixels of the display image on the liquid crystal display 306 and the number of bits that can be displayed by each pixel from the client terminal in advance. The data amount S _display of the _display image is calculated according to the formula shown in FIG. When the data amount S _env of the cut out partial image data is large, the process proceeds to step S607, and when it is small, the process proceeds to step S608.

In step S <b> 607, the projection conversion unit 805 performs projection conversion for displaying the HMD on the extracted partial image data, and generates display image data. In step S610, the display image output unit 806 transfers the display image data to the client terminal. At this time, the data amount S _display of the data to be transferred is smaller than the data amount S _env of the cut out partial image data. As a result, the amount of data transferred from the server terminal to the client terminal can be further suppressed.

  On the other hand, in step S608, the partial image output unit 804 transfers the cut partial image data corresponding to the visual field range to the client terminal as it is without performing projective conversion. As a result, the amount of data to be transferred can be suppressed rather than the data being transferred after projective conversion to the display image.

  In step S609, the projective transformation unit 809 of the client terminal performs projective transformation on the partial image data received from the server device. In step S <b> 611, the display control unit 810 displays the display image data on the liquid crystal display 306 of the HMD 107.

  As described above, in the present embodiment, the data format for transferring image data from the server terminal to the client terminal differs according to the viewing direction of the observer. Of the partial image data cut out from the environment map without projective conversion and the display image data obtained by projective conversion into the cut out partial image data, the smaller data amount is transferred to the client terminal. As a result, the amount of data communication from the server terminal to the client terminal can be reduced, and the delay in image display that occurs due to the large amount of data can be suppressed.

Second Embodiment
In the second embodiment, a preferred embodiment in which a compression process for compressing data transferred from the server terminal to the client terminal is added will be described. Note that the system configuration in the second embodiment is the same as that in the first embodiment, and a description thereof will be omitted.

  FIG. 9 is a block diagram showing a detailed logical configuration of each of the server terminal and the client terminal in the second embodiment. The processing units 901 to 910 are the same as 801 to 810 in FIG. The lossless compression unit 911 performs predetermined lossless compression on the partial image data obtained by cutting out the environment map. The lossless compression is used because the projective transformation for generating the display image has not been performed, and if data loss occurs due to the compression, the image quality may be deteriorated by the projective transformation. On the other hand, the irreversible compression unit 912 performs predetermined irreversible compression on the display image data obtained from the projective transformation unit 905. The lossy compression generally has a higher compression rate than the lossless compression method. Since the display image data has undergone projective transformation, lossy compression is executed with priority given to compression efficiency.

  FIG. 7 is a flowchart showing a flow of data transfer and processing between a server terminal and a client terminal applicable to the second embodiment. The server terminal and the client terminal implement each process by the CPU in each terminal executing a computer program that implements the process shown in the flowchart.

Steps S701 to S705 are the same as the processes from step S601 to step S605 in the first embodiment. Determination unit 903 in S706 calculates the data amount S _L when compressed by a predetermined lossless compression method on the partial image data cut out from the environment map. By the compression algorithm lossless compression unit 911 performs, since rough data can be predicted beforehand, using the predicted value as the data amount S _L. Note that the determination unit 903 may acquire the data amount calculated by the lossless compression unit 911 after the lossless compression unit 911 actually executes the lossless compression.

Step decision unit 903 in S707 calculates the data amount S _C when irreversible compression unit 912 with respect to the display image data is lossy compression. As in the case of lossless compression, the approximate size can be predicted in advance by the compression algorithm executed by the lossy compression unit 912. The determination unit 903 holds the parameters of the irreversible compression executed by the irreversible compression unit 912 in advance and predicts the data amount. Here, the determination unit 903 may acquire the data amount calculated by the lossy compression 911 after the lossy compression unit 912 actually executes the lossy compression.

Determination unit 903 in step S708 compares the data amount _{S L} and the amount of data _{S C.} If larger amount of data S _L when the partial image data and lossless compression proceeds to S709, if it is smaller, the process proceeds to S712. In step S <b> 709, the projective conversion unit 905 performs projective conversion for displaying on the HMD 107 on the extracted partial image data. Thereby, image data for display in the HMD is generated. In step S710, the lossy compression unit 912 applies lossy compression to the display image data. In step S711, the display image output unit 906 transmits the display image data subjected to lossy compression to the client terminal.

  In step S712, the lossless compression unit 911 applies lossless compression to the cut out partial image data. In step S713, the partial image output unit 904 transmits the reversibly compressed partial image data to the client terminal. In S714, the decoding unit 913 of the client terminal decodes the received compressed data. Further, the projective transformation unit 909 performs a projective transformation for display on the partial image data obtained by decoding.

  In step S <b> 715, the display control unit 910 outputs the display image data to the HMD 107 and displays it on the liquid crystal display 306. At this time, when the display image data is received from the server terminal, the decoding unit 914 decodes the compressed data.

  As described above, in the second embodiment, it is possible to reduce the communication amount of data transferred from the server terminal to the client terminal.

<Other embodiments>
In the above-described embodiment, the data amount of data to be transferred is calculated according to the calculation formula each time. However, the data amount of the image is not calculated each time, and the lookup is configured by a combination of the input value and the calculation result in advance. You may use a table. Since the size of the partial image data corresponding to the visual field range depends on the φ direction, a lookup table is created based on the result of comparing the data amount of the partial image data for each φ direction with the data amount of the display image data. create. When the angle information φ is received from the client terminal as the angle information, it is possible to determine which of the cut partial image data and the projective transformed image data should be transferred by referring to the lookup table. The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

802 Image cutout unit 803 Determination unit 804 Partial image output unit 805 Projection conversion unit 806 Display image output unit

Claims (7)

A transmission device that transmits image data according to the line-of-sight direction of an observer to a reception device having a display unit,
An acquisition means for acquiring an observer's gaze direction;
Holding means for holding an environment map consisting of luminance information from all directions;
Extracting means for extracting partial image data of the visual field range from the environment map with reference to the visual field range according to the line-of-sight direction;
Generating means for performing projection conversion according to the display unit on the partial image data and generating display image data;
A determination unit that compares the data amount of the partial image data with the data amount of the display image data and determines which of the partial image data and the display image data is transmitted to the receiving device;
A transmission apparatus comprising: a transmission unit configured to transmit one of the partial image data and the display image data based on a result determined by the determination unit.   The transmission device according to claim 1, wherein the determination unit calculates a data amount of the partial image data based on a size of the visual field range and a bit number of each pixel in the environment map.   The determination unit calculates a data amount of the display image data based on a size of an image displayed on the display unit and a number of bits that can be displayed on the display unit. Or the transmission apparatus of 2. A first compression means for performing lossless compression on the partial image data;
Second compression means for performing lossy compression on the display image data,
The transmission means transmits the compressed data obtained from the first compression means when transmitting the partial image data, and is obtained from the second compression means when transmitting the display image data. The transmission apparatus according to claim 1, wherein the transmission apparatus transmits compressed data.   5. The transmission device according to claim 1, wherein the field-of-view range is larger in size as the line-of-sight direction is larger from the origin in the environment map.   A computer program that causes a computer to function as the transmission device according to claim 1 by being read and executed by a computer. A transmission method for transmitting image data corresponding to a viewing direction of an observer to a receiving device having a display unit,
Get the observer ’s gaze direction,
Holds an environment map consisting of luminance information from all directions,
With reference to the visual field range according to the line-of-sight direction, the partial image data of the visual field range is extracted from the environment map,
Projective transformation according to the display unit is performed on the partial image data, and display image data is generated,
Compare the data amount of the partial image data and the data amount of the display image data, determine whether to transmit one of the partial image data and the display image data to the receiving device,
A transmission method characterized by transmitting either the partial image data or the display image data based on the determined result.

Images