JP2012079211A - Display system and display method of stereoscopic image - Google Patents

Abstract

An image displayed on a stereoscopic display is operated from a portable device.
A stereoscopic image display system and display method includes a portable device, a stereoscopic display unit that displays a stereoscopic image including a virtual object corresponding to a viewpoint position, and information connected to the portable device via a wireless network. And processing equipment. The portable device accepts an operation by the user from a touch panel included in the display unit included in the portable device, and transmits the accepted operation to the information processing device via the wireless network. The information processing device interprets the operation received from the mobile device in accordance with the mode corresponding to the solid angle formed by the display surface of the mobile device display unit and the display surface of the 3D display unit, and results of interpretation Accordingly, the state of the virtual object displayed on the stereoscopic display unit is updated.
[Selection] Figure 1

Description

  The present invention relates to a display system and a display method that enable an operation on a stereoscopic image.

  There is an augmented reality (AR) technique in which the spatial coordinates presented by the display are projected onto spatial coordinates outside and around the display. In this technology, the three-dimensional positional relationship between the camera and the marker is analyzed from the orientation and shape of the image marker reflected on the camera, and the video is synthesized and presented on the display in accordance with the parameters of the three-dimensional viewpoint position. Patent Document 1 describes a marker presentation method used in such a technique.

  A stereo stereoscopic display is used for intuitive display of the three-dimensional viewpoint position. At this time, as a method of performing an intuitive operation on a three-dimensional space, Patent Document 2 shows an example of an instrument for selecting an object three-dimensionally.

JP 2010-32282 A JP 2006-302101A

  The image used for viewing a 3D video scene includes a composition that looks down on all objects in the scene including itself (hereinafter referred to as the third person viewpoint) and a composition that is viewed from the position of the person (hereinafter referred to as the first person viewpoint). There are two types.

  In general, using a first-person viewpoint, that is, a composition from the viewpoint position, the user's visual environment can be intuitively understood. A method of extending the operation of the user viewpoint by the self-position identification method found in Patent Document 1 is shown. However, in such an environment, the instruction operation to the object is mainly defined by the direction from the viewpoint position. Therefore, when selecting an object on the image or instructing an angle, an intuitive operation in the depth direction is performed. It is difficult to specify a distance, and it is difficult to specify and specify an object superimposed in the depth direction.

  In order to improve the indeterminacy in the three-dimensional object designation, an instrument that directly designates the depth such as a three-dimensional mouse as in Patent Document 2 has been proposed. However, since a three-dimensional mouse is generally limited in the space that can be operated, it is intuitive to give instructions that include depth to general landscapes that include extremely distant objects such as mountains in the background. Is difficult.

  On the other hand, when combined with a depth instruction from a third person viewpoint different from the first person viewpoint, an instruction with less ambiguity can be given. For this reason, it is common in conventional operations on a display to divide the work screen into a plurality of parts and perform the work while alternately checking both the first person viewpoint and the third person viewpoint. However, since switching between multiple viewpoints requires switching the video while occupying the display, it is not suitable for an intuitive operation in a collaborative environment where a plurality of people see the same image.

  The disclosed stereoscopic image display system and display method include a portable device, a stereoscopic display unit that displays a stereoscopic image including a virtual object corresponding to a viewpoint position, and information processing connected to the portable device via a wireless network. Equipment. The portable device accepts an operation by the user from a touch panel included in the display unit included in the portable device, and transmits the accepted operation to the information processing device via the wireless network. The information processing device interprets the operation received from the mobile device in accordance with the mode according to the angle formed by the display surface of the display unit of the mobile device and the display surface of the stereoscopic display unit. In response, the state of the virtual object displayed on the stereoscopic display unit is updated.

  In another aspect of the stereoscopic image display system and the display method, the information processing device accepts an operation from the touch panel of the mobile device, the plane to be operated in the stereoscopic image, the display surface of the display unit of the mobile device, and the stereoscopic display As a virtual plane corresponding to the solid angle formed relative to the display surface, the image is displayed on the three-dimensional display together with the grid on the virtual plane.

  In still another aspect of the stereoscopic image display system and display method, the information processing device displays a virtual plane in which the position of the virtual object is clearly indicated on the display unit of the portable device.

  According to still another aspect of the stereoscopic image display system and display method, the information processing device includes a direction from the stereoscopic display to the portable device as a viewpoint position, and displays a stereoscopic image including a virtual object on the stereoscopic display unit.

  In still another aspect of the stereoscopic image display system and display method, when the display unit of the portable device is a stereoscopic display of a type different from the stereoscopic display unit, the information processing device is configured to display a stereoscopic image including a virtual object, The content displayed on the stereoscopic display corresponding to the viewpoint position is simulated, and the result of the simulation is displayed on the display unit of the portable device.

  According to the present invention, the user can operate the video displayed on the stereoscopic display from the portable device while maintaining the intuitive understanding of the stereoscopic.

It is a hardware block diagram of the display system of a stereo image. It is a figure which shows the structure of a portable apparatus. 2 is a diagram illustrating a configuration of an information processing device 130. FIG. It is a flowchart of the process of initialization of the composite display system of a stereo image. It is a schematic diagram which shows arrangement | positioning of the virtual object in content space. It is a flowchart of a series of work processes for performing position identification of a portable device. It is the image pattern of the program and marker sent to a portable device. It is a figure which shows the movement of the three-dimensional cursor in mode A, and designation | designated of the virtual object corresponding to it. It is the flowchart which showed a series of processes of the information update process. 6 is a schematic diagram of an operation screen displayed on a mobile device in mode A. FIG. It is a figure which shows the movement of the three-dimensional cursor in mode B, and designation | designated of the virtual object based on it. 6 is a diagram showing an outline of operation in mode C. FIG. In Example 2, it is a schematic diagram of the operation screen displayed on a portable apparatus. In Example 3, it is a schematic diagram which shows the virtual viewpoint position of the three-dimensional display simulated. FIG. 10 is a hardware configuration diagram of a stereoscopic image display system according to a fourth embodiment. FIG. 10 is a diagram illustrating a configuration of a mobile device according to a fourth embodiment. 10 is a schematic diagram of an operation screen displayed on a mobile device in mode A of Embodiment 4. FIG. FIG. 10 is a schematic diagram illustrating a relationship between a virtual object and a display surface in mode A of Example 4. FIG. 10 is a schematic diagram in which an operation target plane is arranged in a virtual space in mode C of the fourth embodiment. FIG. 10 is a top view in which an operation target plane is arranged in a virtual space in mode C of the fourth embodiment. FIG. 10 is a hardware configuration diagram of a stereoscopic image display system according to a fifth embodiment.

  Hereinafter, examples will be described with reference to the drawings.

  FIG. 1 shows a hardware configuration of a stereoscopic image (stereoscopic video) display system. As shown in FIG. 1, a marker 102 for acquiring the position coordinates of the stereoscopic display 101 from another position on a stereoscopic display 101 that presents different images in a plurality of directions (respectively directions 111, 112, and 113). Is attached. When the marker 102 is photographed from another position, the position and angle of the camera that has photographed the marker 102 are determined from the shape of the photographed marker 102 on the two-dimensional image with reference to the position of the three-dimensional display 101 by a method described later. Can be sought.

  FIG. 3 shows the configuration of the information processing device 130. As shown in FIG. 3, the information processing device 130 that drives the stereoscopic display 101 includes a memory 131 and a computing device 132, a video output unit 134 that transmits 2D video information to be displayed on the stereoscopic display 101, and a wireless network unit. Through 133, external devices are linked. The information processing device 130, which is a general general-purpose computer, performs calculation processing by the arithmetic device 132 according to a program held in the memory 131, and outputs the result as a video to the stereoscopic display 101, or communication such as TCP or HTTP Communicate to external devices via protocol. This information processing device 130 functions as a general-purpose computer, that is, a function for switching a plurality of processes and executing them in a pseudo-parallel manner, and performing calculation processing while synchronizing the processes by using an internal clock or network. Have

  Also, the wireless fixed point observation cameras 105 to 106 illustrated in FIG. 1 capture the periphery including the stereoscopic display 101 and periodically transmit the captured peripheral image of the stereoscopic display 101 to the information processing device 130. A fixed point observation camera 108 is provided on the top of the stereoscopic display 101, and the periphery of the stereoscopic display 101 is photographed with a wide viewing angle.

  There is a portable device 120 serving as an interface unit in the sense of a user interface with respect to the stereoscopic display 101. FIG. 2 shows the configuration of the mobile device 120. The portable device 120 includes a display unit 121 equipped with a touch panel that accepts user operations, a camera 122 that captures a surrounding situation, an access unit 123 to a wireless network, a memory 124 that holds and executes a program, and a calculation Driven by device 125. An external program is dynamically read from the wireless network access unit 123 into the memory 124 and driven by the arithmetic device 125, thereby processing the output to the screen 121 and the input result therefrom, and communication protocols such as TCP and HTTP. It has the function to transmit to a wireless network via. In addition, a video output unit 126 for displaying video generated dynamically by the program on the display unit, and a touch panel receiving unit 127 that receives the position of a finger, a pen tip, or the like in contact with the touch panel as coordinate information are provided. In addition, coordinate information such as finger and pen tip is classified based on time and space change until touching the touch panel, and is used in the program after semantic interpretation for operations such as dragging, tapping and zooming with two fingers. . These interpretation categories and separation methods are based on the general technology of these information terminals. The mobile device 120 has an acceleration sensor 128 for estimating the current direction of the mobile device 120 from the acceleration and the geomagnetism. This portable device 120 reads the program data through a general-purpose computer function, that is, a network, switches a plurality of processes and executes them in a pseudo-parallel manner, and uses an internal clock or network to synchronize the processes. It has a function to perform calculation processing.

  FIG. 4 is a flowchart showing a series of steps for initializing the stereoscopic image composition display system. In step 401, the positions of the three-dimensional display 101 and the fixed point observation cameras 105 to 106 are adjusted. In the present embodiment, an orthogonal coordinate system having a center of the display surface of the stereoscopic display 101 as an origin and pointing right (X) toward the display, upward (Y) toward the display, and back (Z) toward the display is used. Define. Further, the display of the mobile device 120 defines a set of unit length vectors as unit vectors 140 in the vertical direction (up), horizontal direction (right), and back direction of the screen. This vector rotates depending on the arrangement of the mobile device 120.

  The information processing device 130 receives the latest video captured by the fixed point observation cameras 105 to 106, and when the marker 102 is included in the received video of the fixed point observation cameras 105 to 107, the two-dimensional of the marker 102 in the video is displayed. The three-dimensional positional relationship between the marker 102 and the fixed point observation cameras 105 and 106 is estimated from the geometric image. From these estimated values, the positions of the fixed point observation cameras 105 to 106 with the center of the display surface of the stereoscopic display 101 as the origin are identified, and the identified position information is held in the memory 131.

  There are several existing techniques that are generally available for 3D position estimation from markers. In this embodiment, a rectangular marker and three-dimensional position information are extracted by the following procedure. First, the marker image is binarized, the black portion of the image is searched, and segmentation is performed to divide the image into closed regions. In each closed region, the end point is taken as the first vertex, and the contour of the region is created by tracing from this vertex. Find each vertex of the square marker from the outline you made. The conversion process is performed to the shape when the area having four vertices is viewed from the front, and the image in the quadrangle is simplified. When pattern matching between the image and the marker pattern is performed and the correspondence between the image and the marker pattern is equal to or greater than a threshold value, the image is assumed to be a marker. A projection matrix for converting the four vertices into corresponding positions on the image is calculated, and an estimated value for the three-dimensional position and direction of the marker from the captured camera is obtained from the parameters.

  In step 402, a thread for issuing a command for instructing the fixed-point observation cameras 105 to 106 to perform photographing at regular intervals is activated. The fixed point observation cameras 105 and 106 of the present embodiment are network cameras having a function capable of spontaneously transmitting an image every predetermined period. Even in the case of a camera operated by the information processing device 130, an equivalent operation can be performed by placing an independent process or thread in the information processing device 130.

  In step 403, the information processing device 130 activates a thread of a TCP (or HTTP) server that accepts a network connection from the portable device 120. The operation when the terminal device 120 is connected to the information processing device 130 will be described later with reference to FIG.

  FIG. 5 is a schematic diagram showing the arrangement of virtual objects in the content space for explaining processing for creating a video to be displayed on the stereoscopic display 101.

  In order to distinguish a virtual space in which the three-dimensional position information (display target) is arranged on the three-dimensional display 101 from a physical three-dimensional space expressing the positions of the three-dimensional display 101 and the portable device 120, the following contents are used. Called space. This origin coordinate 502 corresponds to the center of the display surface of the stereoscopic display 101 as described above, and the surface 501 corresponding to the display surface of the stereoscopic display 101 is the XY plane with Z = 0. FIG. 5 shows such a content space. A plurality of viewpoint positions 511 to 513 are set on the content space, and the three-dimensional coordinates of the content space are projected onto a plane 501 corresponding to the display surface of the stereoscopic display 101 to correspond to each of the set viewpoint positions 511 to 513. Construct a two-dimensional image. These two-dimensional images obtained for each viewpoint position are displayed in the directions 111 to 113 from the stereoscopic display 101.

  In addition, a marker indicating the pointer three-dimensional position 500 is arranged in the content space. The initial position of this pointer is set to the origin position 502, that is, the center of the display surface of the stereoscopic display 101.

  FIG. 6 shows a series of routines for identifying the position of the portable device 120 with respect to the coordinate system of the three-dimensional display 101. When the portable device 120 enters a certain distance from the three-dimensional display 101 and becomes communicable with the wireless network unit 133 of the information processing device 130, the portable device 120 sends a connection message to the information processing device 130 ( Step 601). The information processing device 130 transmits the marker pattern 701 for position identification shown in FIG. 7 and the program 702 executed by the portable device 120 to the portable device 120 (step 602). In step 602, the program on the information processing device 130 side that executes steps 901 to 916 (FIG. 9) for managing input information via the user's portable device 120 and the program transmitted to the portable device 120 are independent of each other. Start the thread. The operation of this thread will be described later. These processes complete the preparation.

  The following steps 610 to 633 operate while temporally coordinating with each of the portable device 120, the fixed point observation cameras 105 to 106, and the information processing device 130 with the synchronized internal timer. An independent process or thread is driven in the program of each device so that these photographing processes are synchronized. Steps 610, 620, and 630 indicate processing to wait until the time when these processes (threads) are synchronized with each device.

  In step 611, the mobile device 120 drives the program 702 and presents a marker image 701 for position identification on the display unit 121 of the mobile device 120.

  In step 612, the mobile device 120 takes a picture with the attached camera 122. In step 622, the information processing device 130 takes an image with the camera 108. In step 632, each of the fixed point observation cameras 105 to 106 performs photographing. It should be noted that the synchronization of each device is adjusted so that the photographing in steps 612, 622, and 632 is performed at approximately the same time.

  After this photographing process, in step 613, the program 702 of the mobile device 120 reads angle information from the acceleration sensor 128 of the mobile device 120 at this time. In step 614, the mobile device 120 transmits the image captured in step 612 and the angle information acquired in step 613 to the information processing device 130.

  In step 633, images taken by the fixed point measurement cameras 105 to 106 are transmitted to the information processing device 130.

  Corresponding to Steps 614 and 633, in Step 623, the information processing device 130 receives the images of the fixed point observation cameras 105 to 106 that photograph the marker 701 presented by the mobile device 120. In addition, the angle information of the mobile device 120 acquired in step 613 and the image by the camera 122 of the mobile device 120 acquired in step 612 are acquired.

  In step 624, the information processing device 130 performs a pattern search process to determine whether or not the marker pattern 701 for position identification is captured in the images captured by the fixed point observation cameras 105 to 106. When the marker 701 is shown, the parameters of the geometric projection matrix are calculated, and the relative position between each of the fixed point observation cameras 105 to 106 and the portable device 120 is estimated. This marker pattern search process is as described above.

  In step 625, pattern search processing is performed to determine whether or not the marker pattern 102 for position identification is included in the image captured by the camera 122 of the portable device 120. When the marker is reflected, the parameters of the geometric projection matrix are calculated, and the relative position between the portable device 120 and the three-dimensional display 101 is estimated. The search process for the marker pattern 102 is as described above.

  In step 627, the relative position obtained in step 624, the angle information from the portable device 120, and the relative position obtained in step 625 are combined to remove outliers that are significantly different from other numerical values. Then, averaging is performed to estimate the final relative position.

  In step 628, a face image pattern is searched from the image of the camera 108 taken in step 622. In order to search for the face image, a program routine for performing the following series of processes is called and executed. A region having a hue close to the skin color is extracted from the face image. For a general face image, a pattern (Haar pattern) holding the black part and white part of the face is called from the memory, and pattern matching with this Haar pattern is performed on the vicinity of the hue area close to the skin color. A region where the result is similar to the threshold value or more is recognized as a face image. From the relationship between the direction in which the camera 108 is installed and the image area, the solid angle where the recognized face image exists is calculated and stored. When a plurality of face images are taken, the face image closest to the position of the information processing device 130 obtained from the images from the fixed point observation cameras 105 to 106 and the direction from the stereoscopic display 101 is displayed on the mobile device 120. It is determined that the face is the user's face, and the position information is used.

  Information update processing 629 is performed using the angle information and position information of the portable device 120. The information update process 629 is a process of rewriting a reference variable for mode selection of an information update thread from the portable device 120, which is executed in steps 901 to 916 of FIG. When these series of processes are completed, the shooting threads of the mobile device 120, the fixed point observation camera, and the information processing device 130 wait until the next shooting synchronization time comes.

  FIG. 9 is a flowchart showing a series of processes of the information update process (step 629). This flowchart includes steps 902 to 905 performed on the information processing apparatus 130 side and steps 911 to 916 performed on the mobile device 120 side. These steps are continuously operated as independent processes or threads, but the mode is switched according to the angle information of the mobile device 120 updated in step 629, and in steps 903 and 905, the mode is changed. Do different processing.

  First, the entire flow part that does not depend on the mode will be described. In step 901, the mode is selected according to the angle information of the mobile device 120 updated in step 629. When the direction of the surface of the stereoscopic display 101 and the surface of the display 121 of the portable device 120 is within the threshold θ (that is, when the display surface of the stereoscopic display 101 and the display surface of the display 121 of the portable device 120 are in the same direction) Mode A. When the display surface of the stereoscopic display 101 and the display surface of the display 121 of the portable device 120 are within the threshold θ from the vertical (that is, when the display surface of the stereoscopic display 101 is vertical, the surface of the display 121 of the portable device 120) Mode B). If the angle is other than that, mode C is set. As described above, the mode is defined according to the relative solid angle between the display surface of the display 121 of the portable device 120 and the display surface of the three-dimensional display 101.

  Step 902 is a procedure for creating an image to be sent to the mobile device 120 according to the mode. The image to be sent to the portable device 120 differs depending on the mode, and drives an image creation processing routine corresponding to each mode A to C. The execution contents of the processing routine in each mode will be described later. In step 903, the image created in the processing routine of step 902 and the type of mode are sent to the portable device 120.

  Steps 911 to 916 are processes performed on the mobile device 12 side. In step 911, the image and mode type information sent in step 903 is received. In step 912, the received image is presented. In step 913, it is checked whether or not the mode type has changed, and if the mode type has changed, read information relating to the touch panel is reset in step 914. In step 915, the access state to the touch panel is checked, and the state of dragging, tapping, or zooming (or no operation has been performed) is acquired. In step 916, the information on the touch state is transmitted to the information processing device 130. Send.

  The information processing device 130 receives the touch state information acquired in step 916 in step 904, and updates the video displayed on the stereoscopic display 101 based on the information in step 905. More specifically, the update processing of the video information in step 905 is to interpret the touch state (touch panel operation) according to the mode and update the video information (virtual object state) by the interpreted operation. This image (video information) also differs depending on the mode, and is driven by a processing routine corresponding to each mode A to C. The contents of each processing routine will be described later. When the series of operations is completed, the process returns to step 901 again to transmit the screen and acquire information on the touch panel of the mobile device 120.

  Steps 902 and 905 are routines that change the interpretation of input information from the portable device 120 depending on the mode type. There are three types of modes A to C as described above. Mode A is a mode in which the stereoscopic display 101 and the display 121 of the mobile device 120 are oriented in the same direction, and thus an object is selected in accordance with the position where an image rendered in a certain angle direction is projected on the display surface. It is. Further, mode B is a mode in which the surface of the display 121 is facing upward, so that an object is selected using a position on the horizontal plane (XZ plane) in the content space. Mode C uses a solid angle formed by the display surface of the three-dimensional display 101 and the display surface 121 of the mobile device 120, and presents a surface parallel to the surface of the display 121 of the mobile device 120 in the content space. In this mode, movement, rotation, and zoom processing are performed. That is, the target plane for operations such as movement, rotation, and zoom processing differs depending on the mode.

  When switching between these modes, a single 3D point on the content space is shared by each mode, and the coordinates around this point are redefined.

  An outline of the contents of operations performed in steps 902 and 905 in the case of mode A will be described. When the position of the mobile display 121 is placed in parallel with the stereoscopic display 101, a selection process in mode A is performed. In this mode A, the display surface 121 of the mobile device 120 and the display surface of the stereoscopic display 101 are made to correspond one-to-one, and an object reflected on the display surface of the mobile device 120 or the stereoscopic display 101 is designated. FIG. 8 is a diagram showing the movement of the three-dimensional cursor and the designation of the corresponding virtual object in mode A. FIG. 10 is a schematic diagram of an operation screen 1002 displayed on the mobile device 120 corresponding to the display display 1001 of the stereoscopic display 101 in mode A.

  At the time of the operation in step 915, the image presented in the direction 511 closest to the relative position on the physical space of the display 121 of the mobile device 120 with respect to the stereoscopic display 101 obtained in step 627 is the mobile device 120. Presented on the display 121. When selecting the position 540 of the touch panel, the user operates with the intention of selecting the position 550 associated with the three-dimensional display 101.

  At this time, in order to obtain the position including the depth of the cursor 500 from the projected image 550 of the cursor presented on the stereoscopic display 101, the viewpoint position 511 of the stereoscopic display 101 near the viewpoint position 803, which is the position of the mobile device 120. The position of collision with the virtual object is calculated by following the line-of-sight direction (Ray) passing through the cursor 550 on the display surface. The cursor position 500 thus obtained is shown in FIG. Even when the projected image 550 of the cursor on the three-dimensional display 101 is continuously moved as viewed from the viewpoint position 511, the projected image 550 of the cursor 500 is not necessarily displayed in the images viewed from other positions such as 512 and 513. It does not necessarily move continuously on the plane 101.

  In step 902A (step 902 in mode A, hereinafter the same applies), the information processing device 130 displays an image (viewpoint in the situation of FIG. 8) that is presented on the stereoscopic display 101 in the direction of the mobile device 120. A reduced copy of the image rendered at 511 is sent to the portable device 120. In step 912, since the portable device 120 presents the sent image on the display 121 that it holds, in step 915, the user interprets that the user is performing a touch operation on the screen surface.

  In step 905A, under the condition of mode A, the user's operation on the touch panel is reflected in the content of the stereoscopic video. If drag processing has been performed, the cursor 500 is displayed at the position of the three-dimensional display 101 corresponding to the final position of the drag operation. At this time, the color of the colliding object is changed by following the line-of-sight direction (Ray) from the cursor, and the cursor 500 is presented at the depth position. At the same time, the display mode such as the color of the object 560 as the selection candidate is displayed. Change and present. When the tap process is performed, the selected object 560 is selected from the current cursor position, and the coordinates of the three-dimensional cursor 500 are set to the center position of the object 560. If there is an already selected object and a drag operation has been performed, the selected object is moved on the XY plane by an amount corresponding to the movement distance of the drag operation.

  Next, an outline of the operation contents performed in Step 902 and Step 905 in the case of Mode B will be described. When the orientation of the surface of the display 121 of the mobile device 120 is placed horizontally on the ground (perpendicular to the direction of gravity), the horizontal position in the content space inside the display surface 121 of the mobile device 120 and the stereoscopic display 101 The object is specified by associating (X-Z plane) one-to-one.

  FIG. 11 is a diagram showing the movement of the three-dimensional cursor in mode B and the designation of the virtual object based on the movement. In step 902B, in order to display the image 1102 on the portable device 120, the information processing device 130 creates an image 1102 when the position information on the XZ plane of the content space is drawn 1101 from above. This image is an image obtained by rotating around the Y axis of the portable device 120 around the current position of the three-dimensional cursor 500. In addition, a grid 1101 is drawn on the virtual space in accordance with the rotation angle around the Y axis of the display 121 of the mobile device 120. Since the grid drawn on the XZ plane of the content space and the image on the XY plane presented on the display 121 of the portable device 120 are in a one-to-one relationship with each other, in step 912, the user performs the X- This is interpreted as a touch operation on the Z plane.

  In Step 905B, under the condition of Mode B, the user's operation on the touch panel is reflected in the stereoscopic video content. When a zoom operation is performed, the image size and the grid size on the XZ plane of the content space are changed according to the magnification indicated by the zoom operation, and the distance that the three-dimensional cursor can be moved by the same drag is set. Change. When the drag operation is performed, the information processing device 130 moves the three-dimensional cursor 500 by the movement distance on the XZ plane corresponding to the movement distance on the display 121 of the portable device 120. When there is a virtual object (object) at the position of the XZ plane corresponding to the three-dimensional cursor, the selection candidate is explicitly presented by changing the display mode such as the color of the object. When tap processing is performed in a situation where a selection candidate object is selected, it is interpreted that the selection candidate object is designated as the selection object, and the selected object 560 is selected from the current cursor position. Then, the coordinates of the three-dimensional cursor 500 are set at the center position of the object 560. When there is an already selected object and a drag operation has been performed, the selected object is moved on the XZ plane by an amount corresponding to the movement distance of the drag operation.

  Next, the outline of the operation performed in Step 902 and Step 905 in the case of Mode C will be described with reference to FIG. When the orientation of the surface of the display 121 of the mobile device 120 is neither mode A nor mode B, the angle of the surface of the display 121 of the mobile device 120 with respect to the coordinates of the 3D display 101 (the display surface of the 3D display 101 and The virtual plane 1200 is presented on the display 121 of the mobile device 120 on the content space in accordance with the relative angle of the display surface of the display 121 of the mobile device 120, and an interaction operation corresponding to the virtual plane 1200 is performed on the mobile device 120. From the touch panel.

  In step 902C, the information processing device 130 passes through the coordinate point of the three-dimensional cursor 500 in the content space from the information on the angle and position of the mobile device 120, and is parallel to the display 121 surface of the mobile device 120 in the physical space. A plane is defined, and the plane of the content space is rendered to be presented to the mobile device 120 as a projection plane. In addition, a grid 1201 is drawn on a plane corresponding to the plane 1200 in the content space of the stereoscopic video.

  In step 912, the user interprets that the user is performing a touch operation on the plane 1200 in the designated content space.

  In Step 905C, under the condition of mode C, the user's operation on the touch panel is reflected on the content of the stereoscopic video. When the zoom operation is performed in the vertical direction, the size of the image presented on the display 121 of the portable device 120 and the size of the grid 1201 on the plane of the content space are changed according to the magnification indicated by the zoom operation. The distance that the three-dimensional cursor 500 can be moved by the same drag is changed. When the drag operation is performed, the three-dimensional cursor 500 is moved on the plane 1200 by the movement distance corresponding to the movement distance indicated by the drag operation. When a virtual object (object) is present at a position corresponding to the three-dimensional cursor 500, selection candidates are explicitly presented by changing the display mode such as the color of the object. When tap processing is performed in a situation where selection candidates are selected, an object that is a selection candidate is selected. If there is an already selected object and a drag operation has been performed, the selected object is moved on the plane 1200 by an amount corresponding to the movement distance of the drag operation.

  By these series of processing routines, the cursor 500 in the three-dimensional display, the object selection, and the movement operation are performed according to the mode.

  Through this series of processing, in this embodiment, the display that can be carried by the user is dynamically combined with the stereoscopic display 101 and the stereoscopic printed material, and the viewpoint position that matches the user's intention is selected. Depth expression control and three-dimensional position designation operation can be realized efficiently.

  Especially when adjusting and processing a stereoscopic video that cannot be expressed by a projection matrix at one viewpoint position, such as an autostereoscopic display, switching between viewpoint position and angle selection modes becomes complicated. It is difficult and inefficient to work while confirming the impression of all compositions of up to several tens of viewpoints using a normal monitor. An intuitive operation on the result can be performed by directly using the three-dimensional display during the adjustment.

  In addition, in the autostereoscopic display, a plurality of videos are switched and displayed for each viewpoint position to be viewed, but may be viewed in a mixed form of any of the plurality of images. For this reason, if you try to draw an object that is far away from the surface of the display, you will see a double image or the image will be blurred, which will be uncomfortable for the user, so use these autostereoscopic displays. In general, the possible depth distance is limited to the vicinity of the surface of the display surface. In addition, the tendency for viewing images to be difficult to view under the condition that the display surface is separated from the display surface is not limited to the autostereoscopic method. When separated from the original image, the separation from the original image increases, and the effects of motion sickness and fatigue tend to increase. When such fatigue and 3D sickness peculiar to a 3D display were born, it was difficult to work for a long time, but by using the method of this example, necessary information was fixedly displayed on the display coordinates on hand. Therefore, reduction of such a three-dimensional sickness effect can be expected.

  In addition, when the display surface (two-dimensional plane) of an autostereoscopic display is selected as a touch panel, as in a normal video display with a single viewpoint, if the viewpoint position is different, the display position will be The position in the associated three-dimensional space can change. In a normal monitor, assuming a single viewpoint, a single line is shown by indicating the direction connecting the viewpoint position and a point on the plane, but in a stereoscopic display in which multiple viewpoint directions are presented simultaneously, When the interaction with the display surface is performed, indefiniteness is generated for the specified object unless it is clearly specified which viewpoint is used to specify the object. In this embodiment, such indefiniteness is generated. An object can be instructed intuitively without holding it.

  In the present embodiment, a method of creating an image viewed from a certain direction and sending the image to the mobile device 120 will be described.

  Each hardware of the present embodiment is the same as that of the first embodiment.

  In the present embodiment, as shown in FIG. 13, in the creation of an image 1300 corresponding to step 902C of the first embodiment, a scroll bar 1301 for controlling the shooting distance (this is the operation screen of the display 121 of the portable device 120) An input value is set as D), a scroll bar 1302 for controlling the shooting angle of view (this input value is set as A), and a button 1303 for acquiring a final shot image.

  In the touch panel operation in step 915, when the scroll bar 1301 is operated and the value D changes, a point that becomes a virtual viewpoint position is created in the projection image creation on the virtual plane 1200 (FIG. 12) performed in step 905C. An image is created by being arranged at a distance D from the virtual plane 1200.

  In addition, when the scroll bar 1302 is operated and the value A changes in the touch panel operation in step 915, the virtual angle of view is changed to the angle A in the projection image creation on the virtual plane 1200 performed in step 905C. To create a video.

  In addition, when the button 1303 is pressed in the touch panel operation 915, a high-resolution image of the content space projected on the virtual plane 1200 at the distance D angle of view A is rendered, and the created image is attached to the mail. Then, it is transmitted to the e-mail address associated with the mobile device 120 in advance.

  In the present embodiment, a method of sending a simulation result of video output (display contents corresponding to each viewpoint position) seen from each viewpoint position when using a stereoscopic display with parameters different from the stereoscopic display 101 used to the portable device 120 Indicates. Such a simulation can be used, for example, to determine the degree to which the video appears to be degraded when video from a plurality of different viewpoints appears to overlap (referred to as crosstalk).

  This will be described with reference to FIG. The hardware is different from that of the first embodiment and the stereoscopic display 1400, and the stereoscopic display 1400 has five virtual viewpoints 1401 to 1405.

  In this embodiment, in the creation of an image corresponding to step 902C of the first embodiment, five types of projection images centered on the respective viewpoint positions 1401 to 1405 are projected on the virtual plane 1200 (FIG. 12) to obtain a high resolution. Images are created, their superimposed images are created, and sent to a mail address associated with the portable device 120 by the method of the second embodiment. The user can estimate how much stereoscopic image crosstalk is generated by the object 560 at the position 500 by looking at the sent image.

  FIG. 15 shows a hardware configuration necessary for realizing the present embodiment. Except for the portable device 1500, the second embodiment is the same as the first embodiment. The portable device 1500 has a display that can present autostereoscopic images. A configuration of the portable device 1500 is shown in FIG. Except that the display 1501 is a stereoscopic display such as a lenticular lens or a parallax barrier, it is the same as the portable device 120 in the first embodiment.

  Most of the operation steps of the present embodiment are the same as those of the first embodiment. However, in the display on the display 1501 of the mobile device 1500 of the marker 701 photographed from each of the fixed point observation cameras 105 to 106, drawing so as to be observed on the display plane, and to the mobile device 1500 in each mode In the display of the three-dimensional space, a three-dimensional image with unevenness information for reflecting the configuration of the three-dimensional space is created and transmitted.

  FIG. 17 is an image corresponding to mode A (FIG. 10) in the first embodiment. FIG. 18 is a schematic diagram showing the relationship between the virtual object in the image of FIG. 17 and the surface of the display 1501. The viewpoint position 1801 is the viewpoint position of the user's face obtained in step 628. Here, rendering processing of each object such as the three-dimensional marker 1740 and the three-dimensional virtual object 1760 is performed as follows. Of the videos that can be presented on the stereoscopic display 1501, a video that is presented in one direction close to the center is used as a representative viewpoint. This representative viewpoint video is created by a projection matrix that projects toward the plane of the virtual space corresponding to the surface 1740 of the stereoscopic display 1501 of the portable device 1500 with the viewpoint position 1801 as the center. For other videos, the relative viewpoint position of each viewpoint of the stereoscopic display is calculated with this representative viewpoint as a reference.

  FIG. 19 is a schematic diagram in which an operation target plane corresponding to mode C (FIG. 12) in the first embodiment is arranged in a virtual space. Through the same steps as in the first embodiment, a plane 1900 corresponding to the plane 1200 (FIG. 12) in the first embodiment is presented on the stereoscopic display 101. An image 1902 is drawn on the surface of the stereoscopic display 1501 of the portable device 1500. In this example, a three-dimensional cursor image 1941 and a virtual object image 1961 are displayed on the stereoscopic display 101, and a three-dimensional cursor image 1940 and a virtual object image 1960 are displayed on the stereoscopic display 1501 of the portable terminal. Each is drawn. FIG. 20 is a top view showing the relationship of the depth position at the time of drawing. The three-dimensional image 1960 and the three-dimensional cursor image 1940 drawn on the stereoscopic display 1501 are arranged at equal positions relative to the virtual plane 1900, the three-dimensional image 1961, and the three-dimensional cursor image 1941. In order to realize such expression, a virtual viewpoint position 2001 is arranged at a position relatively equivalent to the viewpoint position 1801 viewed from the image 1940 of the three-dimensional cursor. That is, the stereoscopic angle from the center of the stereoscopic display 1501 of the portable device 1500 to the viewpoint position 1801 is equal to the stereoscopic angle from the center of the virtual terminal to the virtual viewpoint position 2001, and from the viewpoint position 1801 to the display 1501 surface of the portable device 1500. The ratio of the distance and the distance from the viewpoint position 2001 to the virtual plane 1900 is such that the ratio of the display size of the portable device 1500 and the size of the virtual plane (more precisely, the square 1901 of the grid drawn thereon) is the same position. It is an important viewpoint position. At this time, among the images that can be presented on the stereoscopic display 1501, an image presented in one direction close to the center is used as the representative viewpoint, and the creation of the representative viewpoint image is equivalent to the virtual plane 1900 with the viewpoint position 2001 as the center. This is done by a projection matrix that projects toward the plane of the virtual space. For other videos, the relative viewpoint position of each viewpoint of the stereoscopic display is calculated with this representative viewpoint as a reference.

  By performing such drawing using the portable device 1500 provided with the stereoscopic display 1501, in the case of mode A, a stereoscopic image is presented based on the physical space where the stereoscopic display 1501 is placed. In the case of mode C, a stereoscopic image corresponding to the angle of the portable device 1500 is presented based on the virtual point 1941 where the cursor 1940 is placed. In mode A, since the display surface can be freely arranged near the virtual object, details can be visually recognized and the effect of video sickness can be reduced. In mode C, a stable mayor can be achieved by fixing the position of the object by fixing the three-dimensional cursor, while the angle can be changed intuitively.

  In this embodiment, a method of using a three-dimensional printed material without using a three-dimensional display as the display 101 will be described.

  A hardware configuration example in this case is shown in FIG. . Reference numeral 2101 denotes an instrument to which a three-dimensional printed material is attached, and projects different images on 2111 to 2113, respectively. A series of operations performed by the fixed-point observation cameras 2105 to 2106 in order to acquire the mobile terminal 2120 and its solid angle conform to the description in the previous embodiment. An image is created by designating the position of the virtual plane with the portable device 2120 for this video, but the drawing process of the image on the stereoscopic display is not performed in step 903, and only the image drawing and the video transmission to the portable device 2120 are performed. .

  Thereby, with respect to the three-dimensional printed material, an image from a direction not currently displayed can be acquired and obtained as an image.

  According to the described embodiment, the user can operate the image displayed on the stereoscopic display from the portable device while maintaining an intuitive understanding of the stereoscopic. In addition, it is possible to realize an interface for designating a three-dimensional relationship by an intuitive operation as well as an effect of reducing motion sickness and eye strain caused by viewing fine objects and characters on a three-dimensional display for a long time.

  101: 3D display, 102: Marker associated with the 3D display, 108: Camera, 111-112: Direction in which the 3D display can show an image independently, 120: Mobile device, 121: Display with touch panel of mobile device, 122: Camera of portable device 120, 123: Wireless network access unit, 130: Information processing device, 131: Memory, 132: Computing device, 133: Wireless network unit.

Claims (10)

A stereoscopic device that displays a stereoscopic image including a virtual object corresponding to a viewpoint position, and a portable device including a display unit having a touch panel that receives an operation by a user and a wireless network access unit that transmits the received operation via a wireless network A display unit is connected to the portable device via the wireless network, and corresponds to a mode corresponding to a solid angle formed between the display surface of the display unit of the portable device and the display surface of the stereoscopic display unit. An information processing device that interprets the operation received from the portable device and updates the state of the virtual object displayed on the three-dimensional display unit according to the interpreted result. Image display system. 2. The stereoscopic image display system according to claim 1, wherein the information processing device accepts the operation from the touch panel of the portable device, and displays a plane to be operated in the stereoscopic image of the display unit of the portable device. A stereoscopic image display system, wherein a virtual plane corresponding to a solid angle formed between a display surface and a display surface of the stereoscopic display is displayed on the stereoscopic display together with a grid on the virtual plane. 3. The stereoscopic image display system according to claim 2, wherein the information processing device displays the virtual plane in which the position of the virtual object is clearly displayed on the display unit of the portable device. system. The stereoscopic image display system according to claim 1, wherein the information processing device includes a direction from the stereoscopic display to the portable device as the viewpoint position, and displays the stereoscopic image including the virtual object on the stereoscopic display unit. A stereoscopic image display system characterized by: The stereoscopic image display system according to claim 1, wherein the information processing device includes the virtual object when the display unit of the portable device is a different type of stereoscopic display than the stereoscopic display unit. A stereoscopic image display system characterized by simulating the content of an image displayed on the stereoscopic display corresponding to the viewpoint position and displaying the result of the simulation on the display unit of the portable device. Display of a stereoscopic image by a system including a portable device, a stereoscopic display unit that displays a stereoscopic image including a virtual object corresponding to the viewpoint position, and an information processing device connected to the portable device via the wireless network A method,
The mobile device receives an operation by a user from a touch panel included in a display unit included in the mobile device, transmits the received operation to the information processing device via a wireless network,
The information processing device performs the operation received from the portable device in accordance with a mode corresponding to a solid angle formed between the display surface of the display unit of the portable device and the display surface of the stereoscopic display unit. A stereoscopic image display method comprising: interpreting and updating a state of the virtual object displayed on the stereoscopic display unit according to the interpreted result. The stereoscopic image display method according to claim 6, wherein the information processing device accepts the operation from the touch panel of the portable device, and displays an operation target plane in the stereoscopic image of the display unit of the portable device. A method for displaying a stereoscopic image, comprising: displaying on a stereoscopic display together with a grid on the virtual plane as a virtual plane corresponding to a solid angle formed between a display surface and a display surface of the stereoscopic display. The stereoscopic image display method according to claim 7, wherein the information processing device displays the virtual plane in which the position of the virtual object is clearly displayed on the display unit of the portable device. Method. The stereoscopic image display method according to claim 6, wherein the information processing device includes a direction from the stereoscopic display to the portable device as the viewpoint position, and displays the stereoscopic image including the virtual object on the stereoscopic display unit. A method for displaying a stereoscopic image, characterized by: The stereoscopic image display method according to claim 6, wherein when the display unit of the portable device is a stereoscopic display of a type different from the stereoscopic display unit, the information processing device includes the virtual object. A stereoscopic image display method, comprising: simulating the content of an image displayed on the stereoscopic display corresponding to the viewpoint position; and displaying the result of the simulation on the display unit of the portable device.

Images