JP2014116843A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

An arbitrary image is designated from among three-dimensional images displayed on a multi-viewpoint three-dimensional display according to a user's viewpoint.
A multi-view three-dimensional image information acquisition unit 601 acquires three-dimensional image information from a storage device or a network control unit. The 3D image generation unit 602 generates a 3D image based on the 3D image information acquired by the 3D image information acquisition unit 601. The three-dimensional image designating unit 603 includes a plurality of three-dimensional image information included in the three-dimensional image information input to the three-dimensional display based on a user instruction using a touch panel, a keyboard, a mouse, and the like and an imaging device. Thus, a part of the image information of the three-dimensional image information is designated. By this designation, the viewpoint desired by the user can be designated accurately, and an image at a predetermined position can be designated accurately.
[Selection] Figure 6

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program that display a multi-view three-dimensional image and allow a user to designate a partial image of the three-dimensional image viewed from an arbitrary viewpoint.

  Conventionally, it is known that a touch panel is arranged on a display screen of a two-dimensional display that displays a two-dimensional image, and a user directly specifies a part of the two-dimensional image displayed on the display screen with a finger or the like. . In the touch panel of this two-dimensional display, the display image and the touched display position correspond one-to-one, so that the image desired by the user can be designated accurately.

  On the other hand, Patent Document 1 discloses a technique in which a three-dimensional image of an operation component such as a button or a switch is displayed on a display screen of a three-dimensional display, and a user touches the three-dimensional image using a touch panel. . Furthermore, it is disclosed that the display state of the three-dimensional image is changed three-dimensionally. In this patent document 1, since the three-dimensional image obtained by imaging from one viewpoint is displayed on the display screen, the display image and the touched display position can be associated with each other on a one-to-one basis.

Japanese Patent Laid-Open No. 10-105735

  However, when a multi-viewpoint three-dimensional image is displayed on the display screen, the display image differs depending on the user's viewpoint even at the same display position. Not supported. For this reason, simply detecting the display position touched by the user cannot identify from which viewpoint a part of the display image is touched, and cannot accurately specify the image desired by the user. There's a problem.

  Here, the “multi-viewpoint three-dimensional image” is a three-dimensional image of a plurality of viewpoints obtained by imaging the same subject from a plurality of viewpoints. The “viewpoint” indicates the position of both eyes of the user as viewed from the position of the three-dimensional display, and an image for the user's right eye and an image for the left eye are delivered to one viewpoint.

  In order to solve such a problem, an image processing apparatus according to the present invention includes an image display unit that inputs multi-viewpoint 3D image information and displays a 3D image, and a 3D image displayed by the image display unit. And image designating means for designating a part of the image.

  According to the present invention, it is possible to accurately specify an image at a predetermined position displayed on a desired three-dimensional display according to the viewpoint of the user.

1 is a schematic diagram illustrating an example of an image processing apparatus according to an embodiment of the present invention. It is a figure explaining the display which concerns on one Example of this invention in detail. It is a figure which shows the relationship between the multi viewpoint based on one Example of this invention, and a captured image. It is a figure which shows the appearance of the multi-viewpoint three-dimensional image by the difference in viewpoint which concerns on one Example of this invention. It is a figure which shows the relative positional relationship of the user and display which concern on one Example of this invention, and the image which a user sees with a display. It is a figure which shows an example of the functional block of the image process part which concerns on one Example of this invention. It is a figure which shows the flowchart of the image designation | designated process of the multiview 3D image which concerns on one Example of this invention. It is a figure which shows an example of the procedure which selects the desired viewpoint from the multidimensional 3D image which concerns on one Example of this invention. It is a figure which shows the flowchart of the image designation | designated process of the multiview three-dimensional image which concerns on the other Example of this invention. It is a figure showing the principle of viewpoint recognition which concerns on the other Example of this invention. It is a flowchart which shows an example of the viewpoint recognition process which concerns on the other Example of this invention. It is a figure showing the display principle of the three-dimensional display of a general parallax barrier system. It is a figure which shows an example of the display of the image by the parallax barrier system which concerns on the other Example of this invention.

  Prior to the description of the embodiments, terms used in this specification will be briefly described.

  The “two-dimensional image” indicates a two-dimensional image itself that is displayed on a display device and represents, for example, a person or a landscape.

  The “three-dimensional image” indicates a three-dimensional image itself displayed on a three-dimensional display, for example, representing a person or a landscape. That is, the three-dimensional image is composed of a plurality of two-dimensional images having parallax.

  “2D image information” or “3D image information” indicates a signal source for displaying the 2D image or 3D image.

  Next, the three-dimensional display will be briefly described. The three-dimensional display device simultaneously displays a plurality of two-dimensional image information indicating a plurality of two-dimensional images on one screen of the display device as three-dimensional image information. Specifically, the two-dimensional image displayed on the three-dimensional display is based on at least two or more pieces of two-dimensional image information viewed from each viewpoint of the left eye and the right eye. Therefore, the three-dimensional image displayed three-dimensionally in the vicinity of the same position on the display screen of the three-dimensional display greatly varies depending on the position of the user's viewpoint.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. In the present embodiment, an image processing apparatus will be described as an example, but the present invention is not limited to this.

<Example 1>
FIG. 1 is a schematic diagram illustrating an example of an image processing apparatus 1 according to the first embodiment. In FIG. 1, an image processing apparatus 1 according to this embodiment includes a general personal computer (hereinafter referred to as a PC) 2 and a display 3 for displaying a three-dimensional image based on three-dimensional image information. The PC 2 acquires 3D image information and the like from the network line 4 and the like via the network control unit 5.

  The PC 2 is controlled by a CPU (Central Processing Unit) 10. The PC 2 includes a storage device 20 that holds 3D image information, a program used in the present embodiment, and the like. The PC 2 also includes a ROM (Read Only Memory) 30 that stores the various settings and programs, and a RAM (Random Access Memory) 40 that functions as a temporary storage device. The display 3 has a three-dimensional display 50 that displays a three-dimensional image. The display device 3 has a touch panel 60 on the display surface of the three-dimensional display device 50, specifies the displayed three-dimensional image by directly touching it, and performs various input operations to the user's PC as with the keyboard 70. Enable. The keyboard 70 is connected to a mouse touch pad 80 provided on the PC 2. The imaging device 90 is a thing equipped with a general display, and is provided in the vicinity of the three-dimensional display 50 in order to recognize a user's position. A printer 100 that outputs printed matter is connected to the PC 2 via the network control unit 5.

  FIG. 2 is a diagram for explaining the display 3 shown in FIG. 1 in more detail. 2A, a lenticular multi-viewpoint display 200 used as the three-dimensional display 50 includes a liquid crystal display unit 201 and a lenticular lens 202 on the upper surface thereof. Furthermore, the touch panel 60 is provided on the upper surface of the lenticular lens 202, and the imaging device 90 is provided at the center of the screen.

  FIG. 2B shows each lenticular lens 202 and each display pixel with a guaranteed relative position for displaying a multi-viewpoint three-dimensional image. The lenticular multi-view display 200 in FIG. 2B enables display of a multi-view three-dimensional image when the user views the lenticular multi-view display 200 from six directions. Therefore, each lenticular lens 202 has a series of seven display pixels from pixel 203 to pixel 204. Note that the lenticular multi-viewpoint display device 200 is generally realized by a known technique and can perform color display. However, in order to simplify the description, it is assumed that monochrome display is possible in this embodiment. In the subsequent drawings, the arrangement of display pixels is expressed one-dimensionally. In the following description, the term “viewpoint” simply indicates the position of both eyes of the user as viewed from the position of the lenticular multi-viewpoint display 200. Then, the user's right eye image and left eye image are delivered to one “viewpoint”.

  FIG. 2C and FIG. 2D show pixels that can be seen when the user views from viewpoints 1 and 4 at different positions. That is, due to the optical characteristics of the lenticular lens 202, only the pixel 2 is visible to the right eye of the user at the viewpoint 1 and only the pixel 1 is visible to the left eye of the user at the viewpoint 1. Further, only the pixel 5 is visible to the right eye of the user at the viewpoint 4, and only the pixel 4 is visible to the left eye of the user at the viewpoint 4. Such display of a three-dimensional image by the lenticular lens 202 is a known principle, and a description thereof will be omitted in this specification.

  FIG. 3 is a diagram illustrating a relationship between six viewpoints from six directions and images captured to display images corresponding to the viewpoints. At each viewpoint, two two-dimensional images corresponding to images viewed with the left and right eyes of the user are captured. In order to display the same subject three-dimensionally continuously from viewpoint 1 to viewpoint 6, image N delivered to the user's left eye at viewpoint N (N is an integer from 1 to 6) is the user at viewpoint N-1. It is the same as the image delivered to the right eye. If the two-dimensional image information indicating these images is divided and displayed with liquid crystal display pixels that maintain a relative position with the lenticular lens 202, the same three-dimensional image as when the subject is viewed from each viewpoint shown in FIG. 3 is obtained. Can see.

  For example, the pixel 1 shown in FIGS. 2C and 2D is a part of the two-dimensional image information indicating the two-dimensional image (image 1) captured on the left side of the viewpoint 1 in FIG. Is a part of the two-dimensional image information indicating the two-dimensional image (image 2) captured on the right side of the viewpoint 1. The pixel 3 is a part of two-dimensional image information indicating a two-dimensional image (image 3) captured on the left side of the viewpoint 3, and the pixel 4 is a two-dimensional image (image 4) captured on the right side of the viewpoint 3. Is a part of the two-dimensional image information. The pixel 4 is a part of two-dimensional image information indicating a two-dimensional image (image 4) captured on the left side of the viewpoint 4, and the pixel 5 is a two-dimensional image (image 5) captured on the right side of the viewpoint 4. Is a part of the two-dimensional image information. Further, the pixel 6 is a part of two-dimensional image information indicating a two-dimensional image (image 6) captured on the left side of the viewpoint 6, and the pixel 7 is a two-dimensional image (image 7) captured on the right side of the viewpoint 6. Is a part of the two-dimensional image information.

  A multi-view three-dimensional image and how a part of the three-dimensional image is seen will be described in detail with reference to FIG. If the subject in FIG. 3 is the subject 400 in FIG. 4A, the two-dimensional image viewed from the left and right eyes at each viewpoint in FIG. 3, that is, the two-dimensional image in FIG. 407 seven images are obtained.

  That is, a part 408 of the subject 400 appears at the right end of the screen in the image 1 (image 401) that is the captured image for the left eye of the viewpoint 1, and conversely, the image 7 (image 407) that is the captured image for the right eye of the viewpoint 6. Then it looks at the left edge of the screen.

  Next, when the two-dimensional image indicated by the image 407 is divided from the image 401 captured in FIG. 4B and displayed on the liquid crystal display pixels that maintain the relative position with the lenticular lens 202, the image shown in FIG. The same three-dimensional image can be seen as when the subject is viewed from the viewpoint shown. FIG. 5 shows a case where the subject 400 is displayed on the display 3 as a multi-viewpoint three-dimensional image.

  FIG. 5 shows, as an example, the relative positional relationship between the user and the display 3 when the user views the display 3 from the viewpoint 2 and the viewpoint 5, and an image that can be seen by the user on the display 3.

  In FIG. 5, the user 500 is viewing the display device 3 obliquely with an angle substantially at the viewpoint 2. Therefore, the image 402 shown in FIG. Similarly, an image 403 shown in FIG. 4B appears on the right eye of the user 500. As a result, the subject 400 in FIG. 4A is visually recognized as a screen 501 as a three-dimensional image viewed from the viewpoint 2. At this time, if the user 500 touches a position 502 on the screen with the touch panel 60 arranged on the front surface of the screen with a finger or the like, the part 408 of the subject 400 is touched and designated.

  Similarly, the user 503 is viewing the display device 3 at an angle of substantially the viewpoint 5. Therefore, an image 405 shown in FIG. 4B is shown on the left eye of the user 503. Similarly, an image 406 shown in FIG. 4B is shown on the right eye of the user. As a result, the subject 400 in FIG. 4A is viewed as a screen 504 as a three-dimensional image viewed from the viewpoint 5. At this time, the user 503 touches the touch panel 60 arranged in front of the screen with a finger or the like at a position substantially the same as the position 502 on the screen relative to the display 3, that is, the position indicated by the position 505 on the screen. However, this does not mean that a portion 408 of the subject 400 is touched. This is a problem newly disclosed by the present invention.

  FIG. 6 is a diagram illustrating an example of functional blocks of the image processing unit 600 in the image processing apparatus 1 according to the present embodiment. The image processing unit 600 is controlled by the CPU 10 and includes a multi-viewpoint three-dimensional image information acquisition unit 601, a three-dimensional image generation unit 602, a three-dimensional image designation unit 603, and a three-dimensional image control unit 604.

  The multiview 3D image information acquisition unit 601 acquires 3D image information from the storage device 20 or the network control unit 5. The 3D image generation unit 602 generates a 3D image based on the 3D image information acquired by the 3D image information acquisition unit 601. The 3D image specifying unit 603 is generated by the 3D image generating unit 602 and displayed on the 3D display unit 50 based on a user instruction using the touch panel 60, the keyboard 70, the mouse 80, and the imaging device 90. Specify a dimensional image. By this designation, a part of the image information of the three-dimensional image information is designated. Details of this will be described later.

  The three-dimensional image control unit 604 controls the three-dimensional image designated by the three-dimensional image designation unit 603 by the CPU 10 in accordance with a user instruction using the touch panel 60, the keyboard 70, the mouse 80, and the like. For example, the three-dimensional image control unit 604 performs control so that image processing such as rotation processing and enlargement / reduction processing is performed on a designated three-dimensional image.

  FIG. 7 is a flowchart showing image designation processing of a multi-viewpoint three-dimensional image in the image processing apparatus 1 of the present embodiment. This flowchart is realized by the CPU 10 expanding and executing a program stored in the ROM 30 or the storage device 20 in the RAM 40, and each unit shown in FIG. 1 is controlled according to the processing of the following flowchart. The flow of processing of this embodiment will be described in detail using this flowchart.

  In the present embodiment, before instructing display processing for an image, processing for designating which viewpoint to instruct the image from which viewpoint is performed is performed. That is, in the three-dimensional image designation unit 603, a process of designating which viewpoint viewed from among the multi-viewpoint three-dimensional images is to be processed is performed.

  In step S <b> 700, the three-dimensional image generation unit 602 generates a three-dimensional image and displays the three-dimensional image on the three-dimensional display 50 in accordance with execution of an application program or the like that can browse a captured three-dimensional image as an album. . When the three-dimensional image is displayed, the process proceeds to step S701.

  In step S <b> 701, the three-dimensional image designation unit 603 detects an instruction to display a three-dimensional image viewed from the designated (viewpoint) direction. For example, when the CPU 10 designates a desired viewpoint image with a viewpoint designation button or the like in the application program, it is determined that there is a request to display a three-dimensional image viewed from the designated viewpoint in front. Then, the three-dimensional image designating unit 603 designates the three-dimensional image of the designated viewpoint from the multi-viewpoint three-dimensional image displayed on the three-dimensional display 50. When a three-dimensional image is designated, the process proceeds to step S702.

  In step S702, the three-dimensional image generation unit 602 sequentially displays a three-dimensional image viewed from each viewpoint of the multi-view three-dimensional image having the above-described six viewpoints according to the movement of the scroll bar of the touch panel 60. 50 is displayed on the front. It is assumed that 3D image information for 3D display conforms to a predetermined standard. The three-dimensional image generation unit 602 is stored in the storage device 20 or the RAM 40, and from among the image information acquired by the three-dimensional image information acquisition unit 601, the display storage element ( A three-dimensional image is generated by selectively supplying image information of each viewpoint to an unillustrated). When a three-dimensional image is generated and displayed, the process proceeds to step S703.

  In step S703, the three-dimensional image designation unit 603 detects the presence or absence of a touch input by the user. If it is detected that there is a touch input, the process proceeds to step S704. If it is detected that there is no touch input, the process returns to step S700.

  In step S704, the three-dimensional image designating unit 603 identifies which of the various display instructions the touch input by the user is based on the position touched by the user on the touch panel 60 and the operation of the finger at the time of the touch. To do. The display instruction is a so-called general touch operation such as double tap (twice pressing), scrolling, pinching out (spreading the screen with two fingers), pinching in (squeezing the screen with two fingers), etc. This means instructing the display state of an object. When the position touched by the user and the display instruction are identified, the process proceeds to step S705.

  In step S <b> 705, the three-dimensional image specifying unit 603 specifies an image touched on the touch panel 60 by the user. The designated image is an image displayed at the touch position in the three-dimensional image of the viewpoint displayed according to the viewpoint selection button 801 immediately before touching. When the user does not operate the viewpoint selection button 801 and the presence or absence of the touch input by the user is detected in step S703, the viewpoint image desired by the user is displayed as a multi-viewpoint three-dimensional image substantially in front. 4.

  The touch image specified in step S705 is an image displayed at the touch position of the viewpoint three-dimensional image displayed in accordance with the viewpoint selection button 801. When the viewpoint selection button 801 selects the viewpoint 6, the relative position of the touch image with the lenticular lens 202 of the three-dimensional display 50 is guaranteed. As shown in the three-dimensional image information 803, the image data displayed almost at the front pixel position is the images 6 and 7 shown in FIG. 3, that is, the two-dimensional image 7 for the right eye of the viewpoint 6 and the left eye of the viewpoint 6. This is a two-dimensional image 6 of FIG.

  Therefore, in this embodiment, the touch image is an image of the touch position in the two-dimensional image for the left eye of the user among the three-dimensional images displayed at that time. Of the two types of two-dimensional images for both eyes, whether to use the right eye or the left eye may be selectively switched each time or according to the touch position. Thus, the initial setting is made by using the left eye (or right eye). When the touched image is designated, the process proceeds to step S706.

  In step S706, the three-dimensional image control unit 604 controls the processing of the image specified in step S705. In the image processing control, interpolation processing is performed on the three-dimensional image of the viewpoint 6, that is, the images 6 and 7 shown in FIG. This is because the image portion at the touch position in the two-dimensional image for the left eye of the viewpoint 6 displayed by the three-dimensional image generation unit 602 among the three-dimensional images stored and held by the three-dimensional image acquisition unit 601 is displayed. Do it to the center. A known technique is used for enlargement interpolation processing of a three-dimensional image performed while considering parallax. For example, it is disclosed in detail in “Nobuhiro Tsunashima et al .: Creation of intermediate image data from a binocular stereo image considering occlusion, 3D Image Conference '95 pp.174-177 (1995)”. Also disclosed in “Takeo Agatsuma et al .: Parallax estimation using edge information for intermediate image generation, 3D Image Conference '95 pp.190-194 (1995)”.

  In this embodiment, it is predetermined that the touch operation identified in step S704 is a double tap for instructing enlargement, and the left-eye two-dimensional image is used. Therefore, in step S706, the processing of the image for enlarged display is controlled around the two-dimensional image for the left eye shown in FIG. 3, that is, the image portion at the touch position identified in step S704 on image 6. When the image processing is controlled, the process returns to step S700.

  FIG. 8 is a diagram illustrating a procedure for selecting a desired viewpoint from a multi-view three-dimensional image according to the present embodiment. The information displayed in the display area 800 at the center of the 3D display 50 is for displaying a 3D image of a specified viewpoint from the 3D image information for displaying the 3D image of the multiview described in FIG. To the three-dimensional image information. That is, the display area 800 controls the display so that the 3D image viewed from each viewpoint is sequentially displayed so that the user can view the 3D image only when viewing the 3D display 50 from the front. Specifically, the viewpoint selection button 801 displayed at the bottom of the screen is scrolled in the direction of the arrow by operating the touch panel 60. Then, the display image in the display area 800 is switched to a three-dimensional image in the order of viewpoint 6, viewpoint 5, viewpoint 4, viewpoint 3, viewpoint 2, and viewpoint 1 in accordance with the position of the button 801.

  A multi-viewpoint three-dimensional image is a three-dimensional image viewed from each viewpoint within one lenticular lens 202. In order to display a multi-viewpoint three-dimensional image, the three-dimensional image is displayed by three-dimensional image information 803 including all the two-dimensional image information constituting the multi-viewpoint three-dimensional image. However, when the process proceeds to step S <b> 702, for example, a three-dimensional image indicated by the three-dimensional image information 804 for the viewpoint 6 is displayed in the display area 800 according to the position of the viewpoint selection button 801. The three-dimensional image information 804 for the viewpoint 6 is changed to information for displaying the three-dimensional image of the viewpoint 6 when viewed from substantially the front of the three-dimensional display 50. That is, the three-dimensional image generation unit 602 displays the two-dimensional image information 6 and the two-dimensional image information 7 of the viewpoint 6 at the pixel position in the substantially central portion of the lenticular lens 202, and fixes the other pixel positions to black data. Thereby, when viewed from the front, the three-dimensional image of the viewpoint 6 can be seen, and nothing can be seen from other angles. That is, in step S702, a process of rewriting a part of the three-dimensional image information indicating the multi-viewpoint three-dimensional image to change (or generating) the three-dimensional image information indicating the specified viewpoint three-dimensional image is performed.

  Therefore, when the user moves the viewpoint selection button 801 in the direction of the arrow, the three-dimensional image information to be displayed is changed from the three-dimensional image information 804 to display the three-dimensional image of the viewpoint 6 to another viewpoint according to the position. Is changed to the three-dimensional image information. As a result, a three-dimensional image viewed from the viewpoint designated by the viewpoint selection button 801 can be sequentially displayed.

  In addition, processing that is not displayed (invalidation processing) is performed on a three-dimensional image other than the three-dimensional image of the designated viewpoint. Accordingly, it is possible to prevent an image different from a desired image from being selected when the user views the three-dimensional display 50 at a different angle. The user scrolls the viewpoint selection button 801 to display a three-dimensional image from a desired viewpoint, and if necessary, in the displayed three-dimensional image, select an image portion to be further specified on the touch panel. 60 can be touched.

  The relationship between the three-dimensional image information and the two-dimensional image information will be supplemented with reference to FIG. As described above, the two-dimensional image information is a signal source for displaying a two-dimensional image. For example, the two-dimensional image information 6 indicating the two-dimensional image corresponding to the left eye among the three-dimensional images of the viewpoint 6 is displayed at a predetermined pixel position where the relative position with the lenticular lens 202 of the three-dimensional display 50 is guaranteed. Image data. The three-dimensional image information includes a plurality of these two-dimensional image information, and also includes information indicating a relative position where each of the two-dimensional image information is arranged. In the example of FIG. 8, the three-dimensional image information is information representing the structure indicated by the three-dimensional image information 803 and 804. The three-dimensional image information and the two-dimensional image information may include other information such as various types of information obtained at the time of imaging in addition to the information or data described above.

  The present invention is characterized in that a desired three-dimensional image is designated for a multi-viewpoint three-dimensional image. Therefore, it is limited to the three-dimensional image of the designated arbitrary viewpoint or the content of the processing control for the whole image or a part of the image after designating the touch image in the image, and the subsequent processing means. There is no. In particular, in this embodiment, the 3D image at the viewpoint 6 is enlarged and displayed. However, the 3D images at other viewpoints can be enlarged in association with the image at the touch position. As a result, it is easy to return from step S706 to step S700 and display a three-dimensional image enlarged around the touch position as a multi-viewpoint three-dimensional image.

  Alternatively, when the image at the touch position is an object independent of the background image, the auxiliary information related to the object is displayed two-dimensionally using a balloon or the like, which is the control of the three-dimensional image information in the present invention. In particular, a multi-view three-dimensional image targeted by the present invention can be viewed by changing the viewpoint of an image portion that is hidden behind a shadow of another object image depending on the viewpoint. Thus, the control of the three-dimensional image information in the present invention also includes cutting out the hidden object image as a three-dimensional image independently as an object image independent of other images.

  As described above, according to the present embodiment, the user can specify a part of the displayed image on the touch panel with respect to the three-dimensionally displayed multi-viewpoint three-dimensional image, and can instruct an enlarged display of the image. become. Also, when specifying an image, it is possible to reliably specify an image from the viewpoint desired by the user by invalidating the three-dimensional image information related to the other viewpoint, so that an image different from the desired image It is possible to prevent the input of instructions.

<Example 2>
In the first embodiment, before touching an image on the touch panel 60, an embodiment is described in which a user selects a viewpoint three-dimensional image from which a desired image is easily touched from among images displayed in a multi-viewpoint three-dimensional manner. did. In the second embodiment, an embodiment in which an image desired by a user is directly designated by a touch operation without depending on such an operation will be described.

  In the present embodiment, the position of the user who operates and observes the three-dimensional display 50 is imaged by the imaging device 90. Thereafter, the position of the user with respect to the three-dimensional display 50 is recognized from the captured image, and the viewpoint of the user with respect to the three-dimensional display 50 is determined. A touch image is designated based on the determined user viewpoint or the touch position of the touch panel 60 with the viewpoint.

  FIG. 9 is a flowchart showing image designation processing of a multi-viewpoint three-dimensional image in the image processing apparatus 1 of the present embodiment. In the present embodiment, descriptions of step S900 and steps S903 to S906, which are the same processes as in the first embodiment described in FIG. 7, are omitted.

  In step S901, the 3D image designation unit 603 executes processing for recognizing the viewpoint of the 3D image displayed in step S900. The processing in step S901 will be described in detail with reference to FIGS.

  FIG. 10 is a diagram illustrating the principle of viewpoint recognition according to the present embodiment. The principle of designating the user's viewpoint from the position of the user's face recognized in the captured image 1000 will be described in detail with reference to FIG. In general, a three-dimensional display using the parallax of both eyes allows the observer to see the best three-dimensional image at a position away from the display by a predetermined distance Lm. That is, it can be estimated that the imaged user is imaged at a position approximately Lm away from the three-dimensional display 50. Accordingly, it is possible to obtain the distance Lo, which is a deviation of the user position from the center of the display device, assuming that the distance Li that is the distance between both eyes of the user in the captured image 1000 is typically 8 cm. In addition, the user's viewpoint is the position of the user on the virtual line 1001 shown at a distance Lm from the front surface of the 3D display 50 in FIG. 10, that is, the direction aiming at the center of the 3D display 50 from the point P.

  When the multi-viewpoint three-dimensional display 50 is viewed from the position of the point P, an image of the viewpoint that the user is viewing by determining which viewpoint of the three-dimensional image captured and displayed in advance is best viewed. Can be specified.

  That is, if the distance Lo from the center of the three-dimensional display 50 is a positive value, it is either the viewpoint 1, the viewpoint 2, or the viewpoint 3, and the distance Lo from the center of the three-dimensional display 50 is a negative value. Then, it is either viewpoint 6, viewpoint 5, or viewpoint 4.

  As shown in FIG. 10, by comparing the estimated position P of the user with the position where the line of sight of each displayed three-dimensional image intersects the virtual line 1001, a line of sight close to the user position P is obtained. It can be determined that the image corresponding to the viewpoint is a three-dimensional image desired by the user. L1 is a threshold for identifying viewpoint 3 and viewpoint 2, L2 is a threshold for identifying viewpoint 2 and viewpoint 1, -L1 is a threshold for identifying viewpoint 4 and viewpoint 5, and -L2 is for identifying viewpoint 5 and viewpoint 6. It is a threshold to do. These can be the distances from the center of the three-dimensional display 50 at the intermediate point where the line 1001 and both lines of sight intersect.

  FIG. 11 is a flowchart illustrating details of step S901 in the viewpoint recognition processing according to the present embodiment. The viewpoint recognition process in step S901 is executed by the three-dimensional image designation unit 603.

  In step S <b> 1111, the 3D image designating unit 603 controls the imaging device 90 to capture the front of the observer with the imaging device 90 installed at the center of the 3D display 50, and the captured image 1000 in FIG. 10. Is obtained. Image data indicating the captured image 1000 is temporarily stored in the RAM 40 of FIG. 1, and the process proceeds to step S1112.

  In step S1112, the three-dimensional image designation unit 603 recognizes the position of the user's face using a known pattern matching face recognition technique. In general, human face recognition can easily recognize the position of the center of gravity by paying attention to the presence of a pair of eyes arranged side by side. If the position of the user's face is recognized, the process proceeds to step S1113.

  In step S <b> 1113, the three-dimensional image designation unit 603 designates the user's viewpoint from the recognized face position, and detects the distance Lo between the viewpoint position and the center position of the three-dimensional display 50. If the distance Lo is detected, the process proceeds to step S1113.

  In step S1114, the three-dimensional image designation unit 603 determines the sign of the distance Lo. If the sign of the distance Lo is positive, the process proceeds to step S1115.

  If Lo> L2 in step S1115, the three-dimensional image designating unit 603 designates the user's viewpoint as the viewpoint 1. If Lo ≦ L2, the process proceeds to step S1116.

  If Lo> L1 in step S1116, the three-dimensional image designating unit 603 designates the user's viewpoint as the viewpoint 2. If Lo ≦ L1, the three-dimensional image designating unit 603 designates the viewpoint of the user as 3.

  If it is determined in step S1114 that the sign of the distance Lo is negative, the process proceeds to step S1120.

  In step S1120, if Lo <−L2, the three-dimensional image designating unit 603 designates the user's viewpoint as the viewpoint 6. If Lo ≧ −L2, the process proceeds to step S1121.

  In step S1121, if Lo <−L1, the three-dimensional image designating unit 603 designates the user's viewpoint as the viewpoint 5. If Lo ≧ −L1, the three-dimensional image designating unit 603 designates the user's viewpoint as the viewpoint 4. In the captured image 1000 shown in FIG. 10, the viewpoint of the user is designated as viewpoint 2.

  With the above processing, the three-dimensional image that the user is viewing in the viewpoint recognition processing in step S901 is designated as the viewpoint 2. Thereafter, the three-dimensional image designating unit 603 performs the processing in steps S902 to S904, and if the setting is a two-dimensional image for the left eye in step S905, the left eye image for viewpoint 2 is used in the same manner as in the first embodiment. Perform the process.

  According to the process of the present embodiment, the user's operation for designating a viewpoint from a multi-view three-dimensional image can be made unnecessary as compared with the process described in the first embodiment. For this reason, it becomes an Example excellent in operativity than Example 1.

  Note that, as a means for identifying the user's viewpoint, the present invention uses the imaging device 90 included in the normal display 3. However, for example, when the display 3 has two sets of imaging devices 90, the distance to the user can be identified more accurately, so that the user's viewpoint can be specified with higher accuracy than in this embodiment.

  In addition, the present invention can be implemented in a technique that includes two sets of infrared transmission / reception means, emits infrared rays from the display 3 in a substantially non-directional manner in the user direction, and detects the position of the user from the intensity of the reflected light. Alternatively, other generally known means such as a method of directly identifying the user's line of sight, a method of identifying from the relationship between the position of the display and the positional information such as GPS transmitted by the user, etc. are not limited to this embodiment. Therefore, the present invention can be implemented and an excellent effect can be obtained.

<Example 3>
Examples 1 and 2 are examples using a multi-viewpoint three-dimensional display 50 using a lenticular lens 202 as a parallax division method. However, the three-dimensional display is not limited to the one using the lenticular lens 202, and other three-dimensional displays can be used. In the third embodiment, an example in which a parallax barrier type three-dimensional display is used will be described.

  FIG. 12 is a diagram showing the display principle of the parallax barrier type three-dimensional display 1200 used in this embodiment. For example, the parallax barrier type three-dimensional display 1200 shown in FIG. 12 emits light from the display pixel for the other eye on the liquid crystal display unit 1201 and the line of sight seen from the left and right eyes in front of the display screen. An optical parallax barrier 1210 for shielding is provided.

  Accordingly, as shown in FIG. 12, the optical path of the left-eye pixel is blocked by the parallax barrier 1210 from the viewpoint of the right eye, and conversely, the optical path of the right-eye pixel is blocked by the parallax barrier 1210 from the viewpoint of the left eye. . As a result, since images of different viewpoints captured from the viewpoints of both eyes can be seen by both eyes, three-dimensional display is performed.

  The present embodiment using the parallax barrier type three-dimensional display 1200 will be described in detail with reference to FIG. In FIG. 13, the liquid crystal display unit 1201 includes a right-eye display pixel 170 and a left-eye display pixel 171 as adjacent pairs. When displaying the three-dimensional image viewed from the viewpoint 1, the three-dimensional image information 172 displays the two-dimensional image information of the image 2 shown in FIG. 3 in a strip shape at the right eye pixel position. In addition, the two-dimensional image information of the image 1 may be decomposed into a strip shape and displayed at the pixel position for the left eye. Here, as shown in FIG. 13, as the multi-viewpoint three-dimensional image, the three-dimensional image information to be sequentially displayed is switched and displayed in the same manner as in the first and second embodiments. For example, an image of a desired viewpoint can be designated. Also, as the two-dimensional image information used when designating the touch image, the two-dimensional image information for the left or right eye set in advance as in the case of the first and second embodiments may be used. The three-dimensional image designating unit 603 can select two-dimensional image information or three-dimensional image information to be displayed according to the principle of the three-dimensional display to be used and display the selected information on the three-dimensional display.

  That is, the three-dimensional image information of each viewpoint when selecting the viewpoint described with reference to FIGS. 7 and 8 can be similarly implemented by directly using the three-dimensional image information 172 and 173.

  In the present embodiment using the parallax barrier type three-dimensional display 1200, multi-viewpoint three-dimensional display cannot be performed at the same time, but display with high pixel density is possible, and an image can be specified more accurately.

<Other examples>
In the previous embodiment, the example in which the touch panel 60 is used to instruct a part of the display image that is displayed in three-dimensional three-dimensional display has been disclosed. However, the image processing apparatus shown in FIG. 1 that implements the present invention is configured. The PC 2 is a general personal computer. Therefore, the embodiment described in detail using the touch panel 60 can be similarly performed by pointing a part of an image displayed on the screen using a plurality of keys of the mouse, the touch pad 80, or the keyboard 70. Also, control for general designated images such as enlargement, movement, and cropping can be instructed. Therefore, it goes without saying that the effects of the present invention can be obtained similarly even if these input means are used.

  In the disclosed embodiment, three-dimensional image information for display desired by the user is controlled around a part of the image displayed on the screen. However, the entire viewpoint image is designated when the user designates an arbitrary viewpoint image from among the multi-viewpoint three-dimensional images or when the user's viewpoint is designated without showing a part of the screen with a touch panel or the like. I will do it. Therefore, it is possible to easily control the three-dimensional image information of the entire viewpoint image starting from the center of the viewpoint image.

  That is, it is clear from the prior art for a two-dimensional image that the object of the present invention is achieved and an effect is obtained without necessarily specifying a part of the designated three-dimensional image on the touch panel or the like.

  The present invention can also be realized by executing the following processing. That is, software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (12)

Image display means for inputting multi-viewpoint 3D image information and displaying a 3D image;
An image processing apparatus comprising: an image designating unit that designates a partial image of the three-dimensional image displayed by the image display unit.   The image processing apparatus according to claim 1, further comprising an image control unit that controls an image of a part of the three-dimensional image.   The image processing apparatus according to claim 2, wherein the image control unit performs processing control by interpolation processing on a part of the three-dimensional image specified by the image specifying unit.   4. The image designating unit according to claim 1, wherein the image designating unit designates an image of a part of the three-dimensional image based on the three-dimensional image information of an arbitrary viewpoint from the multi-viewpoint three-dimensional image information. The image processing apparatus according to item 1.   The image designating unit designates a partial image of the three-dimensional image by a user designating a position corresponding to the three-dimensional image displayed by the image display unit. 4. The image processing apparatus according to any one of items 3.   The image processing apparatus according to claim 4, wherein the image designating unit designates a partial image of the three-dimensional image at an arbitrary viewpoint by determining a relative position of a user. It further comprises imaging means arranged in the vicinity of the image display means,
The image designating unit displays a pixel position in the three-dimensional image information of an arbitrary viewpoint by determining a relative position of the user with respect to the image display unit by the imaging unit, and a part of the three-dimensional image The image processing apparatus according to claim 6, wherein the image is designated.   5. The image processing according to claim 4, wherein the image designating unit invalidates image data corresponding to a three-dimensional image other than a three-dimensional image corresponding to an arbitrary viewpoint from the three-dimensional image information. apparatus.   The image designating unit designates a part of the three-dimensional image by designating a position corresponding to the three-dimensional image information displayed by the image display unit using a touch panel, a mouse, or a touch pad. The image processing apparatus according to claim 5.   The image processing apparatus according to claim 1, wherein the image display unit uses a parallax barrier system or a lenticular lens system. An image display step for inputting multi-viewpoint 3D image information and displaying the 3D image;
And an image designating step for designating an image of a part of the three-dimensional image displayed in the image displaying step.   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 11.

Images