JP2018109740A - Display device, program, and display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display device which makes it easier to see images with different qualities.SOLUTION: A display device 30, which displays a first image CE with which a second image P with different qualities from that of the first image is overlapped, includes: determination means 35 for determining whether the second image to be seen; and image presentation means 36 for switching between a display of the existence of the second image and a non-display according to the result of determination by the determination means.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a display device, a program, and a display method.

  Imaging devices capable of capturing images with a wider angle than conventional digital cameras are becoming popular. As one of such imaging apparatuses, there is known an omnidirectional camera that combines a plurality of fish-eye lenses and the like to create an omnidirectional image in which 360 degrees around is captured by one imaging. The user can arbitrarily determine the viewing direction for the omnidirectional image and view the omnidirectional image from various directions.

  However, considering the load of image processing and the like, image processing of a spherical image with a wide imaging range is likely to be restricted, and the image quality is deteriorated compared to an image of a normal digital camera (for example, the resolution is low and whiteout is likely to occur) The color reproducibility is low). For this reason, when a user enlarges and browses a part of the omnidirectional image, there is an inconvenience that the image quality is deteriorated and it is difficult to obtain detailed information.

  For such inconvenience, a technique for fitting (superimposing) a telephoto image on a wide-angle image has been devised (see, for example, Patent Document 1).

  However, simply superimposing a high-quality image on a low-quality image has a problem that it is difficult for the user to determine where the high-quality image is superimposed on the low-quality image. This tendency becomes remarkable in an image that has a wide angle of view such as an omnidirectional image and the user cannot see the entire image at once. As an example, it is conceivable that the display device displays some mark in a place where a high-quality image is superimposed, but this causes another inconvenience that the mark becomes an obstacle when the user views a high-quality image.

  In view of the above problems, an object of the present invention is to provide a display device that facilitates browsing of a plurality of images having different qualities.

  The present invention provides a display device that displays the first image on which a second image having a quality different from that of the first image is superimposed, and determines whether or not the second image is browsed And an image presenting means for switching between display and non-display of the presence of the second image according to the determination result of the determination means.

  It is possible to provide a display device that facilitates browsing of a plurality of images having different qualities.

It is an example of the figure explaining an omnidirectional image. It is an example of the figure explaining the display method of a spherical image. 1 is a diagram illustrating a system configuration example of an image processing system having an omnidirectional camera and a display device. It is an example of the hardware block diagram of an imaging device. It is an example of the hardware block diagram of a display apparatus. It is a use image figure of a spherical camera. It is a figure explaining the outline | summary of the process until an omnidirectional image is produced from the image imaged with the omnidirectional camera. It is a figure explaining the outline | summary of the process until an omnidirectional image is produced from the image imaged with the omnidirectional camera. It is an example of the figure explaining a user's eyes | visual_axis. It is an example of the figure which illustrates typically expansion and reduction of a viewing area of a spherical image. It is an example of the functional block diagram which shows the function which a digital camera, a spherical camera, and a display apparatus have in a block form. It is a figure which shows an example of the position parameter which matches the projection system conversion to the coincidence area of a plane image, and a plane image and a coincidence area. It is an example of the figure explaining the projection system conversion image and mask data which a projection system conversion part produces. It is an example of the figure which illustrates the superimposition of the plane image on the omnidirectional image typically. It is an example of the figure which shows the equirectangular projection of the omnidirectional image on which the planar image was superimposed, and the omnidirectional image stuck on the omnidirectional sphere. It is a figure which shows typically the plane image superimposed on the omnidirectional image. It is an example of the flowchart figure which shows the procedure in which a display apparatus superimposes a planar image on an omnidirectional image. It is an example of the functional block diagram explaining the function of an image superimposition part to a block form. It is an example of the figure explaining the relative position of the plane image with respect to a browsing area. It is an example of the figure explaining how to obtain the angle of view a and the angle of view b. It is an example of the figure explaining the relationship between a browsing area | region and the diagonal vertex of a planar image. It is an example of the figure which illustrates the display method of a frame typically. It is an example of the figure explaining the example of the flame | frame displayed on an omnidirectional image. It is an example of the flowchart figure which shows the procedure which controls the presence or absence of the display of a frame, when a display apparatus displays a plane image. It is an example of the figure explaining the process of the automatic expansion of the planar image to a browsing area. It is an example of the figure explaining the process of the automatic expansion of a plane image in case the whole plane image is not contained in the browsing area | region. It is an example of the flowchart figure which shows the procedure which controls the presence or absence of the display of a frame, when a display apparatus displays a plane image. It is an example of the figure explaining the display and non-display of a frame. It is an example of the flowchart figure which shows the procedure which controls the presence or absence of the display of a frame, when a display apparatus displays a plane image. It is an example of the flowchart figure which shows the procedure which controls the presence or absence of the display of a frame, when a display apparatus displays a plane image. It is a figure which shows the example of a display of a frame. (Example 2) which is an example of the flowchart figure which shows the procedure which controls the presence or absence of the display of a frame by click or touch, when a display apparatus displays a plane image. It is an example of the flowchart figure which shows the procedure which controls the presence or absence of the display of a flame | frame by clicking or a touch, when a display apparatus displays a plane image. It is an example of the figure explaining the gaze detection by a display apparatus. It is an example of the flowchart figure which shows the procedure which controls the presence or absence of the display of a flame | frame according to a gaze position, when a display apparatus displays a plane image. It is an example of the figure explaining the distance of a browsing area | region and a planar image. It is an example of the flowchart figure which shows the procedure which controls the presence or absence of the display of a frame according to the distance with the center of a browsing area, when a display apparatus displays a planar image. It is an example of the flowchart figure which shows the procedure which controls the presence or absence of the display of a flame | frame according to the position of a mouse cursor, when a display apparatus displays a plane image.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

<Overview of spherical image>
FIG. 1 is an example of a diagram illustrating the omnidirectional image CE. 1A shows an omnidirectional image CE represented by a three-dimensional solid sphere CS, and FIG. 1B shows an omnidirectional image CE represented by equirectangular projection. The omnidirectional image CE generated by the omnidirectional camera has a three-dimensional structure in which an image as shown in FIG. The virtual camera IC corresponds to the viewpoint of the user. In FIG. 1, the viewpoint is at the center of the omnidirectional image CE. The user U rotates the three axes around the X axis, the Y axis, and the Z axis passing through the virtual camera IC, and is displayed on an arbitrary viewing area T (display or display software) of the omnidirectional image CE. Range) can be displayed. The user can also enlarge or reduce the browsing area T.

<Outline of Display Device of Present Embodiment>
FIG. 2 is an example of a diagram illustrating a method for displaying an omnidirectional image according to the present embodiment. FIG. 2A shows an example of the omnidirectional image CE. Assume that a display device to be described later superimposes a planar image P on the omnidirectional image CE. The display device displays a frame 40 indicating that the planar image P exists on the omnidirectional image CE. In FIG. 2A, a frame 40 indicating the outer edge (frame) of the planar image P is displayed. The user can confirm that the planar image P is pasted on the omnidirectional image CE and the position and size of the planar image P using the frame 40.

  It is assumed that the user has noticed the frame 40. Since a high-quality image with high resolution can be displayed, the user enlarges the omnidirectional image CE. Thereby, the planar image P is displayed largely on the display 508 as shown in FIG. The display device erases the frame 40 when a predetermined condition is satisfied. Thereby, it can suppress that the flame | frame 40 obstructs browsing of the plane image P.

The predetermined condition for erasing the frame 40 is that the user U is estimated to see the planar image P. Specifically:
(i) When the user U clicks the plane image P
(ii) When the ratio of the angle of view occupied by the planar image P to the angle of view of the current viewing area T is greater than or equal to a predetermined value. Erasing may be controlled.
(iii) When the coordinates of the mouse cursor do not overlap with the plane image P
(iv) After a predetermined time has elapsed since the viewing area T of the omnidirectional image CE has been changed, it is estimated that the user sees the image in addition to the actual user viewing.

  As described above, the display device of the present embodiment can supplement the omnidirectional image CE having a low resolution with the planar image P by superimposing the planar image P on the omnidirectional image CE. Since the position of the planar image P is displayed in the frame 40, the user can easily grasp where the planar image P is. When the user views the planar image P, the frame 40 is erased, so that the frame 40 can be prevented from obstructing the viewing of the planar image P.

<Terminology>
Superposition means that two or more objects are combined into one. In the present embodiment, the superimposition may include the meanings of pasting, fitting, synthesis, and superposition.

  The browsing of the claims means that the user views the second image with consciousness. The first image is, for example, an omnidirectional image, and the second image is, for example, a planar image.

  The presence of the second image is a visual feature that allows the user to notice the presence of the second image. Specifically, it is indicated by some image component. An example of the image component is the frame 40.

  Also, since the definition of high image quality varies depending on the image, it may vary depending on the user's viewing purpose, but generally it means that the image faithfully represents a scene. It can also be referred to as high image quality. For example, an image having a high resolution, a wide dynamic range, high color reproducibility, little noise, and the like. In addition, quality includes high and low image quality, differences between still images and moving images, and differences in projection methods.

<System configuration example>
FIG. 3 is a diagram illustrating a system configuration example of the image processing system 100 including the omnidirectional camera 20 and the display device 30. FIG. 3A shows an image processing system 100 having an omnidirectional camera 20 and a normal digital camera 9. The normal digital camera 9 is generally an image pickup apparatus (camera) that picks up an image with a focal length of 35 mm or more when converted into a 35 mm film, but may include an image pickup apparatus that can pick up a wide-angle image of about 24 mm to 35 mm. Simply, it can be said that the imaging device captures the planar image P instead of the omnidirectional camera 20. In other words, the imaging apparatus captures an image that can be displayed entirely in the browsing area T.

  The digital camera 9 has an image sensor that has at least a higher resolution than the omnidirectional image CE (a large number of pixels with respect to the angle of view to be imaged). In addition, since the imaging conditions (exposure time, shutter speed, white balance, etc.) are optimized in an imaging range narrower than that of the omnidirectional camera 20, whiteout is difficult and color reproducibility is often high. Thus, the digital camera 9 generates a so-called high-quality planar image.

  Thereby, in the omnidirectional image CE, an image that becomes rough when enlarged can be supplemented with a high-quality image of the planar image P. In the configuration of FIG. 3A, the digital camera 9 also serves as the display device 30. A specific example of the digital camera 9 is, for example, a digital still camera or a digital video camera, but may be other devices including an imaging device. For example, a smartphone, a tablet terminal, a PDA (Personal Digital Assistant), a wearable PC (Personal Computer), or the like may be used. In this case, the omnidirectional camera 20 may be externally attached to a smartphone or the like.

  The omnidirectional camera 20 and the digital camera 9 can communicate with each other by USB, Bluetooth (registered trademark), wireless LAN, or the like. The omnidirectional image CE captured by the omnidirectional camera 20 is transmitted to the digital camera 9. Similarly, the digital camera 9 captures the planar image P. Since the 360-degree landscape is reflected in the omnidirectional image CE, the planar image P becomes a part of the omnidirectional image CE.

  As the display device 30, the digital camera 9 superimposes the planar image P on the omnidirectional image CE. Specifically, for example, the user U selects the planar image P and the omnidirectional image CE and instructs superposition. Alternatively, when the digital camera 9 is set to the superimposing mode and the planar image P is captured, the digital image is automatically superimposed on the omnidirectional image CE with which the planar image P matches. There is little inconvenience because the images are not superimposed if the degree of coincidence of the images is low. Further, the digital camera 9 as the display device 30 switches between displaying and hiding the frame 40 and displays the frame 40 on the display of the digital camera 9. As described above, when the digital camera 9 is the display device 30, the omnidirectional camera 20 and the digital camera 9 are used to superimpose the omnidirectional image CE and the planar image P and to display and hide the frame 40. Can be switched.

  FIG. 3B shows an image processing system 100 having an omnidirectional camera 20, a normal digital camera 9, and a display device 30. The omnidirectional camera 20 and the display device 30 and the digital camera 9 and the display device 30 can communicate with each other by USB, Bluetooth (registered trademark), wireless LAN, or the like. The omnidirectional image CE captured by the omnidirectional camera 20 is transmitted to the display device 30. A planar image P captured by the digital camera 9 is also transmitted to the display device 30. In addition to transmission, the display device 30 may read the omnidirectional image CE or the planar image P stored in the storage medium from the storage medium.

  The display device 30 is an information processing device such as a PC, a smartphone, a tablet terminal, or a PDA. In addition, a multifunction device having a function as an information processing device, a projector, a video conference terminal, or the like may be used.

  In the configuration of FIG. 3B, the display device 30 superimposes the planar image P on the omnidirectional image CE. Further, the display and non-display of the frame 40 are switched and displayed on the display of the display device 30. More specifically, the user U can know which planar image P is a part of which omnidirectional image CE by comparing the planar image P with the omnidirectional image CE. A planar image P to be pasted is selected, and the display device 30 is pasted. You may superimpose automatically like the digital camera 9. Thus, the display device 30 can acquire the omnidirectional image CE and the planar image P, superimpose the planar image P on the omnidirectional image CE, and switch between display and non-display of the frame 40. In the description of FIG. 4 and subsequent figures, the configuration of FIG. 3B will be described in mind unless otherwise specified.

  FIG. 3C shows an image processing system 100 in which an omnidirectional camera 20, a normal digital camera 9, a display device 30, and an image superimposing server 70 are connected via a network N. The network N is constructed by a LAN constructed in a user environment, a provider network of a provider that connects the LAN to the Internet, a line provided by a line operator, and the like. When the network N has a plurality of LANs, the network N is called a WAN or the Internet. The network N may be constructed by either wired or wireless, and wired and wireless may be combined. In addition, when the omnidirectional camera 20 and the digital camera 9 are directly connected to the public network, they can be connected to the provider network without going through the LAN.

  The image superimposing server 70 may be a general information processing device or server device, and a specific example of the display device 30 may be the same as that in FIG. However, the image superimposing server 70 is preferably compatible with cloud computing. Cloud computing refers to a usage pattern in which resources on a network are used without being aware of specific hardware resources.

  The omnidirectional image CE captured by the omnidirectional camera 20 is transmitted to the image superimposing server 70. A planar image P captured by the digital camera 9 is also transmitted to the image superimposing server 70. In the configuration of FIG. 3C, the image superimposing server 70 superimposes the planar image P on the omnidirectional image CE. More specifically, the display device 30 receives the operation of the user U, accesses the image superimposing server 70, and displays a list of omnidirectional images CE and planar images P (thumbnail images and the like). The user U can know which planar image P is a part of which celestial sphere image CE by comparing the planar image P with the celestial sphere image CE. Is notified to the image superimposing server 70. The image superimposing server 70 may automatically superimpose.

  The image superimposing server 70 superimposes the planar image P on the omnidirectional image CE and transmits the image data of the viewing area T to the display device 30. It is assumed that the viewing direction and the angle of view in the initial state are determined in advance. The image superimposing server 70 transmits, as a Web server, image data in the viewing area T and a display program (described in a script language) for receiving a user operation to the display device 30. The display device 30 executes the display program, accepts an operation of the user U, and transmits the line-of-sight direction and the angle of view to the image superimposing server 70. The image superimposing server 70 updates the viewing area T according to the line-of-sight direction and the angle of view, and transmits image data of the viewing area T to the display device 30. Since the image superimposing server 70 switches between display and non-display of the frame 40, the display device 30 can display an image in which the frame 40 is displayed and an image in which the frame 40 is not displayed.

  The image superimposing server 70 may transmit the omnidirectional image CE, the planar image P, and the display program to the display device 30. In this case, the display device 30 determines the viewing area T according to the user's operation, and switches between display and non-display of the frame 40.

  Further, the image superimposing server 70 may calculate a position parameter (described later) from the omnidirectional image CE and the planar image P, and the display device 30 may download the omnidirectional image CE, the planar image P, and the positional parameter. Although the user arbitrarily changes the line-of-sight direction, the display device 30 can superimpose the planar image P on the omnidirectional image CE using the same position parameter. A position parameter creating unit 8 shown in FIG. 11 described later is provided in the image superimposing server 70, and the conversion display unit 7 is included in the display device 30.

<Hardware configuration>
<< Hardsphere camera hardware configuration >>
FIG. 4 is a hardware configuration diagram of the omnidirectional camera. In the following, the omnidirectional camera 20 is an omnidirectional imaging apparatus using two imaging elements, but the number of imaging elements may be three or more. In addition, it is not always necessary to use an apparatus dedicated to omnidirectional imaging. By attaching a retrofit omnidirectional imaging unit to a normal digital camera 9, a smartphone, or the like, it has substantially the same function as the omnidirectional camera 20. May be.

  As shown in FIG. 4, the omnidirectional camera 20 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, operation unit 115, network I / F 116, communication unit 117, antenna 117a, electronic compass 118, gyro sensor 119, and acceleration A sensor 120 or the like is also connected.

  Among these, the imaging unit 101 includes wide-angle lenses (so-called fish-eye lenses) 102a and 102b each having an angle of view of 180 ° or more for forming a hemispherical image, and two imaging units provided corresponding to the wide-angle lenses. Elements 103a and 103b are provided. The image sensors 103a and 103b are image sensors such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor that convert an optical image obtained by a fisheye lens into image data of an electric signal and output the image data. A timing generation circuit for generating a vertical synchronization signal, a pixel clock, and the like, and a register group in which various commands and parameters necessary for the operation of the image sensor are set.

  The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 103 a and 103 b of the imaging unit 101 are connected to a serial I / F bus (I2C bus or the like) separately from the imaging control unit 105. The image processing unit 104 and the imaging control unit 105 are connected to the CPU 111 via the bus 110. Further, ROM 112, SRAM 113, DRAM 114, operation unit 115, network I / F 116, communication unit 117, and electronic compass 118 are connected to the bus 110.

  The image processing unit 104 takes in the image data output from the image sensors 103a and 103b through the parallel I / F bus, performs predetermined processing on each image data, and then superimposes the image data. Then, image data is created by equirectangular projection as shown in FIG.

  In general, the imaging control unit 105 sets a command or the like in a register group of the imaging elements 103a and 103b using the I2C bus with the imaging control unit 105 as a master device and the imaging elements 103a and 103b as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to capture status data of the register groups of the imaging elements 103a and 103b and send it to the CPU 111.

  The imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Some imaging devices may have a preview display function using a display or a function corresponding to moving image display. In this case, output of image data from the image sensors 103a and 103b is continuously performed at a predetermined frame rate (frame / min).

  Further, as will be described later, the imaging control unit 105 also functions as a synchronization control unit that synchronizes the output timing of image data of the imaging elements 103a and 103b in cooperation with the CPU 111. In the present embodiment, the imaging device is not provided with a display unit, but a display unit may be provided.

  The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

  The CPU 111 controls the overall operation of the omnidirectional camera 20 and executes necessary processes. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data being processed, and the like. In particular, the DRAM 114 stores image data being processed by the image processing unit 104 and processed image data based on equirectangular projection.

  The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that has both display and operation functions, and the like. The user U inputs various imaging modes and imaging conditions by operating the operation buttons.

  The network I / F 116 is a general term for an interface circuit (USB I / F or the like) with an external medium such as an SD card or a personal computer. Further, the network I / F 116 may be a network interface regardless of wireless or wired. The image data of the equirectangular projection stored in the DRAM 114 is recorded on an external medium via the network I / F 116 or displayed via the network I / F 116 as a network I / F as necessary. Or transmitted to an external device such as the device 30.

  The communication unit 117 communicates with an external device such as the display device 30 by a short-range wireless technology such as WiFi (wireless fidelity) or NFC via the antenna 117 a provided in the omnidirectional camera 20. The communication unit 117 can also transmit image data based on equirectangular projection to an external device such as the display device 30.

  The electronic compass 118 calculates the direction of the omnidirectional camera 20 from the earth's magnetism and outputs the direction information. This orientation information is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image. The related information includes each data of the image capturing date and time and the data capacity of the image data.

  The gyro sensor 119 detects a change in angle (roll angle, pitching angle, yaw angle) accompanying the movement of the omnidirectional camera 20. The change in angle is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image.

  The acceleration sensor 120 detects acceleration in three axis directions. Based on the detected acceleration, the attitude of the omnidirectional camera 20 (angle with respect to the direction of gravity) is detected. By having both the gyro sensor 119 and the acceleration sensor 120, the accuracy of image correction is improved.

  Note that the hardware configuration of the single-lens reflex camera is the same as or different from that in FIG.

<< Hardware configuration of display device >>
FIG. 5 is a hardware configuration diagram of the display device. The display device 30 includes a CPU 501 that controls the operation of the entire display device, a ROM 502 that stores a program used to drive the CPU 501 such as an IPL (Initial Program Loader), a RAM 503 that is used as a work area for the CPU 501, and a program for the display device HD (Hard Disk) 504 for storing various data such as. In addition, an HDD (Hard Disk Drive) 505 that controls reading or writing of various data with respect to the HD 504 according to the control of the CPU 501 and a media drive 507 that controls reading or writing (storage) of data with respect to a recording medium 506 such as a flash memory. . In addition, a display 508 for displaying various information such as a cursor, menu, window, character, or image, a network I / F 509 for data communication using the network N, a character, a numerical value, various instructions for inputting, etc. A keyboard 511 having a plurality of keys, and a mouse 512 for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. Further, as shown in FIG. 5, an optical media drive 514 that controls reading or writing of various data with respect to an optical storage medium 513 as an example of a removable recording medium, and the above-described components are electrically connected. For this purpose, a bus line 510 such as an address bus or a data bus is provided.

  It should be noted that the hardware configuration of the image superimposing server 70 is the same as or different from that in FIG.

<Creation of spherical image CE>
The omnidirectional image CE will be described with reference to FIGS. FIG. 6 is a usage image diagram of the omnidirectional camera 20. As shown in FIG. 6, the omnidirectional camera 20 is used by the user U to take an image of a subject around the user U with his hand. The omnidirectional camera 20 has a structure in which the back surfaces of the two imaging elements are opposed to each other, and obtains two hemispherical images by imaging the subject around the user U, respectively.

  Next, an outline of processing until an omnidirectional image CE is created from an image captured by the omnidirectional camera 20 will be described with reference to FIGS. 7 and 8. 7A shows a hemispheric image (front side) captured by the omnidirectional camera 20, FIG. 7B shows a hemispheric image captured by the omnidirectional camera 20 (rear side), and FIG. 7C. These are the figures which showed the omnidirectional image by equirectangular projection. FIG. 8A is a conceptual diagram showing a state where a sphere is covered with an image obtained by equirectangular projection, and FIG. 8B is a diagram showing an omnidirectional image CE.

  As shown in FIG. 7A, the image obtained by the omnidirectional camera 20 is a hemispherical image (front side) curved by the fisheye lens. Further, as shown in FIG. 7B, the image obtained by the omnidirectional camera 20 is a hemispherical image (rear side) curved by the fisheye lens. Then, the hemispherical image (front side) and the hemispherical image inverted at 180 degrees (rear side) are joined together by the omnidirectional camera 20, and as shown in FIG. An image is created.

  Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), as shown in FIG. 8A, an image by equirectangular projection is pasted so as to cover the spherical surface. An omnidirectional image CE as shown in (b) is created. As described above, the omnidirectional image CE is represented as an image in which an image obtained by equidistant cylindrical projection faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. The omnidirectional image CE may be a still image or a moving image. Specifically, the expression (1) described later is used for conversion from the image by the equirectangular projection of FIG. 8A to the three-dimensional omnidirectional image of FIG. 8B.

  Since the omnidirectional image CE is an image pasted so as to cover the spherical surface, it is curved and has a sense of incongruity when viewed by humans. Therefore, the display device 30 displays a part of the viewing area T of the omnidirectional image CE as a flat image P with a small curvature so as not to give a sense of incongruity to humans. The browsing area T is indicated by coordinates (X, Y, Z) in a three-dimensional virtual space. On the other hand, since the display 508 is a two-dimensional plane, the display device 30 cannot display the viewing area T as it is.

  Therefore, the display device 30 obtains an image with less curvature by perspective projection conversion in which a three-dimensional object is projected onto a two-dimensional plane using a 3D computer graphic technique. As described above, the viewing area T of the omnidirectional image CE is displayed on the display 508.

  FIG. 9 is an example of a diagram illustrating the line of sight of the user U. Since the omnidirectional image CE has three-dimensional coordinates, the line-of-sight direction is specified by information specifying the coordinates of the sphere such as three-dimensional coordinates and latitude / longitude. In the present embodiment, the center cp of the viewing area T is the line-of-sight direction.

  Although the user U can change the line-of-sight direction with the keyboard 511 or the mouse 512, assuming that the virtual camera IC does not translate, the virtual camera IC is rolled as a rigid body (rotation about the Z axis), yaw (Y axis) Rotation around the X axis) and pitch (rotation around the X axis) are possible. Of these, the direction of the line of sight changes regardless of whether the rotation of the yaw or the pitch occurs (changing the roll only changes the upper vector of the camera and does not change the line of sight of the camera).

  For example, when the user U rotates the omnidirectional image CE in the horizontal direction, the yaw angle changes. When the user U rotates it in the vertical direction, the pitch angle changes, and the omnidirectional image CE is rotated about the center of the display 508. And roll angle changes. In the present embodiment, the operation of the user U on the Web page is reflected in the line-of-sight direction (roll angle, yaw angle, pitch angle) and the like. How it is reflected is described in advance in a program executed by the display device 30.

  FIG. 10 is an example of a diagram schematically illustrating enlargement and reduction of the viewing area T of the omnidirectional image CE. FIG. 10A shows an initial browsing area T. FIG. When the virtual camera IC exists in the center of the solid sphere CS, the viewing area T is determined by the angle of view α. The angle of view α is an angle at which the diagonal vertex of the viewing area T faces the center of the solid sphere CS, for example.

The angle of view α in the initial state is α ₀ . If the angle of view is reduced as shown in FIG. 10B (α ₁ <α ₀ ), the viewing area T becomes narrower, so the image is enlarged on the display 508, and as shown in FIG. Is increased (α ₂ > α ₀ ), the viewing area T becomes wider, and the image is reduced on the display 508. When the user U performs an operation for enlarging or reducing the omnidirectional image CE, the display device 30 decreases or increases the angle of view according to the operation amount.

  Even if the angle of view increases as shown in FIG. 10C, as long as the virtual camera IC is in the center, the display device 30 cannot display an image behind the virtual camera IC. Therefore, when the user U performs an operation of further reducing from the state of FIG. 10C, the display device 30 moves the virtual camera IC backward. In FIG. 10 (d), the angle of view α2 is not different from that in FIG. 10 (c), but the viewing area T is further expanded by moving the virtual camera IC backward. For this reason, it is possible to further reduce the image as compared with FIG.

<Function related to superimposition of planar image P on omnidirectional image CE>
First, the function regarding the superimposition of the planar image P on the omnidirectional image CE will be described with reference to FIGS. As described above, the configuration of FIG. 3B will be described in mind.

  FIG. 11 is an example of a functional block diagram illustrating the functions of the digital camera 9, the omnidirectional camera 20, and the display device 30 in a block shape. The digital camera 9 has a planar image acquisition unit 11. The planar image acquisition unit 11 is realized by the imaging function of the digital camera 9, and creates image data (planar image P) in which each pixel of a predetermined resolution represents light and shade. The planar image P may be one still image, a plurality of still images, or a moving image.

  The omnidirectional camera 20 has an omnidirectional image acquisition unit 12. The omnidirectional image acquisition unit 12 is realized by the imaging function of the omnidirectional camera 20, and creates 360 degree image data (omnidirectional image CE) around the omnidirectional camera 20. The omnidirectional image CE may be one still image, a plurality of still images, or a moving image.

  The display device 30 mainly includes a position parameter creation unit 8 and a conversion display unit 7. These functions of the display device 30 are functions or means realized by the CPU 501 of the display device 30 illustrated in FIG. 5 executing a program expanded from the HD 504 to the RAM 503.

  First, the position parameter creation unit 8 further includes an omnidirectional image reading unit 21, a planar image reading unit 22, a position parameter calculation unit 23, and a position parameter writing unit 24. The omnidirectional image reading unit 21 reads the omnidirectional image CE from the omnidirectional image acquisition unit 12. The planar image reading unit 22 reads the planar image P from the planar image acquisition unit 11. Reading means acquiring, receiving, reading from a storage medium, or receiving input.

  The position parameter calculation unit 23 specifies the matching area of the omnidirectional image CE that matches the planar image P, and obtains the position parameter PP for specifying the matching area. Details of the position parameter PP will be described with reference to FIG. Since the omnidirectional image CE is distorted by the equirectangular projection, it is preferable that the flat image P is subjected to the same conversion to be distorted. Then, the position parameter PP can be obtained by extracting both feature points and performing matching. For detection of feature points, various detection methods such as edge detection, corner detection, SIFT feature amount and SURF feature amount, and a central point of the same color are known. Alternatively, the sum of the absolute values of the difference between the pixel values of the omnidirectional image CE and the planar image P or the sum of squares of the differences is calculated by shifting one pixel at a time, and the plane when the sum of the absolute values or the sum of squares becomes the smallest The position of the image P may be the matching area.

  Further, if the planar image P is divided into a mesh area and matched with the omnidirectional image CE, it is possible to take into account deviation correction due to lens distortion or the like.

  Since the correspondence between the omnidirectional image CE and the planar image P is uniquely determined, the position parameter PP need only be obtained once. The position parameter writing unit 24 sends the position parameter PP to the position parameter writing unit 24.

  The conversion display unit 7 further includes a planar image reading unit 25, a line-of-sight direction angle-of-view instruction unit 26, an omnidirectional image reading unit 27, a projection method conversion unit 1 (reference numeral 28), and a projection method conversion unit 2 (reference numeral 29). The image superimposing unit 33, the position parameter reading unit 32, and the image display unit 31. The functions of the planar image reading unit 25 and the omnidirectional image reading unit 27 are the same as those of the position parameter creation unit 8. Further, the position parameter reading unit 32 acquires the position parameter PP from the position parameter creating unit 8.

  The line-of-sight direction angle-of-view instruction unit 26 receives the line-of-sight direction and the angle of view (enlargement / reduction) operated by the user U. The viewing area T is determined because the line-of-sight direction and the angle of view are input by operation.

  In addition, since the user U arbitrarily operates the line-of-sight direction and the angle of view (enlargement / reduction), only a part of the planar image P that has undergone the projection method conversion enters the viewing area T, or the projection method has been converted. The planar image P may not enter the browsing area T at all. For this reason, the projection method conversion unit 1 generates mask data indicating a region included in the browsing region T in the projection method conversion image. Details will be described with reference to FIG.

  The projection method conversion unit 2 determines the viewing area T corresponding to the viewing direction and the viewing angle received by the viewing direction view angle instruction unit 26, and the image display unit 31 for the viewing area T of the read omnidirectional image CE. Perspective projection conversion according to the display size is performed to create a display image. Therefore, the display image is a two-dimensional plane image.

  The image superimposing unit 33 generates a superimposed image by superimposing the projection method conversion image on the display image using the mask data. Details will be described with reference to FIG. The image display unit 31 displays the superimposed image on the display 508.

<Position parameter>
FIG. 12 is a diagram illustrating an example of the projection method conversion to the matching area of the planar image P and the position parameter PP for associating the planar image P with the matching area. It is assumed that correspondence between the planar image P and the omnidirectional image CE has already been obtained by matching. The position parameter PP is the latitude and longitude of the omnidirectional image CE where the planar image P exists when the feature points are the best match.

  FIG. 12A shows a planar image P, and FIG. 12B shows an omnidirectional image CE of equirectangular projection. FIG. 12B shows a matching area 301. The position parameter calculation unit 23 divides the planar image P into a grid pattern, and calculates the point (λ, φ) of the coincidence region 301 of the omnidirectional image CE corresponding to the coordinates (x, y) of the intersection of the grids. It is assumed that the position parameter PP. Note that λ is longitude and φ is latitude. FIG. 12C shows an example of the position parameter PP. The coordinates of the grid of the planar image P and the latitude and longitude of the matching area 301 of the omnidirectional image CE are associated with each grid.

<Processing of Projection Method Conversion Unit 1>
FIG. 13 is an example of a diagram illustrating a projection method conversion image 302 and mask data 303 created by the projection method conversion unit 1. FIG. 13A shows a projection method conversion image 302. The projection system conversion image 302 has an area having the same size as the viewing area T of the omnidirectional image represented by the equirectangular projection, and the planar image P after the projection system conversion is pasted on the matching area 301. The planar image P is in a trapezoidal distortion state by the projection system conversion of the planar image P to the coincidence area 301. A portion where the planar image P is not present in the projection system converted image 302 is a gray uniform image 307. Since the uniform image 307 is not used for superposition, any pixel value may be used.

  FIG. 13B shows mask data 303 corresponding to the planar image P of FIG. The mask data 303 is data for extracting the planar image P included in the viewing area T from FIG. The white pixel area of the mask data indicates the browsing area T and the plane image P area. Therefore, the white pixel area of the mask data is equal to or smaller than the area of the planar image P. The mask data in FIG. 13B is a case where the viewing area T is larger than the planar image P. Therefore, in the example of FIG. 13A, the matching area 301 is substantially equal to the white pixel area of the mask data 303. . When the viewing area T does not overlap the flat image P at all, the entire mask data is black pixels. When the viewing area T is smaller than the planar image P, the mask data 303 is only white pixels. Thus, the white pixel and the matching area 301 of the mask data 303 are always the same size and the same position.

  The image superimposing unit 33 performs a mask process using the projection method converted image 302 and the mask data 303. This mask process is a process for extracting pixels corresponding to white pixels of the mask data 303 from the projection method conversion image 302. The image superimposing unit 33 extracts the pixel value at the position of the white pixel from the projection method conversion image 302 and superimposes it on the same position as the projection method conversion image 302 of the display image.

  The boundary between the black pixel and the white pixel in the mask data 303 is preferably provided with a slope that gradually changes from white to black from the white pixel to the black pixel. Thereby, the boundary between the low-resolution spherical image CE and the high-resolution planar image P can be made inconspicuous.

<Processing of image superimposing unit>
The process of the image superimposition part 33 is demonstrated using FIG. FIG. 14 is an example of a diagram schematically illustrating the superimposition of the planar image P on the omnidirectional image CE. As shown in FIG. 14, a display image 304 obtained by perspective projection conversion of the viewing area T by the projection method conversion unit 2, a projection method conversion image 302 (FIG. 13A), and mask data 303 (FIG. 13B). Input to the image superimposing unit 33.

  The image superimposing unit 33 superimposes the projection method conversion image 302 having a wider angle of view on the display image 304. By the mask processing using the mask data 303, the projection method conversion image 302 at the pixel position of the white pixel in the mask data 303 is taken out and overwritten on the display image 304 (in this case, the pixels of the display image 304 are lost at this time). ) In this way, in the superimposed image 305, the high-resolution planar image P1 is arranged on the low-resolution omnidirectional image CE.

  Alternatively, a transparent layer may be prepared, and the projection method conversion image 302 at the pixel position of the white pixel in the mask data 303 may be taken out and placed in the transparent layer. In this case, there is no loss in the display image 304, and for example, the user can switch between display and non-display of the planar image P1.

  The image superimposing unit 33 outputs a superimposed image 305. In the superimposed image 305, a high-resolution planar image P is arranged on a low-resolution omnidirectional image CE.

  In this embodiment, the superimposing method is also characterized. As described above, the display device 30 performs the perspective projection conversion of the omnidirectional image CE and the superimposition of the planar image P in parallel. If the perspective projection conversion is performed after the planar image P is pasted on the omnidirectional image CE, the resolution of the omnidirectional image CE needs to be the same as that of the planar image P, and the amount of data increases. In the present embodiment, since the planar image P is superimposed after the display image 304 is created, an increase in the data amount can be suppressed.

  The display device 30 repeats the superimposition of the omnidirectional image CE and the planar image P in the display cycle of the display 508 (for example, 30 to 60 times per second). By doing so, the display device 30 can hold the planar image P and the omnidirectional image CE one by one, and generate the superimposed image 305 in real time from the viewpoint according to the user's operation.

<Conversion of the equirectangular projection to the celestial sphere image CE>
FIG. 15A is a diagram showing the omnidirectional image CE by the equirectangular projection, and FIG. 15B is an omnidirectional image in which the omnidirectional image CE of the equirectangular projection is pasted on the omnisphere. A spherical image CE is shown. The conversion from the image in FIG. 15A to the image in FIG. 15B is processing performed by the omnidirectional camera 20 when the omnidirectional camera 20 described with reference to FIG.

The coordinates of a three-dimensional point (x, y, z) corresponding to an arbitrary point (λ, φ) in equirectangular projection can be obtained by the following equation, assuming that the radius of the sphere is 1.0.
X = cos (φ) * cos (λ)
Y = sin (φ) (1)
Z = cos (φ) * sin (λ)
Thus, a three-dimensional omnidirectional image CE is obtained from the omnidirectional image CE of equirectangular projection.

<Plane image P in the spherical image CE>
FIG. 16 is a diagram schematically illustrating a planar image P superimposed on the omnidirectional image CE. As described above, since the planar image P is superimposed on the display image 304 as it is, the planar image P is not pasted on the solid sphere CS as shown in FIG. However, when viewed from the user, as shown in FIG. 16, it appears that the planar image P is superimposed on a part of a sphere having the same center position and the same radius as the omnidirectional image CE.

  In addition, as shown in FIG. 16, the planar image P may be superimposed on the three-dimensional omnidirectional image CE. Since the longitude and latitude corresponding to the planar image P can be known by matching (by the position parameter PP), the display device 30 can paste the planar image P on the celestial sphere according to Expression (1). In this case, the hidden surface removal method using the depth information (so-called Z buffer method) or the like is not used, and rendering may be performed with priority.

  In the case where there are a plurality of plane images P, if the position parameter calculation unit 23 calculates the position parameters PP for the respective plane images P, they can be superimposed in the same manner. In this case, when the image is captured by the digital camera 9 having the same number of pixels with the wide-angle image first and the telephoto later, the lower-resolution planar image P is placed on the higher-resolution planar image P. Can be avoided.

<Superposition procedure>
FIG. 17 is an example of a flowchart illustrating a procedure in which the display device 30 superimposes the planar image P on the omnidirectional image CE. The process in FIG. 17 is started when the user gives an instruction to superimpose. Steps S10 to S30 and S40 to S60 are executed in parallel, but may be executed in order.

  The planar image reading unit 22 reads the planar image P (S10). The omnidirectional image reading unit 21 reads the omnidirectional image CE (S20). Then, the position parameter calculation unit 23 calculates the position parameter PP (S30).

  The planar image reading unit 25 reads the planar image P (S40). The omnidirectional image reading unit 27 reads the omnidirectional image CE (S50). The line-of-sight direction angle-of-view instruction unit 26 receives the line-of-sight direction and the angle of view (S60). Step S60 is performed as needed.

  Next, the projection method conversion unit 1 performs the projection method conversion on the plane image P with the position parameter PP, and generates the projection method conversion image 302 (S70).

  Next, the projection method conversion unit 1 generates mask data 303 according to the viewing area T determined by the line-of-sight direction and the angle of view and the planar image P (coincidence region) converted by the projection method (S80).

  Next, the projection method conversion unit 2 performs perspective projection conversion of the omnidirectional image CE according to the viewing area T, and generates a display image 304 (S90).

  Next, the image superimposing unit 33 superimposes the projection method conversion image 302 and the display image 304 using the mask data, and generates a superimposed image 305 (S100).

  Next, the image display unit 31 displays the superimposed image 305 (S110). The display device 30 repeats the processing of steps S60 to S110 in FIG.

<Functions related to frame display control>
Next, functions relating to display control of the frame 40 in the superimposed image 305 will be described with reference to FIGS. FIG. 18 is an example of a functional block diagram illustrating the function of the image superimposing unit 33 in a block shape. The image superimposing unit 33 further includes a determination unit 35 and a frame display unit 36. The determination unit 35 determines whether or not the frame 40 is to be displayed using the current viewing area T and the position of the planar image P (position parameter PP), and the determination result (frame display, frame deletion) is displayed in the frame display unit. 36. The frame display unit 36 displays or deletes the frame 40 according to the determination result.

<Determination of presence / absence of display of frame 40>
When it is estimated that the user U sees the planar image P superimposed on the superimposed image 305 as described above, the frame 40 is deleted, and the frame 40 remains displayed in other states. The determination of the predetermined conditions (i) to (iv) will be described.

  First, a determination method of “(ii) when the ratio of the plane image P to the angle of view of the current viewing area T exceeds a predetermined value” will be described.

  FIG. 19 is an example of a diagram illustrating the relative position of the planar image P with respect to the viewing area T. In FIG. 19A, the entire planar image P is included in the viewing area T. Since the viewing area T is the area of the omnidirectional image displayed on the display 508, the user U is viewing the planar image P almost from the front. If the viewing angle of the viewing area is a and the viewing angle of the planar image P is b, if the viewing angle a> the viewing angle b, the entire planar image P and a part of the omnidirectional image CE are visible. Become. In the situation as shown in FIG. 19A, it is estimated that the user U wants to see the planar image P.

  However, when the omnidirectional image CE is somewhat larger than the planar image P, the user U does not have to determine that he / she wants to view the planar image P. Therefore, for example, when the angle of view a is larger than 1.2 times the angle of view b, the determination unit 35 determines to display the frame 40. That is, when the angle of view a is larger than 1.2 times the angle of view b, it is determined that the ratio of the planar image P to the current angle of view is less than a predetermined value. When the angle of view a is 1.2 times or less of the angle of view b, the determination unit 35 determines that the frame 40 is not displayed. That is, when the angle of view a is 1.2 times or less of the angle of view b, it is determined that the ratio of the planar image P to the current angle of view has become a predetermined value or more.

  The browsing area T is known. Further, since the latitude and longitude of the planar image P are registered in the position parameter PP, the angle of view b can be obtained from the position parameter PP. Details will be described with reference to FIGS.

  In FIG. 19B, the viewing area T is narrower than the planar image P. That is, the user U sees only the planar image P. In this case, of course, the frame 40 is not displayed.

  In FIG. 19 (c), only the upper side of the planar image P is the viewing area T. That is, the user U is looking above the planar image P. However, since the viewing area T is completely included in the planar image P (the angle of view a is 1.2 times or less of the angle of view b), the determination unit 35 determines that the frame 40 is not displayed.

  In FIG. 19D, the deviation (angle) between the center of the viewing area T and the center of the planar image P is large, and a part of the viewing area T protrudes from the planar image P. In this case, since the planar image P is smaller than the viewing area T (the angle of view a is 1.2 times or less of the angle of view b), the determination unit 35 determines to display the frame 40.

  It should be noted that the threshold value for whether or not to display the frame 40 is set to 1.2 times as long as the value is larger than 1.0. For example, it can be determined as 1.1 to 1.5 times. Moreover, the user U may be able to set a threshold value.

FIG. 20 is an example of a diagram for explaining how to obtain the angle of view a and the angle of view b. The angle of view a is obtained from the viewing area T determined by the user U by operation. When the distance from the virtual camera IC to the viewing area T is f and the length of the diagonal line of the viewing area T is 2L, the angle of view α, the distance f, and L have the relationship of Expression (2).
Lf = tan (α / 2) (2)
Since the coordinates of the diagonal vertices of the viewing area T are known by the operation of the user U, the diagonal length 2L can be easily calculated. The distance f is known from the radius of the celestial sphere or that the user U has operated. Therefore, the angle of view α can be calculated from the equation (2).

  The angle of view b can be calculated in the same manner, but the angle of view b may be calculated when the viewing area T includes even a portion of the planar image P. FIG. 21 is a diagram showing the relationship between the viewing area T and the planar image P. The determination unit 35 determines whether each of the four vertices determines whether the planar image P is in the viewing area T. The four vertices of the viewing area T are A1 to A4, and the four vertices of the planar image P are B1 to B4. The vertices B1 to B4 are specified by, for example, the longitude and latitude of the omnidirectional sphere, but may be specified by coordinates on the display 508.

  For example, the vertex B1 is determined as follows.

The longitude of vertex A1 ≦ the longitude of vertex B1 ≦ the longitude of vertex A2, and
It can be similarly determined for the latitude of the vertex A1 ≧ the latitude of the vertex B1 ≦ the latitude vertices B2, B3, and B4 of the vertex A4. If the diagonal vertices (B1 and B3, B2 and B4) are in the viewing area T, the angle of view b can be calculated from the position parameter PP according to equation (2). As shown in FIG. 21, when all of the diagonal vertices B1 to B4 of the planar image P are not in the browsing area T, the angle of view b is set by the diagonal vertex A3 in the browsing area T from the vertex B1 in the browsing area T. Is calculated. The diagonal vertex of vertex B1 is vertex A3, the diagonal vertex of vertex B2 is vertex A4, the diagonal vertex of vertex B3 is vertex A1, and the diagonal vertex of vertex B4 is vertex A2. This is set in the display device 30.

<Frame display method>
The location (coincidence region 301) of the planar image P in the omnidirectional image is represented by the position parameter PP. Among the positional parameters PP in FIG. 12C, a grid having a value of x = 0.5 or y = 0.5 is the outer edge of the planar image P. Since the latitude and longitude of the grid having a value of x = 0.5 or y = 0.5 is set in the position parameter PP, the frame display unit 36 draws a dotted line that becomes the frame 40 at the latitude and longitude. .

  Or you may perform the projection system conversion similar to the projection system conversion part 1. FIG. The frame 40 can be obtained by projecting the outer edge rectangle of the planar image P.

  FIG. 22 is an example of a diagram for schematically explaining the display method of the frame 40. The frame display unit 36 prepares a transparent layer having the same size (number of pixels) as the omnidirectional image CE of equirectangular projection. Then, the frame display unit 36 draws a dotted line that becomes the frame 40 at the latitude and longitude of the lattice having the values x = 0.5, −0.5 or y = 0.5, −0.5 in the position parameter PP. To do.

  The frame display unit 36 uses the mask data to extract an area corresponding to the white pixel of the mask data from the transparent layer. Then, the frame 40 representing the outer edge of the planar image P is displayed by superimposing the extracted transparent layer on the superimposed image 305.

  When the frame 40 is hidden, the frame display unit 36 may hide the transparent layer.

<Display example>
FIG. 23 is an example of a diagram illustrating an example of the frame 40 displayed on the omnidirectional image CE. FIG. 23A to FIG. 23D show a state in which the user U operates the display device 30 and gradually enlarges the omnidirectional image CE. In FIG. 23A, almost the entire celestial sphere image CE is included in the viewing area T. In the state of FIG. 23A, since the angle of view a is larger than 1.2 times the angle of view b, the frame 40 is displayed. In addition, when the distance between the center of the planar image and the center cp of the viewing area T is less than the threshold (for example, the threshold is a length of 1/4 of the viewing angle of the viewing area T), not the ratio of the angle of view. Also good.

  Even when the omnidirectional image CE is gradually enlarged as shown in FIGS. 23B and 23C, the frame 40 is displayed because the angle of view a is larger than 1.2 times the angle of view b. In FIG. 23D, since the angle of view a is 1.2 times or less of the angle of view b, the frame 40 is not displayed.

<Operation procedure>
FIG. 24 is an example of a flowchart illustrating a procedure for controlling whether or not the frame 40 is displayed when the display device 30 displays the planar image P. The procedure in FIG. 24 starts when the display device 30 displays the omnidirectional image CE. It is assumed that the planar image P has already been superimposed on the omnidirectional image CE.

  The determination unit 35 determines whether the viewing area T has been changed or whether the mouse has been operated (S210). The browsing area T being changed means that the user U has changed the line-of-sight direction, or has performed an enlargement or reduction operation. Specifically, the line-of-sight direction angle-of-view instruction unit 26 changes the operation of the user U to the line-of-sight direction and the angle of view and sends it to the image superimposing unit 33. Further, it may be determined whether or not the mouse is simply operated. Thereby, even if the browsing area T is not changed, the processing after step S220 proceeds with some mouse event as a trigger.

  The image superimposing unit 33 identifies the viewing area T based on the viewing direction and the angle of view (S220). That is, the latitude and longitude ranges corresponding to the viewing area T in the omnidirectional image CE are specified.

  Next, the determination unit 35 determines whether or not a part of the planar image P is included in the viewing area T (S230). If it does not enter at all, it is not necessary to display the frame 40, so the process proceeds to step S280.

  If at least a part of the planar image P is included in the viewing area T (Yes in S230), the determination unit 35 calculates the angle of view b of the planar image P (S240). Also, the viewing angle a of the viewing area is calculated (S250).

  Then, the determination unit 35 compares the ratio a / b between the angle of view a and the angle of view b with the threshold value 1.2, and determines whether the ratio a / b is larger than the threshold value 1.2 (S260).

  If the determination in step S260 is Yes, it is not estimated that the user U views the planar image P, so the determination unit 35 determines to display the frame 40 (S270). The frame display unit 36 takes out the browsing area T from the layer of the frame 40 using the mask data and displays it. Thereby, the location and size of the planar image P can be notified to the user U.

  If the determination in step S260 is No, it is estimated that the user U sees the planar image P, so the determination unit 35 determines not to display the frame 40 (S280). The frame display unit 36 hides the layer of the frame 40. Thereby, when the user U sees the plane image P, it can suppress that the flame | frame 40 becomes obstructive.

  The image display unit 31 displays the superimposed image 305 on which the planar image P is superimposed on the omnidirectional image CE and the frame 40 is displayed or not displayed on the display 508 (S290).

<Judgment method of (i)>
Next, a determination method of “(i) When user U clicks planar image P” will be described. When the user U clicks on the planar image P, it becomes clear that the user U wants to view the planar image P. Therefore, the frame display unit 36 needs to display the frame 40 even if the condition (ii) is not satisfied. Disappears. Further, since it is clear that the user U wants to view the planar image P, if the projection method conversion unit 2 displays the planar image P on the entire surface of the viewing area T, the user U operates so that the planar image P is in front. It is not necessary and convenient. Specifically, this is a display method in which the viewing area T is changed from FIG. 23 (a) to FIG. 23 (d) when the user U clicks the planar image P indicated by the frame 40 in FIG. 23 (a). . Such a display method will be described as automatic enlargement of the planar image P. In addition, when displaying on the apparatus provided with the touch panel, the same operation | movement is performed when the plane image P is touched.

  The automatic enlargement of the planar image P will be described with reference to FIG. FIG. 25 is an example of a diagram illustrating the process of automatically expanding the planar image P to the browsing area T. It is assumed that the user U clicks in a state where the mouse cursor 311 is overlapped with the planar image P. In the case of a touch panel, it corresponds to the user U tapping the planar image P with a finger. A mouse cursor or touch panel is called a pointing device. Hereinafter, in order to simplify the description, the mouse cursor 311 will be described as an example.

  The line-of-sight direction angle-of-view instruction unit 26 converts the coordinates on the display 508 where the mouse cursor 311 is clicked into the three-dimensional coordinates of the omnidirectional image CE. This conversion corresponds to a conversion opposite to the perspective projection conversion. The determination unit 35 converts the three-dimensional coordinates of the mouse cursor 311 into latitude and longitude, and determines whether or not the coordinates of the mouse cursor 311 are included in the plane image P. Judgment is made as follows.

Longitude of vertex B1 ≦ longitude of coordinates of mouse cursor ≦ longitude of vertex B2, and
The latitude of the vertex B1 ≧ the latitude of the coordinates of the mouse cursor ≦ the latitude of the vertex B4 When the coordinates of the mouse cursor 311 are included in the planar image P, the determination unit 35 determines that the planar image P has been clicked.

  In this case, the projection method conversion unit 2 automatically enlarges the planar image P. The projection method conversion unit 2 performs image processing for gradually enlarging the planar image P to match the viewing area T. Specifically, a line 312 connecting the vertex A1 and the vertex B1, a line 312 connecting the vertex A2 and the vertex B2, a line 312 connecting the vertex A3 and the vertex B3, and a line 312 connecting the vertex A4 and the vertex B4 are equally spaced. Interpolate. In FIG. 25, the line 312 is divided into four. Interpolation points on the line 312 obtained by the interpolation are P1i to P4i (i is an integer of 1 to 4).

  The process of automatically enlarging the planar image P is a process of reducing the angle of view a of the viewing area T. The image display unit 31 reduces the viewing area T to an area connecting the interpolation points P13, P23, P33, and P43 (decreases the angle of view a). Next, the viewing area T is reduced to an area connecting the interpolation points P12, P22, P32, and P42. Next, the viewing area T is reduced to an area connecting the interpolation points P11, P21, P31, and P41. Next, the viewing area T is reduced to an area connecting the vertices B1, B2, B3, and B4 of the planar image P.

  By such image processing, it seems to the user U that the planar image P gradually increases. Since the perspective projection conversion is performed on the viewing area T and displayed on the entire surface of the display 508 (the entire surface of the display software), the user U can see the planar image P large.

  In addition, even when the entire planar image P is not included in the browsing area T, it can be automatically enlarged similarly. FIG. 26 is an example of a diagram illustrating the process of automatically enlarging the planar image P when the entire planar image P is not included in the viewing area T. In FIG. 26, only the vertex B4 of the planar image P is in the viewing area T. However, since the coordinates of the vertices B1 to B3 of the planar image P are known only by not being displayed in the viewing area T, they can be automatically enlarged as in FIG.

<Operation procedure>
FIG. 27 is an example of a flowchart illustrating a procedure for controlling whether or not to display the frame 40 when the display device 30 displays the planar image P. In the description of FIG. 27, differences from FIG. 24 are mainly described.

  The processing in steps S210 and S220 may be the same as in FIG. Next, the determination unit 35 determines whether or not the planar image P is clicked (S222). If the determination in step S222 is No, the subsequent processing is the same as in FIG.

  If the determination in step S222 is Yes, the determination unit 35 determines not to display the frame 40 (S224).

  Then, the projection method conversion unit 2 automatically enlarges the planar image P (S226). With the above processing, the user U does not need to operate so that the planar image P is in front.

<Display control of the frame 40 based on the coordinates of the mouse cursor>
Next, a determination method of “(iii) When the mouse cursor coordinates do not overlap the planar image P” will be described. That is, the determination unit 35 determines that the frame 40 is displayed when the coordinates of the mouse cursor overlap with the planar image P, and determines that the frame 40 is not displayed when the coordinates of the mouse cursor do not overlap with the planar image P. To do. The method for determining whether or not the coordinates of the mouse cursor overlap with the planar image P has been described above.

  FIG. 28 is an example of a diagram for explaining display and non-display of the frame 40. In FIG. 28A, since the planar image P and the mouse cursor 311 overlap, the frame 40 is displayed. In FIG. 28B, since the planar image P and the mouse cursor 311 do not overlap, the frame 40 is not displayed.

  FIG. 29 is an example of a flowchart illustrating a procedure for controlling whether or not to display the frame 40 when the display device 30 displays the planar image P. In the description of FIG. 29, differences from FIG. 27 are mainly described.

  The processing in steps S210 to S230 may be the same as in FIG. When it is determined Yes in step S230, the determination unit 35 determines whether or not the mouse cursor overlaps the planar image P (S232).

  When the mouse cursor overlaps with the planar image P (Yes in S232), the determination unit 35 determines to display the frame 40 (S270). When the mouse cursor does not overlap with the planar image P (No in S232), the determination unit 35 determines that the frame 40 is not displayed (S280).

  By doing so, the display device 30 displays the frame 40 only when the user U overlays the mouse cursor on the planar image P, so that the user U can easily view the omnidirectional image CE. Further, when the user U moves the mouse cursor and the mouse cursor and the plane image P overlap, the frame 40 is displayed, so it is easy to find where the plane image P is.

  In contrast to FIG. 29, when the coordinates of the mouse cursor overlap with the plane image P, the determination unit 35 determines that the frame 40 is not displayed, and when the coordinates of the mouse cursor do not overlap with the plane image P. It may be determined that the frame 40 is displayed. In this case, since the time during which the frame 40 is displayed is generally longer than the time during which the frame 40 is not displayed, the user U can easily find the planar image P. If the mouse cursor is superimposed on the planar image P, the frame 40 is deleted. A plane image P can be seen.

<Display control of frame 40 based on rotation of omnidirectional image CE>
Next, a determination method of “(iv) After a predetermined time has elapsed since the viewing area T of the omnidirectional image CE is changed” will be described. The determination unit 35 determines to display the frame 40 for a predetermined time immediately after the omnidirectional image CE is rotated.

  FIG. 30 is an example of a flowchart illustrating a procedure for controlling whether or not the frame 40 is displayed when the display device 30 displays the planar image P. In the description of FIG. 30, differences from FIG. 29 are mainly described.

  The processing in steps S210 to S230 may be the same as in FIG. When it is determined Yes in step S230, the determination unit 35 determines to display the frame 40 (S270). That is, when the viewing area is changed in Step S10 (when the omnidirectional image CE is rotated), if the viewing area T contains the planar image P, the frame display unit 36 displays the frame 40.

  Next, the determination unit 35 determines whether or not a predetermined time has elapsed since the frame 40 was displayed (S272). The predetermined time is several seconds, for example. It is sufficient that the user U notices the frame 40, and the user U may set it.

  When the predetermined time has elapsed, the determination unit 35 determines not to display the frame 40 (S280). That is, the frame 40 can be deleted when the browsing area T is rotated or enlarged / reduced and a predetermined time elapses. Therefore, the user U can easily notice the flat image P, and the browsing of the flat image P is hardly disturbed.

<Frame display example>
The frame 40 is an image component for informing the user U of the location of the planar image P or calling attention, but any image component can be used to notify the user U of the location of the planar image P or call attention. Form may be sufficient. The frame 40 can be arbitrarily changed in hue or luminance.

  FIG. 31A shows an example of the form of the frame 40. The frame 40 in FIG. 31A is displayed with white pixels. In the case of white pixels, the user U can easily notice the location of the planar image P when the omnidirectional image CE is dark. In the case of black pixels as shown in FIG. 23, the user U can easily notice the location of the planar image P when the omnidirectional image CE is bright. The frame display unit 36 may automatically switch between white pixels and black pixels according to the average of the pixel values of the planar image P. You may give the display effect that the flame | frame 40 blinks or the dotted line of the flame | frame 40 rotates.

  Further, as shown in FIG. 31B, the frame display unit 36 may change the hue in the frame 40. For example, when the omnidirectional image CE is color, the frame display unit 36 converts the planar image P into black and white. In this case, it is not necessary to display the frame 40 itself. The frame display unit 36 converts the planar image P into black and white.

  Similarly, the planar image P may be a sepia color or may be reduced in color. Further, when the omnidirectional image CE is black and white, the planar image P may be colored. In this case, it is preferable that the planar image P is originally color. The frame display unit 36 converts the omnidirectional image CE into black and white.

Further, the brightness of the planar image P may be reversed. The luminance is calculated by “Y = 0.299R + 0.587G + 0.114B”. When RGB takes a value of 0 to 255, the luminance also takes a value of 0 to 255. The reversal of luminance means that the luminance is replaced with an opposite value around 127.
Value after inversion = | Value before inversion−255 |
By doing so, the brightness of the planar image P is different from the surroundings, so that the user can notice the planar image P at a glance. In addition to luminance inversion, gradation inversion, brightness inversion, hue inversion, and the like may be performed.

  Further, the frame display unit 36 may blink the planar image P. Blinking means that the state in which the color tone is changed with respect to the aspect of the planar image P and the normal state are alternately switched. For example, the frame display unit 36 alternately switches between a state in which the color tone is changed and a normal state every few seconds. By doing so, the user becomes more easily aware of the planar image P.

  In addition, as shown in FIG. 31C, the frame display unit 36 may display an icon 313 indicating the planar image P. Since the user U can guess that the flat image P exists inside the icon 313, the flat image P can be viewed by enlarging the icon 313 so that the inside of the icon 313 is in front.

<Summary>
As described above, the image processing system 100 according to the present embodiment can supplement the low-quality omnidirectional image CE with the flat image P by superimposing the flat image P on the omnidirectional image CE. Since the position of the planar image P is displayed in the frame 40, the user can easily grasp where the planar image P is. When the user views the planar image P, the frame 40 is erased, so that the frame 40 can be prevented from obstructing the viewing of the planar image P.

  In the first embodiment, the display or non-display of the frame 40 is controlled by the triggers (i) to (iv). However, in this embodiment, some other examples of the trigger of the display or non-display of the frame 40 will be described. To do.

  In the present embodiment, the components denoted by the same reference numerals in the first embodiment perform similar functions, and therefore, only the main components of the present embodiment may be mainly described.

A. Clicking or tapping the planar image P In (i) of the first embodiment, it has been described that the frame is hidden by clicking or touching. Conversely, when the user clicks or touches the planar image P, the frame is displayed or It may be hidden.

  FIG. 32 is an example of a flowchart illustrating a procedure for controlling whether or not to display a frame when the display device 30 displays the planar image P. In the description of FIG. 32, differences from FIG. 24 are mainly described. The processing from step S210 to step S290 may be the same as in FIG.

  Subsequent to step S290, the frame display unit 36 determines whether or not the planar image P is clicked or tapped (S32-1). Regardless of whether or not the frame is displayed, whether or not the clicked or tapped position overlaps the planar image P is determined using the mask data.

  If the determination in step S32-1 is Yes, the frame display unit 36 determines whether a frame is currently displayed (S32-2).

  When the frame is displayed, the frame display unit 36 hides the frame (S32-3), and when the frame is not displayed, the frame display unit 36 displays the frame (S32-4).

  When the frame is displayed or clicked or touched, the projection method conversion unit 2 automatically enlarges the planar image P (S224).

  Thus, even when the ratio a / b between the angle of view a and the angle of view b is less than 1.2 and no frame is displayed, the frame can be displayed by clicking or tapping. Further, when a frame is displayed, clicking or tapping can hide the frame and automatically enlarge the planar image. Therefore, the user can switch between displaying and hiding frames according to his / her preference.

  In the process of FIG. 32, a frame is displayed when the ratio a / b between the angle of view a and the angle of view b is greater than 1.2, but simply switching between display and non-display by a user's click or tap. Also good.

  FIG. 33 is an example of a flowchart illustrating a procedure for controlling whether or not to display a frame when the display device 30 displays the planar image P. In the processing of FIG. 33, no frame is displayed even if a planar image is included in the viewing area.

  Subsequent to step S220, the frame display unit 36 determines whether or not the planar image P is clicked or tapped (S32-1).

  When the planar image is clicked or tapped, the frame display unit determines whether a frame is currently displayed (S32-2).

  When the frame is displayed, the frame display unit 36 hides the frame (S32-3), and when the frame is not displayed, the frame display unit 36 displays the frame (S32-4).

  According to such processing, the user can switch between display and non-display of the frame by clicking or tapping the planar image. In this process, the display and non-display of the frame are switched by clicking or tapping, so that automatic enlargement by clicking becomes difficult. However, automatic enlargement may be performed from a menu displayed by right-clicking, etc. You may go.

  As a modified example, the frame display unit 36 may automatically display a frame on a planar image in the viewing area, and the frame may be automatically deleted when a certain time elapses after the frame is displayed. The user can recognize the presence of a planar image and browse a spherical image without a frame. Moreover, the display and non-display of a frame can be switched by clicking or tapping.

B. Switching between display and non-display of a frame by gaze detection It is also possible to switch between display and non-display of a frame by gaze detection instead of click or touch.

  FIG. 34 is an example of a diagram for explaining line-of-sight detection by the display device 30. The display device 30 of this embodiment has a line-of-sight detection device 90. The line-of-sight detection device 90 has a camera and captures a facial image including at least eyes. The gaze detection device 90 analyzes the image data of the face image and detects the gaze direction. The line-of-sight direction is detected based on the relative position between the reference point and the moving point. As an example, there are a method of selecting an eye for the reference point and an iris for the moving point, and a method of selecting a corneal reflection for the reference point and a pupil for the moving point. When corneal reflection is used, the line-of-sight detection device 90 emits a point light source. The line-of-sight detection device 90 identifies a face part from the image data, identifies characteristic parts such as eyebrows, nostrils, eyes, and lips from the face part, and identifies the position of the eye from these arrangement relationships. When the position of the eye is specified, the eye head, moving point, corneal reflection, and pupil can be detected. A table in which the line-of-sight direction is associated with the relative position of the reference point and the moving point is facilitated in advance, and the line-of-sight detection device detects the line-of-sight direction with reference to this table. The line-of-sight direction is represented, for example, by a vector in a three-dimensional coordinate system with the origin at the center of the image sensor of the line-of-sight detection device. If the relative position of the viewing area with reference to the image sensor of the line-of-sight detection device is clear, the coordinates (line-of-sight position) of the line-of-sight arrival point on the display (viewing area) of the display device 30 are specified. The frame display unit 36 periodically acquires these coordinates and determines whether or not they overlap with the planar image P.

  FIG. 35A is an example of a flowchart illustrating a procedure for controlling whether or not to display a frame when the display device 30 displays the planar image P. FIG. In the description of FIG. 35A, differences from FIG. 32 are mainly described.

  The frame display unit 36 determines whether or not the planar image P is watched based on the line-of-sight position (S35-1). Although the frame may or may not be displayed, the line-of-sight position is not used for control when the frame is displayed as described below. The frame display unit 36 determines whether or not the line-of-sight position overlaps the planar image P using the mask data.

  When the determination in step S35-1 is Yes, the frame display unit 36 determines whether a frame is currently displayed (S35-2).

  When the frame is displayed, the frame display unit 36 keeps the frame displayed (S35-3), and when the frame is not displayed, the frame display unit 36 displays the frame (S35-4).

  When the planar image is clicked (S35-5), when the frame is displayed (S35-6), the frame display unit 36 hides the frame (S35-7), and the projection method conversion unit. 2 automatically enlarges the planar image P (S224). The frame display unit 36 displays a frame when no frame is displayed (S35-8).

  As described above, even if the frame is not displayed, the frame is displayed only by the user's gaze. Therefore, when the planar image P exists at the line-of-sight position, the user can notice the planar image P. In addition, when the frame is already displayed, the frame remains displayed even when the planar image P is watched, so that the frame can be prevented from hunting. When a planar image is clicked, the frame can be automatically enlarged when the frame is displayed, and the frame can be displayed when the frame is not displayed.

  When the user wants to delete a frame intentionally, a method of pressing a button for that purpose or clicking or tapping as shown in FIG. 33 can be considered. Alternatively, the frame may be automatically deleted when a certain time has elapsed after the frame is displayed.

  Further, as shown in FIG. 35B, a frame may be displayed when the user gazes at the planar image P and clicks or taps it.

  In FIG. 35B, the frame display unit 36 determines whether or not the planar image P is watched based on the line-of-sight position (S35-1).

  When the determination in step S35-1 is Yes, the frame display unit 36 displays the frame when the planar image is clicked (S35-5) (S35-8), and does not display the frame unless it is clicked (S35-). 7).

  In such a process, since a frame is not displayed only by gaze, it can suppress that a user feels troublesome. You can hide the frame if you don't watch it.

C. A frame is displayed when the planar image P exists near the center of the viewing area. If the planar image P exists near the center of the viewing area, it is assumed that the user is interested in the subject of the planar image P. By displaying the frame, the user can be notified of the presence of the planar image P.

  FIG. 36 is a diagram illustrating the distance between the viewing area and the planar image P. The frame display unit 36 determines whether or not to display a frame by calculating a distance L between the center of the viewing area and the center of the planar image P and comparing it with a threshold value. In FIG. 36A, since the distance L is equal to or greater than the threshold, no frame is displayed. In FIG. 36B, the frame is displayed because the distance L is less than the threshold value.

  FIG. 37 is an example of a flowchart illustrating a procedure for controlling whether or not to display a frame when the display device 30 displays the planar image P. In the description of FIG. 37, differences from FIG. 35 are mainly described.

  If it is determined in step S230 that the planar image P is in the viewing area, the frame display unit 36 calculates the distance between the viewing area and the planar image P and determines whether the distance is less than a threshold (S37-1). ). The threshold value may be determined experimentally. For example, 1/3 of the number of diagonal pixels in the viewing area, 1/4, etc.

  If the distance between the viewing area and the planar image P is less than the threshold, the frame display unit 36 determines to display a frame (S37-2). If the distance is equal to or greater than the threshold, the frame display unit 36 determines not to display the frame. (S37-3). The subsequent processing is the same as in FIG.

  As described above, when there is a subject that the user wants to see near the center of the viewing area, the frame is displayed, so that the user can notice the presence of the planar image P.

D. Displayed when the mouse cursor is overlapped with the planar image P and displayed when it is not overlapped In Example 1, displayed when the mouse cursor is overlapped with the planar image P, not displayed when not overlapped However, conversely, when the mouse cursor overlaps with the plane image P, it may be hidden and displayed when it does not overlap. By doing so, the user can browse the planar image P without a frame by overlaying the mouse cursor.

  FIG. 38 is an example of a flowchart illustrating a procedure for controlling whether or not to display a frame when the display device 30 displays the planar image P. In the description of FIG. 38, differences from FIG. 37 are mainly described.

  Subsequent to step S230, the frame display unit 36 determines whether or not the mouse cursor overlaps the planar image P (S38-1).

  When the mouse cursor overlaps with the plane image P, the frame display unit 36 determines that no frame is displayed (S38-2). When the mouse cursor does not overlap with the plane image P, the frame display unit 36 displays a frame. Then, it is determined (S38-3). The subsequent processing may be the same as in FIG.

  As described above, when the mouse cursor does not overlap with the planar image P, the frame is displayed so that the user can notice the location of the planar image P. When the mouse cursor overlaps with the planar image P, the planar image is displayed. P can be viewed without a frame.

<Summary>
As described above, the display device 30 of this embodiment can control the display or non-display of the frame 40 triggered by various events.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the scope of the present invention. And substitutions can be added.

  For example, the above (i) to (iv) automatically switch display / non-display of the frame 40, but the user U may intentionally specify display / non-display.

  Further, the display and non-display of the frame 40 are switched while the plane image P is displayed. However, the image superimposing unit 33 displays the plane image P only when the frame 40 is displayed. The planar image P may be hidden when not displayed.

  Further, the display of the omnidirectional image may be performed by browser software, or may be performed by application software for displaying the omnidirectional image CE.

  Further, the omnidirectional image of the present embodiment may be image data having an angle of view that cannot be displayed in the viewing area T. For example, an image having an angle of view of 180 degrees to 360 degrees only in the horizontal direction may be used.

  In addition, the configuration examples in FIGS. 11 and 18 are divided according to main functions in order to facilitate understanding of processing by the omnidirectional camera 20, the display device 30, and the digital camera 9. The present invention is not limited by the way of dividing the processing unit or the name. The processing of the omnidirectional camera 20, the display device 30, and the digital camera 9 can be divided into more processing units according to the processing content. Moreover, it can also divide | segment so that one process unit may contain many processes.

  The determination unit 35 is an example of a determination unit, the frame display unit 36 is an example of an image presentation unit, the line-of-sight direction angle-of-view instruction unit 26 is an example of an operation reception unit, and the image display unit 31 is an image display unit. It is an example. The line-of-sight detection device 90 is an example of a line-of-sight position detection unit.

7 conversion display unit 8 position parameter creation unit 9 digital camera 20 omnidirectional camera 23 position parameter calculation unit 24 position parameter writing unit 26 gaze direction angle of view instruction unit 30 display device 32 image superimposing unit 33 image superimposing unit 35 determining unit 36 Frame display unit 100 Image processing system

Japanese Patent Laid-Open No. 2006-96487

Claims (18)

A display device for displaying the first image on which a second image having a quality different from that of the first image is superimposed,
Determining means for determining whether or not the second image is viewed;
An image presentation unit that switches between display and non-display of the presence of the second image according to a determination result of the determination unit;
A display device.   The display device according to claim 1, wherein the image presentation unit switches between display and non-display of an image component indicating a position or size of the second image according to a determination result of the determination unit. When the determination means determines that the second image is browsed, the image presentation means does not display the image component,
The display device according to claim 2, wherein when the determination unit determines that the second image is not browsed, the image presentation unit displays the image component.   The determination means compares the ratio of the area where the second image is displayed to the viewing area of the first image displayed on the display device with a threshold value, and determines whether or not the second image is browsed. The display device according to claim 2, which determines whether or not. Operation receiving means for receiving an operation on the first image;
The determination means determines whether or not the coordinates received by the operation reception means overlap the second image,
4. The display device according to claim 2, wherein when the determination unit determines that the coordinates overlap the second image, the image presentation unit does not display the image component. 5. When the determination unit determines that the coordinates overlap the second image when the second image is displayed on the display device,
The display device according to claim 5, further comprising an image display unit that gradually enlarges and displays the second image to a display size of the display device. The determination means determines whether or not the coordinates of the pointing device for receiving the operation by the operation reception means overlap the second image,
When the determination unit determines that the coordinates of the pointing device overlap the second image, the image presentation unit displays the image component,
The display device according to claim 5 or 6, wherein when the determination unit determines that the coordinates of the pointing device do not overlap the second image, the image presentation unit does not display the image component. The determination means determines whether or not the coordinates of the pointing device for receiving the operation by the operation reception means overlap the second image,
When the determination unit determines that the coordinates of the pointing device overlap the second image, the image presentation unit does not display the image component,
The display device according to claim 5 or 6, wherein when the determination unit determines that the coordinates of the pointing device do not overlap the second image, the image presentation unit displays the image component. When the operation accepting unit accepts a change in the viewing area of the first image,
7. The image presentation unit displays the image component for a predetermined time after the viewing region is changed, and the image presentation unit does not display the image component when a predetermined time has elapsed since the viewing region is changed. The display device described in 1.   The first image is an omnidirectional image in which 360 degrees around the imaging device is imaged, and the second image is an image that can be entirely displayed in a viewing area of the first image. The display device according to any one of 9. Operation receiving means for receiving an operation on the first image;
The determination means determines whether or not the coordinates received by the operation reception means overlap the second image,
4. The display device according to claim 2, wherein when the determination unit determines that the coordinates overlap with the second image, the image presentation unit displays the image component. Gaze position detection means for detecting the gaze position of the user,
The determination means determines whether the line-of-sight position detected by the line-of-sight position detection means overlaps the second image;
4. The display device according to claim 2, wherein when the determination unit determines that the line-of-sight position overlaps the second image, the image presentation unit displays the image component. The determining means determines whether the distance between the center of the first image and the center of the second image is less than a threshold;
The display device according to claim 2, wherein the image presentation unit displays the image component when the determination unit determines that the distance is less than a threshold.   The display device according to claim 2, wherein the image component is a frame indicating an outer edge of the second image.   The display device according to claim 2, wherein the image component is an image indicating the second image.   The fact that the second image exists is indicated by a mode in which the hue or luminance of the second image is different from that of the first image. The display device described in 1. A display device for displaying the first image on which a second image having a quality different from that of the first image is superimposed;
Determining means for determining whether or not the second image is viewed;
Image presenting means for switching between display and non-display of the presence of the second image according to the determination result of the determination means;
Program to function as. A display method performed by a display device that displays the first image on which a high-quality second image having a quality different from that of the first image is superimposed,
A step of determining whether or not the second image is viewed;
An image presenting means switching between display and non-display of the presence of the second image according to the determination result of the determination means;
Display method.