JP2012191380A - Camera, image conversion apparatus, and image conversion method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a display image more natural and easier to view than conventional display images, from a distorted image captured using a wide-angle lens.SOLUTION: An imaging part 101 obtains a captured image using a wide-angle lens 102. In a space model storage part 111, there is stored a space model in which a floor, a ceiling, a wall, a table, etc. included in an imaging space are made into models (templates). A projection conversion part 109 maps the captured image so as to be projected in the space model. A plane conversion part 112 obtains the display image using the image mapped in the space model.

Description

  The present invention relates to a camera, an image conversion apparatus, and an image conversion method for obtaining an image with a natural and unnatural feeling from a distorted image captured using a wide-angle lens such as a fisheye lens.

  An image captured using a wide-angle lens such as a fisheye lens is generally an image that is distorted toward the peripheral edge of the image. Conventionally, distortion correction is often applied to such a distorted image.

  FIG. 1 shows a state of conventional distortion correction. FIG. 1 shows an image obtained by capturing a black and white horizontal stripe pattern parallel to each other using a wide-angle lens. FIG. 1A shows an image before distortion correction, and it can be seen that the peripheral portion is distorted due to optical distortion. FIG. 1B shows an image after distortion correction. Distortion at the peripheral portion is eliminated, but the peripheral portion is enlarged, resulting in an unnatural image.

  Therefore, conventionally, many distortion correction techniques for obtaining as natural an image as possible while correcting the distortion have been proposed.

  For example, Patent Document 1 discloses a technique for obtaining a natural image with little sense of incongruity by projecting a distorted image onto a predetermined projection surface, and performing conversion according to the distance from the origin on the projected image. Is disclosed.

  In Patent Document 2, the pixel position on the virtual object plane is projected on the correction plane, and the pixel position projected on the correction plane is further reprojected on the display plane. A technology for preventing enlargement and obtaining a natural image is disclosed.

  Patent Document 3 discloses a technique for obtaining an easy-to-see image by partially changing the enlargement ratio of the image. In Patent Document 3, for example, an easy-to-see image is obtained by increasing the enlargement ratio of an image (such as a person) at the center of the image that is often displayed small when a wide-angle lens is used.

JP 2008-52589 A JP 2008-31890 A Japanese Patent No. 4279613

  By the way, all of the techniques described in Patent Documents 1-3 are considered to have a certain effect in terms of obtaining an image having a natural and uncomfortable feeling from a distorted image. However, it is desired to obtain a display image that is even more natural and less uncomfortable.

  The present invention has been made as a result of a thorough examination of what kind of images look natural.

  The present invention obtains a display image that is more natural and less uncomfortable (easy to see) than a conventional image from a distorted image captured by using a wide-angle lens by adopting a novel configuration that has never existed before. An object is to provide a camera, an image conversion apparatus, and an image conversion method.

  One aspect of the camera of the present invention includes a lens that forms an image of a subject, an image pickup device that converts an optical image formed by the lens into an electric signal, an image pickup unit that obtains a picked-up image, and a spatial model A spatial model storage unit that stores the image, a projection conversion unit that maps the captured image so as to project the captured image onto the spatial model, and a plane conversion unit that obtains a display image using the image mapped to the spatial model. It has.

  One aspect of the image conversion apparatus of the present invention is a spatial model storage unit that stores a spatial model of an imaging space, a projection conversion unit that maps so as to project a captured image onto the spatial model, and is mapped to the spatial model. And a plane conversion unit that obtains a display image using the obtained image.

  According to the present invention, it is possible to obtain a display image that is more natural and less uncomfortable than a conventional image, from a distorted image captured using a wide-angle lens.

It is a figure which shows the mode of the conventional distortion correction, FIG. 1A is a figure which shows the image before distortion correction, FIG. 1B is a figure which shows the image after distortion correction 1 is a block diagram illustrating a configuration of a camera according to a first embodiment. FIG. 3A is a diagram for explaining spherical back projection, FIG. 3A is a diagram showing orthographic projection, FIG. 3B is a diagram showing equidistant projection, and FIG. 3C is a diagram showing stereoscopic projection. FIG. 4A is a diagram showing an example of a spatial model, and FIG. 4B is a diagram for explaining the projection onto the spatial model. FIG. 5A is a diagram for explaining a process of forming a two-dimensional display image by developing a three-dimensional model on a plane, FIG. 5A is a diagram showing an example of developing the spatial model, and FIG. 5B is a diagram showing another example of developing the spatial model FIG. 5C is a diagram showing a two-dimensional display image formed using the developed spatial model. 6A is a diagram for explaining viewpoint conversion, and FIG. 6B is a diagram showing an output image obtained by performing viewpoint conversion. The figure which shows the example of the two-dimensional display image obtained using each scene (space model) FIG. 8A is a diagram illustrating an example of a space model for an in-vehicle camera, and FIG. 8B is a diagram illustrating an example of a two-dimensional display image obtained using the space model. 9A is a diagram illustrating an example of a spatial model for a surveillance camera, and FIG. 9B is a diagram illustrating an example of a two-dimensional display image obtained using the spatial model. FIG. 10A is a diagram illustrating an example of performing zoom processing by deforming a spatial model, FIG. 10A is a diagram illustrating a spatial model and a spherical image, FIG. 10B is a diagram in which a projection image is developed on a two-dimensional plane, and FIG. Figure showing a display image 11A is a diagram illustrating an example of forming an all-around image, FIG. 11A is a diagram illustrating a spatial model and a spherical image, FIG. 11B is a diagram in which a projection image is developed on a two-dimensional plane, and FIG. 11C is a diagram illustrating a two-dimensional display image. FIG. 12A is a diagram illustrating an example of performing zoom processing on an all-around image, FIG. 12A is a diagram illustrating a spatial model and a spherical image, FIG. 12B is a diagram in which a projection image is developed on a two-dimensional plane, and FIG. Illustration 1 is a block diagram illustrating a configuration example of a camera according to an embodiment 1 is a block diagram illustrating a configuration example of a camera according to an embodiment FIG. 15A is a diagram showing an example of shifting a spatial model, FIG. 15A is a diagram showing a spatial model and a spherical image, FIG. 15B is a diagram showing a projection image developed on a two-dimensional plane, and FIG. 15C is a diagram showing a two-dimensional display image. FIG. 16A and FIG. 16B are diagrams showing examples of spherical images, FIG. 16C is a diagram showing a spatial model in the form of a two-dimensional display, and FIG. The figure which shows the final two-dimensional display image obtained by projecting the spherical image onto the model

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[1] Configuration FIG. 2 shows the configuration of the camera in the present embodiment. The camera 100 has an imaging unit 101. The imaging unit 101 includes a wide-angle lens 102 such as a fisheye lens, and forms an object image on the imaging element 103 by the wide-angle lens 102. The image sensor 103 is composed of a CCD or the like, and obtains a captured image signal by converting the formed optical image into an electric signal. The camera processing unit 104 performs γ processing, matrix processing, and the like on the captured image signal output from the image sensor 103. The camera processing unit 104 outputs the processed captured image signal to the spherical back projection conversion unit 105.

  The spherical backprojection conversion unit 105 inputs a captured image signal whose pixel coordinates are two-dimensional from the camera processing unit 104 and inputs a lens model from the lens model selection unit 106. The lens model is information indicating the focal length of the wide-angle lens 102 used at the time of imaging, the lens projection conversion characteristics (such as fθ, 2ftanθ / 2, and fsinθ). This lens model is changed when the wide-angle lens 102 is replaced or when the focal length or the like is changed by a user operation. The lens model selection unit 106 changes the lens model (information such as projection conversion) when the wide-angle lens 102 itself is replaced. The lens model selection unit 106 changes the lens model (focal length, projection conversion, etc.) in accordance with a user operation signal input from the operation input unit 107 via the setting unit 108.

  The operation input unit 107 includes a touch panel, operation buttons, and the like, and is a part where the user operates the camera.

  The spherical backprojection converter 105 virtually projects the two-dimensional image projected on the image sensor 103 onto the hemisphere while preserving the incident angle from the subject on the entire surface of the wide-angle lens 102 to the lens. When the wide-angle lens 102 is a lens having an angle of view of 180 degrees such as a fisheye lens, the spherical backprojection conversion unit 105 obtains a hemispherical image. This conversion is performed by calculation using a lens model.

  In the first place, the phenomenon in which the light that has passed through the wide-angle lens 102 is projected onto the two-dimensional image sensor 103 is a geometric phenomenon. Therefore, if the lens model of the wide-angle lens 102 used at the time of imaging is used, From which position of the wide-angle lens 102 the image formed at the position can be easily obtained by calculation.

  For example, as shown in FIG. 3, the image height r formed on the image sensor 103 is projected by a lens such as orthographic projection (FIG. 3A), equidistant projection (FIG. 3B), or stereoscopic projection (FIG. 3C). A point on the sphere can be converted by calculation using a conversion formula. In FIG. 3, f indicates the focal length. For example, when orthographic projection (FIG. 3A) is used, since r and f are already given out of r = f × sin θ, a point on the spherical surface can be easily obtained by calculating θ. Note that the calculation for back projecting an image once formed on a two-dimensional coordinate onto a spherical surface is also described in Patent Document 2, for example. The captured image projected on the spherical surface by the spherical back projection conversion unit 105 (hereinafter referred to as a spherical image) is sent to the projection conversion unit 109.

  The projection conversion unit 109 receives a space model from the space model selection unit 110 and projects a captured image on the space model.

  The spatial model selection unit 110 selects the spatial model set by the setting unit 108 from among a plurality of spatial models stored in the spatial model storage unit 111 and outputs the selected spatial model to the projection conversion unit 109. The spatial model selected by the spatial model selection unit 110 is a spatial model selected by the user from the operation input unit 107 and is set by the setting unit 108. When the spatial model is set by the setting unit 108, for example, an imaging signal obtained by the imaging unit 101 may be input to the setting unit 108, and the setting unit 108 may set a spatial model most suitable for the imaging signal.

  The space model is a so-called template that is a three-dimensional space model that approximates an imaging space captured by the camera 100. For example, when the imaging space is indoors, the space model approximates a three-dimensional position such as a floor, a ceiling, a wall, and a table existing in the imaging space. When the imaging space is in front of the vehicle, the space model approximates a three-dimensional position of the imaging space such as a road, a building, and a surface of interest. When the imaging space is a monitoring space, the space model approximates the front surface, the side surface, the ceiling (upper surface), and the floor (lower surface). In the space model storage unit 111, a space model of an assumed imaging space is stored in advance.

  FIG. 4A shows an example of a space model, and FIG. 4B shows a state of projection onto the space model performed by the projection conversion unit 109.

  The example of the space model shown in FIG. 4A is a space model when the imaging space is indoors. The upper and lower half disks in the half cylinder represent the ceiling, the floor, or the desk, respectively, and the half cylinder is the front and side surfaces. (For example, a wall).

  As shown in FIG. 4B, the projection conversion unit 109 arranges the spherical image obtained by the spherical back projection conversion unit 105 in the spatial model, and projects the spherical image onto the spatial model with the center of the hemisphere as the projection point ( Mapping). The projection conversion unit 109 performs projection on the space model and outputs the result to the plane conversion unit 112.

  The plane conversion unit 112 obtains a two-dimensional display image based on the image mapped to the space model. In the present embodiment, the following two methods are presented as methods for obtaining a two-dimensional display image from an image mapped to a spatial model.

  Method (i): An image of a three-dimensional surface mapped to a spatial model is developed in a plane, and in a plane-developed image, a portion where no image exists in the two-dimensional display image is deformed by enlarging / reducing the plane-developed image. A method for obtaining a two-dimensional display image.

  In this method, the plane conversion unit 112 develops a three-dimensional space model on a two-dimensional plane as shown in FIGS.

  Thus, the plane conversion unit 112 forms a two-dimensional display image as shown in FIG. 5C from the image projected onto the three-dimensional space model. At this time, as is apparent from the drawing, the developed image has a gap between the upper and lower surfaces and the side surface (that is, a portion where no image is present with respect to the display image). Thus, the two-dimensional display image as shown in FIG. 5C is obtained by extending the upper and lower surfaces and the side surfaces so as to fill this gap.

  Method (ii): First, as shown in FIG. 6A, the plane conversion unit 112, when viewing the image mapped to the spatial model from a certain viewpoint, the horizontal viewing angle θ to each pixel position, The gaze angle φ in the vertical direction is calculated. Next, as shown in FIG. 6B, the plane conversion unit 112 performs two-dimensional conversion from the image mapped to the spatial model by performing coordinate conversion so that θ and φ correspond to the x coordinate and the y coordinate of the two-dimensional display. Get a display image.

[2] Example of spatial model FIG. 7 shows an example of a two-dimensional display image obtained by the camera 100.

  The two-dimensional display image on the upper right side is a two-dimensional display image for a video conference camera. This image can be obtained using a spatial model for a video conference camera. That is, the right-side upper two-dimensional display image is obtained by projecting the hemispherical backprojection image onto the space model for the video conference image by the projection conversion unit 109 and converting it into a two-dimensional display image by the plane conversion unit 112. In the example shown in the figure, the two-dimensional display image is divided into a desk part 11, a wall part (may be called a person part or a target part) 12, and a ceiling part 13 based on a spatial model.

  The two-dimensional display image in the right middle stage is a two-dimensional display image for an in-vehicle camera. This image can be obtained using a space model for an in-vehicle camera. In other words, the right-side middle two-dimensional display image is obtained by projecting the hemispherical backprojection image onto the space model for the vehicle-mounted camera by the projection conversion unit 109 and converting it into a two-dimensional display image by the plane conversion unit 112. In the example of the figure, the two-dimensional display image is divided into a road portion 21, a building portion 22, and an empty portion 23 based on a space model.

  The two-dimensional display image on the lower right side is a two-dimensional display image for a surveillance camera. This image can be obtained using a spatial model for a surveillance camera. That is, the lower right two-dimensional display image is obtained by projecting the hemispherical backprojection image onto the space model for the surveillance camera by the projection conversion unit 109 and converting it into a two-dimensional display image by the plane conversion unit 112. In the example shown in the figure, the two-dimensional display image is divided into a front portion (may be referred to as a portion of interest) 31, a side portion 32, a lower portion 33, and an upper portion 34 based on a spatial model.

  The spatial model for video conference for forming the two-dimensional display image on the upper right side of FIG. 7 is the same as the spatial model shown in FIG. 4 is different from FIG. 4 in that the lower surface of the space model is a floor in FIG. 4, but the lower surface of the space model for video conference is a desk.

  FIG. 8 shows an example of a space model for an in-vehicle camera (FIG. 8A) and an example of a two-dimensional display image obtained using the space model (FIG. 8B). The space model in FIG. 8A has a lower surface m1 corresponding to a road, an upper surface m2 corresponding to the sky, and a side surface m3 corresponding to a building. In addition to this, the spatial model of FIG. 8A has a surface of interest m4. A hemispherical backprojection image as shown in FIG. 4B is arranged on the front side of the attention surface m4, that is, the viewpoint P0 side. That is, the backprojected image is arranged between the attention surface m4 and the viewpoint P0. The lower surface m1 and the upper surface m2 can be adjusted in the vertical direction indicated by arrows. The side surface m3 can be adjusted in the left-right direction indicated by the arrow. The attention surface m4 can be adjusted in a direction approaching the viewpoint P0 and a direction away from the viewpoint P0. The viewpoint P0 can be adjusted in the vertical and horizontal directions. These adjustments may be performed by the projection conversion unit 109, for example.

  In the two-dimensional display image of FIG. 8B obtained by using the spatial model of FIG. 8A, the region indicated by reference numeral 25 is substantially the most noticeable region. The size of the region 25 can be adjusted by adjusting the distance of the attention surface m4 from the viewpoint P0. Specifically, the area of the attention area (which may be referred to as a front area) 25 increases as the attention surface m4 is arranged closer to the viewpoint P0. Note that as the viewpoint P0 is moved upward, an overhead image can be obtained. When zooming in, the viewpoint is brought closer to the space model.

  FIG. 9 shows an example of a spatial model for a surveillance camera (FIG. 9A) and an example of a two-dimensional display image obtained using the spatial model (FIG. 9B). 9A has a lower surface n1 corresponding to a floor or the ground, an upper surface n2 corresponding to a ceiling or the sky, and a side surface n3 corresponding to a wall or the like. In addition to this, the spatial model of FIG. 9A has an attention surface (which may be referred to as a front surface) n4. A hemispherical backprojection image as shown in FIG. 4B is arranged on the front side of the attention surface n4, that is, the viewpoint P0 side. That is, the backprojected image is arranged between the attention surface n4 and the viewpoint P0. The lower surface n1 and the upper surface n2 can be adjusted in the vertical direction indicated by arrows. The side surface n3 can be adjusted in the left-right direction indicated by the arrow. The attention surface n4 can be adjusted in a direction approaching the viewpoint P0 and a direction away from the viewpoint P0. The viewpoint P0 can be adjusted in the vertical and horizontal directions. These adjustments are performed by the projection conversion unit 109. By moving the viewpoint, it is possible to perform conversion by enlarging the part that approaches the space model and reducing the part that moves away.

  In the two-dimensional display image of FIG. 9B obtained using the spatial model of FIG. 9A, the size of the front portion 31 can be adjusted by adjusting the distance of the surface of interest n4 from the viewpoint P0. Specifically, the area of the front portion 31 increases as the attention surface n4 is arranged closer to the viewpoint P0.

[3] Zoom processing and omnidirectional image Here, the zoom processing and the omnidirectional image can be performed by adjusting the surface constituting the spatial model or adjusting the position of the spherical image arranged in the spatial model. It is also possible to perform a forming process or the like. These processes will be described with reference to FIGS.

  FIG. 10 shows an example in which zoom processing is performed by deforming a spatial model. Here, the method described in FIG. 5 for developing a three-dimensional space model on a plane is used. This is different from the method of moving the viewpoint described in FIG. In FIG. 10A, the farther the projection plane is from the projection point, the larger the projected image on the projection plane, and the zoom processing is realized. In FIG. 10A, when the lower surface of the space model is moved in the upward direction indicated by the arrow, the boundary between the front surface and the lower surface moves, and the lower surface is slightly enlarged. This operation is performed not for the purpose of zooming but to change the boundary between the lower surface and the front surface. FIG. 10B shows a diagram in which an image obtained using the spatial model of FIG. 10A is developed on a two-dimensional plane, and FIG. 10C shows a two-dimensional display image obtained from the developed image of FIG. 10B.

  Thus, by moving the space model far from the projection point, the projected image is enlarged, and partial zoom can be realized on the two-dimensional image to be displayed. In addition, by changing the boundary where the planes of the space model intersect, alignment with the boundary part of the live-action image (the boundary between the floor and the wall) is achieved, and an image that is easy to see the two-dimensional image to be displayed (the image at the boundary of the live-action subject) However, since the live-action video has also changed, a person does not feel uncomfortable with the deformation of the image.

  FIG. 11 shows an example of forming an all-around image (360-degree field of view). First, as shown in FIG. 11A, a spherical image is arranged at the center of a cylindrical space model. Note that the spherical image is acquired by an imaging unit with two fisheye lenses (180-degree field of view) facing forward and backward. Then, a projection point is set at the center of the spherical surface, and the spherical image is projected onto the cylinder. Next, as shown in FIG. 11B, the images projected on the space models 11A1, 11A2, and 11A3 of FIG. 11A are developed on a two-dimensional plane. Next, as shown in FIG. 11C, deformation (enlargement and reduction) is performed to obtain a two-dimensional display image from the developed image of FIG. 11B.

  FIG. 12 shows an example in which zoom processing is performed on the entire surrounding image. As shown in FIG. 12A, the spherical image is arranged at a position shifted from the center of the cylindrical space model 12A1, 12A2, 12A3, a projection point is determined at the center of the spherical surface, and the spherical image is projected onto the column of the spatial model. . At this time, a projection image with a larger zoom is obtained as the position is farther from the projection point. In the example of FIG. 12A, the image projected on the left side surface is zoomed. In the example of FIG. 12A, zoom processing is realized based on the positional relationship between the space model and the spherical image, but similar zoom processing can also be realized by changing the position of the projection point. Next, as shown in FIG. 12B, the image obtained using the spatial model of FIG. 12A is expanded and deformed into a two-dimensional plane. Next, as shown in FIG. 12C, a two-dimensional display image is obtained from the developed image of FIG. 12B.

[4] Adjustment of Boundary Here, in the image conversion according to the present embodiment, it is important to match the boundary of the object in the space model with the boundary of the object in the image projected onto the space model. In the configuration shown in FIG. 2, the process of matching the boundaries may be performed by shifting the space model and / or the spherical image via the operation input unit 107 while the user looks at the image displayed on the display. .

  Further, as shown in FIGS. 13 and 14, boundary detection units 201 and 301 for detecting the boundary of the spherical image are provided, and the space model or the spherical image is shifted based on the detection result, so that the surface in the space model And the boundary of the object in the image projected on the space model are matched.

  The configuration of FIG. 13 will be described. A camera 200 in FIG. 13, in which parts corresponding to those in FIG. 2 are assigned the same reference numerals, has a boundary detection unit 201 and a spatial model adjustment unit 202. The boundary detection unit 201 detects the boundary of the spherical image output from the spherical back projection conversion unit 105, that is, the edge of the image. This edge detection can be easily realized by a known technique. The edge of the image is, for example, a boundary between a desk and a wall. The space model adjustment unit 202 deforms the space model based on the detection result by the boundary detection unit 201. Specifically, the space model adjustment unit 202, when projecting by the projection conversion unit 109, matches the boundary of the object in the image projected on the space model with the boundary of the plane intersection in the space model. Deform the model.

  The configuration of FIG. 14 will be described. A camera 300 shown in FIG. 14, in which parts corresponding to those in FIG. The boundary detection unit 301 detects the boundary of the spherical image output from the spherical backprojection conversion unit 105, that is, the edge of the image. The spherical image adjustment unit 302 shifts the spherical image based on the detection result by the boundary detection unit 301. Specifically, the spherical image adjustment unit 302 is configured so that the object boundary in the image projected onto the spatial model coincides with the boundary of the plane intersection in the spatial model when the projection conversion unit 109 projects. Shift the image.

  FIG. 15 shows an example of deforming the space model. In the space model in FIG. 15A, the tip of the lower surface of the space model is moved in the vertical direction indicated by the arrow so as to match the object boundary of the spherical image. FIG. 15B shows a diagram in which an image obtained using the spatial model in FIG. 15A is developed on a two-dimensional plane, and FIG. 15C shows a two-dimensional display image obtained from the developed image in FIG. 15B.

  FIG. 16 shows a specific example in which a spherical image is projected onto a space model.

  16A and 16B show examples of spherical images. FIG. 16A shows an example of an image in which a desk is shown large, and FIG. 16B shows an example of an image in which a desk is shown normally.

  FIG. 16C shows an image of the space model in a form displayed on the two-dimensional display. Shift so that the boundary between the person area and the desk area matches the boundary of the image in FIG. 16A or 16B (the same as described in FIG. 15C). Note that, as described above, the boundary of the spherical backprojection image may be shifted, and the image may be generated with the spatial model fixed. When the boundary of the spherical backprojection image is shifted, the position of the boundary between the desk and the person is the same position on the display screen of the display. When the intersection position of the plane of the space model is deformed, the position of the boundary between the desk and the person moves up and down on the display screen of the display.

  FIG. 16D shows a final two-dimensional display image obtained by projecting a spherical image onto a space model.

  A captured image obtained using a conventional wide-angle lens is greatly distorted in the peripheral portion due to the characteristics of the wide-angle lens. In addition, when distortion correction is performed in consideration of the lens projection method, the plane facing the camera is corrected correctly, but those that are not facing the camera and general objects that are not facing the plane are corrected to an extremely large level or vice versa. There is a place where the correction is extremely small. In contrast, in the present embodiment, by using a spatial model, a two-dimensional display image can be obtained with the same shape and size as when an actual object is observed. That is, in the example of FIG. 16D, the person area and the desk area are displayed in a shape that is almost similar to the shape actually observed. A greatly distorted portion occurs at the boundary of the image (the boundary between the desk region and the person region), but it is close to an image observed by a human and can be displayed as an image with little discomfort.

[5] Effect As described above, according to the configuration of the present embodiment, the captured image obtained by using the wide-angle lens 102 is mapped so as to be projected onto the spatial model, and this mapping is a plane of the spatial model. The two-dimensional display image is obtained based on the mapped image by matching the boundary (intersection) of the object and the boundary of the object shown in the image. From the distorted image, it is possible to obtain a display image that is more natural and easier to see than before (displayed close to the shape observed by humans, and the distorted portion does not feel distorted).

  That is, using the space model, the boundary of the surface of the space model is deformed so as to be the boundary of the object of the real subject (image shift or space model deformation), and this space model projection image is converted into a two-dimensional plane. Images projected on each surface of the spatial model by this method generate distortion, but the generated distortion occurs at the boundary of the actual subject. When observed on the display, the greatly distorted part is the boundary of the image (desk area and person Occurs at the boundary of the region). Thereby, it becomes a shape close | similar to the image which a human observed, and it can display as an image with little discomfort.

  In addition, when the method of the embodiment for developing into a two-dimensional image is used, distortion components are mainly processed by extending the image of each surface when forming a two-dimensional display image (FIGS. 5A, 5B, and 10B). 11B, FIG. 12B, and FIG. 15B). That is, the distortion component is concentrated on the boundary of the object. However, even if the image is distorted to some extent at the boundary of the object, the unnatural appearance is small. As a result, the two-dimensional display image obtained by the configuration of the present embodiment is a display image that is more natural and easier to see than in the past.

  In addition, by moving the viewpoint, it is possible to change the ratio of the areas shown on the front and bottom display, and to compress the bottom (desk area) and increase the front (person area). At the same time, it is possible to realize an easy-to-see display by realizing a conversion that does not feel uncomfortable.

  In the above-described embodiment, the case where the spherical back projection conversion unit 105, the projection conversion unit 109, the plane conversion unit 112, the space model storage unit 111, and the space model selection unit 110 are provided in the camera has been described. May be provided as an image conversion device separately from the camera.

  Further, the spherical backprojection conversion unit 105 is not necessarily required. If a spatial model that takes into account the optical relationship between the wide-angle lens 102 and the image sensor 103 is prepared, the captured image obtained by the image capturing unit 101 is input to the projection conversion unit 109 without being subjected to back projection conversion on the spherical surface. You may let them.

  INDUSTRIAL APPLICABILITY The present invention can obtain a natural and less discomfort display image from a distorted image captured using a wide-angle lens such as a fisheye lens, and is suitable for a surveillance camera, an in-vehicle camera, a video conference camera, and the like.

100, 200, 300 Camera 101 Imaging unit 102 Wide angle lens 105 Spherical backprojection conversion unit 109 Projection conversion unit 110 Spatial model selection unit 111 Spatial model storage unit 112 Planar conversion unit 201, 301 Boundary detection unit 202 Spatial model adjustment unit 302 Spherical image Adjustment section

Claims (8)

An imaging unit that has a lens that forms an image of an object and an image sensor that converts an optical image formed by the lens into an electric signal;
A spatial model storage for storing the spatial model;
A projection conversion unit that maps the captured image so as to project the captured image onto the spatial model;
A plane conversion unit that obtains a display image using an image mapped to the spatial model;
A camera comprising: The plane conversion unit is
The image mapped to the spatial model is developed in a plane, and in the image developed in a plane, a portion where no image exists with respect to the display image is obtained by enlarging and / or reducing the deformed image on the plane, Get a display image,
The camera according to claim 1. The plane conversion unit is
When the image mapped to the spatial model is viewed from a certain viewpoint, the horizontal line-of-sight angle to each pixel position is θ, and the vertical line-of-sight angle is φ, and θ and φ are x of the two-dimensional display. Obtaining a two-dimensional display image from the image mapped to the spatial model by performing coordinate transformation so as to correspond to the coordinate and the y-coordinate;
The camera according to claim 1. A boundary detection unit for detecting a boundary of the captured image;
An adjustment unit that adjusts the boundary of an object in the captured image to the boundary of intersection of planes in the spatial model by shifting the captured image and / or the spatial model based on a detection result by the boundary detection unit. Further comprising
The camera according to any one of claims 1 to 3. The spatial model storage unit stores a plurality of spatial models,
The camera further includes a space model selection unit that selects one space model from the plurality of space models.
The camera according to any one of claims 1 to 4. A back projection conversion unit that virtually converts the captured image into an image on the lens;
The projection conversion unit maps the captured image on the lens virtually obtained by the back projection conversion unit so as to project the image on the spatial model.
The camera according to any one of claims 1 to 5. A spatial model storage unit for storing a spatial model of the imaging space;
A projection conversion unit that maps the captured image so as to project the captured image onto the spatial model;
A plane conversion unit that obtains a display image using an image mapped to the spatial model;
An image conversion apparatus comprising: Mapping the captured image to project onto a spatial model of the imaging space;
Obtaining a display image using an image mapped to the spatial model;
An image conversion method including: