HK1209541B - Image transformation and multi-view output systems and methods - Google Patents

Description

Image conversion and multi-view output system and method

Background Art

Digital imaging systems typically include one or more lenses and a digital image sensor. The digital image sensor captures light from the object or scene being imaged through the lens and converts the light into an electronic signal. The electronic signal is digitized and stored in semiconductor memory as digital image data. Such digital imaging systems are used in a variety of consumer products, industrial, and scientific applications, including mobile phones, digital still and video cameras, webcams, and other devices to produce still images and/or videos.

Most image sensors today are composed of two-dimensional pixel arrays, either complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors. Today's digital image sensors can include millions of pixels to provide high-resolution images.

The quality of digital images, including still images and video images produced by digital imaging systems, can depend on a variety of factors. In digital imaging systems with wide-angle photographic lenses (e.g., fisheye lenses), lens distortion can significantly affect the quality of digital images. Lens distortion causes straight lines or objects in a scene to appear curved in the image. The most common form of distortion is radial symmetry, which arises from the symmetry of the lens. Radial distortion can be categorized into one of two main types: barrel distortion and pincushion distortion. Barrel distortion is common in images captured by wide-angle lenses, while pincushion distortion is typically present in images captured by zoom or telephoto lenses.

In barrel distortion, the image's magnification decreases with distance from the optical axis. The effect is that the image appears projected onto the circumference of a sphere or barrel. In pincushion distortion, the image's magnification increases with distance from the optical axis. The visual effect is that lines not passing through the center of the image bow inward toward the center, like a pincushion. Compound distortion, which is a combination of barrel and pincushion distortions, begins as barrel distortion near the center of the image and gradually becomes pincushion distortion toward the edges of the image.

In a typical digital imaging system, errors caused by lens distortion can be corrected by performing signal processing on an image signal processor (ISP) formed on a semiconductor chip die. The ISP receives digital image data from a digital image sensor, which is also commonly formed on a semiconductor chip die. To initiate the correction process for the distorted digital image data, a large amount of buffer memory is used to store hundreds of lines of distorted image data. The required memory must be large enough and cannot be included as part of the chip that serves as the ISP. Therefore, additional memory chips are necessary, such as dynamic random access memory (DRAM) devices. Such additional devices affect the size and cost of the digital imaging system. In addition, such additional devices cause delays when accessing the memory and complicate the memory access bandwidth issue.

Summary of the Invention

In one embodiment, an image conversion and multi-view output method generates output views from an original image. The original image is received, and the coordinates of each pixel in the output view are de-mapped to positions in the original image using a coordinate mapping method. The coordinate mapping method corrects at least one of (a) perspective and (b) distortion in the original image. The intensity of each pixel in the output view is determined based on the corresponding mapped pixel position information in the original image.

In another embodiment, an image conversion and multi-view output system generates output views from an original image. The system includes a lookup table stored in non-volatile memory that maps output view coordinates to the original image; and an inverse mapper for generating the output view from the original image using the lookup table. The coordinate mapping corrects at least one of (a) perspective and (b) distortion in the original image.

In another embodiment, a method is provided for generating a coordinate mapping stored in a lookup table and used by an image transformation and multi-view output system, the system comprising an imaging lens; an image sensor for capturing an original image from the lens; and an output device for displaying a corrected image. Parameters of the image transformation and multi-view output system are received, including a distortion curve and a distortion center of the imaging lens. An inverse perspective correction is determined based on the angular orientation of the optical axis of the imaging lens relative to a principal ray incident on the lens from an object in the lens' field of view. An inverse distortion correction is determined based on one or both of the distortion curve and the distortion center. A coordinate mapping is generated based on the inverse perspective correction and the inverse distortion correction, wherein the coordinate mapping includes an inverse transformation for use by the transformation and multi-view output system to generate output views from the original image.

BRIEF DESCRIPTION OF THE DRAWINGS

1 shows an exemplary image conversion and multi-view output system configured with a vehicle having a rearview camera, a display, and an optional multi-view selection switch, in one embodiment.

FIG. 2 shows a schematic diagram of the image conversion and multi-view output system in the embodiment of FIG. 1 in more detail.

3 shows an exemplary line grid imaged by the camera of FIG. 1 and processed by the image conversion and multi-view output system of FIG. 1 .

FIG. 4 shows an imaging configuration diagram in which the camera of FIG. 1 is used to capture raw image data of an object.

FIG. 5 shows an original image of the object in FIG. 3 captured by the camera in FIG. 4 .

FIG. 6 shows output views generated from the original image of FIG. 5 by the image conversion and multi-view output system of FIG. 1 .

FIG. 7 shows an exemplary image conversion and multi-view output method in one embodiment.

FIG. 8 shows an exemplary original image and three exemplary output views produced by the image conversion and multi-view output system of FIG. 1 .

FIG. 9 shows a distorted original image and an output view produced by the system of FIG. 1 performing the perspective and distortion correction method of FIG. 7 .

FIG. 10 is a schematic diagram showing a coordinate mapping generator used to generate the lookup table of FIG. 2 according to one embodiment.

FIG. 11 shows a method for generating the lookup table of FIG. 2 in one embodiment.

DETAILED DESCRIPTION

In lens-based video imaging systems that employ wide viewing angles, such as those used in automotive driver-assistance cameras, delays between frames can reduce the system's usefulness. This delay reduces the viewer's ability to appropriately respond to the images captured by the system. The resulting choppy or sputtering video stream prevents potential users, such as those driving, from viewing the complete image. Therefore, the ability to correct image distortion while displaying the image at a "smooth" frame rate, as described in conjunction with Figures 1 through 11 and below, is a valuable feature of these video technologies.

Distortion removal methods involving inverse mapping have been used to assist video technology. Inverse mapping maps pixels in the output image back to their positions in the original distorted image. In a forward mapping algorithm, each pixel in the distorted image is mapped to a pixel in the corrected image. However, such algorithms are computationally inefficient because many pixels in the distorted image are not mapped to the corrected image. Inverse mapping methods are more efficient and faster, requiring fewer computational resources than forward mapping, as they only map pixels in the original image.

One approach to speeding up distortion removal in imaging systems is to pre-calculate the inverse mapping and store the results in a lookup table. For example, an imaging system including a CMOS image sensor can perform the mapping algorithm (forward or inverse) on-chip for each captured image and then display the transformed image. Replacing the algorithm with a lookup table containing the algorithm's results can reduce the time required for image distortion correction and can be applied to other image transformations.

Fast image conversion is important for producing live, distortion-free video streams from distorted images, such as those captured by fisheye cameras. For certain imaging applications, as described below, a "multi-view" feature is advantageous because it allows the user to select and view one or more converted regions within an original image. For example, automotive rearview cameras and surveillance video systems, which use wide-angle cameras with significant distortion, would benefit from such selection and conversion. Providing multiple views from captured video using computationally expensive distortion removal algorithms limits the video frame rate, resulting in choppy video output. This choppy output degrades the quality of the video and hinders the user's ability to quickly react to the information presented. Prior art image distortion correction systems lack the combination of multiple convertible images and dynamically (or "real-time") reconfigurable multi-view output.

The present invention relates to methods and related systems for converting raw digital images to remove image artifacts, such as distortion and perspective errors associated with images captured by wide-angle lenses, such as fisheye lenses. An exemplary method for image correction and selecting sub-view regions for a multi-view video stream enables real-time changes in the selection of multiple display views.

FIG1 shows an exemplary image conversion and multi-view output system 100 for an automobile 102 configured with a rearview camera 110, a display 120, and an optional multi-view selection switch 122. Camera 110 has an imaging lens 112 and an image sensor 116. Imaging lens 112 is a wide-angle lens, such as a fisheye lens. Image sensor 116 may be implemented, for example, in CMOS, but may also be implemented in other technologies without departing from the scope of the present invention. System 100 may be implemented, for example, in a single integrated circuit, which may be configured in either camera 110 or display 120.

In one example of operation, imaging lens 112 images an object in its field of view 113 onto image sensor 116, which generates raw image data 117 and sends it to image conversion and multi-view output system 100. Image conversion and multi-view output system 100 converts at least a portion of raw image data 117 into output view data 121, which is displayed as output view 123 on display 120. If included, an optional multi-view selection switch 122 can be used to select an alternate output view from the views generated by image conversion and multi-view output system 100 and display the corresponding converted version of raw image data 117 on display 120. Image conversion and multi-view output system 100 can also simultaneously generate and display multiple views in output view 123 from raw image data 117 on display 120.

Digital image data may be captured by an image sensor having a two-dimensional array of pixels. The image sensor 116 has, for example, a two-dimensional array of pixels, each pixel having a position and an intensity. A pixel position is the two-dimensional coordinate location of a pixel in the pixel array. For example, the pixel array in a VGA image sensor has dimensions of 640×480, and its pixel positions are numbered using a zero-based numbering range from (0,0) to (639,479). The raw image data captured by the sensor comprises a two-dimensional array of pixel intensity values. With few exceptions, locations in the raw image data determined by a mapping will have non-integer coordinates corresponding to positions between pixels. Pixel intensity is the brightness of a pixel when the digital image data is displayed, for example, as an integer between 0 and 255.

Display device 120 displays output view data 121, which is represented as a two-dimensional array of pixel values, each having a position and an intensity value. Thus, when referring to digital image data (e.g., raw image data 117 or output view data 121), this specification constructs reference pixels that can be represented as image data. Each pixel that constitutes such data has a pixel position and a pixel intensity value.

FIG2 is a schematic diagram illustrating further exemplary details of the image conversion and multi-view output system 100 shown in FIG1 . The image conversion and multi-view output system 100 includes a memory 202, an inverse mapper 206, and an optional multi-view selection module 208. The memory 202 is configured to store received raw image data 117 (e.g., from the image sensor 116 of the camera 110), a lookup table 204, and generated output view data 121. The memory 202 is implemented at least in part as non-volatile memory (e.g., for storing the lookup table 204) and at least in part as volatile memory (e.g., for temporarily storing the raw image data 117 and the output view data 121). The lookup table 204 includes at least one coordinate map 205 that maps the output view data 121 to at least a portion of the raw image data 117, thereby correcting the perspective of the raw image data 117 based on the output view data 121. The coordinate map 205 may also correct for distortion introduced by the imaging lens 112 , thereby improving the quality and clarity of the output view 123 generated from the raw image data 117 .

The inverse mapper 206 includes a coordinate mapping module 212 and an intensity setting module 216. The coordinate mapping module 212 maps each pixel of the output view data 121 to a corresponding portion of the original image data 117 based on the lookup table 204. The intensity setting module 216 retrieves pixel intensity values from the mapped portion of the original image data 117 and sets the pixel intensity values in the output view data 121 based on these intensity values. In one embodiment, implementing the inverse mapper 206 in hardware allows for a high frame rate to be achieved when generating the output view data 121 from the original image data 117. That is, perspective and distortion correction of the original image data 117 is performed using the lookup table 204 to generate the output view data 121 at a sufficiently high speed, thereby ensuring that updates to the output view 123 are not delayed and that the perceived motion remains smooth, making the user unaware of any transitions.

In one example of operation, for each pixel in the output view data 121, the coordinate mapping module 212 determines the location in the original image data 117 using the coordinate mapping 205 in the lookup table 204. The intensity setting module 216 determines the intensity value for the pixel in the output view data 121 based on the information in the original image data 117 at the determined location. Because only pixels in the original image data 117 corresponding to pixels in the output view data 121 are processed, computational load is reduced and power is conserved. Pixels in the original image data 117 that are not mapped to the output view data 121 are not processed. The inverse mapper 206 is conveniently implemented in hardware because: (a) it directly maps pixels of the output view data 121 to the original image data 117, and (b) a single mapping operation using the lookup table 204 is used to simultaneously perform one or both of perspective correction and distortion correction.

To ensure that at least some of the image information from image sensor 116 is transferred to output view 123, each pixel position in the output view data is "backward" mapped to a position in the original image data 117. This mapping is considered an "inverse" mapping because the attributes of the output view data are mapped to the attributes of the input data.

In one embodiment, the lookup table 204 includes a plurality of coordinate mappings 205, each of which generates a different output view 123 from the raw image data 117. A user can select, using the multi-view selection switch 122, for example, a particular coordinate mapping 205 to generate a desired output view 123. Alternatively, the coordinate mappings 205 can be configured to generate a plurality of sub-views within the output view 123, each of which is mapped to a portion of the raw image data 117. As shown in the example below, the simultaneous mapping of portions of the raw image data 117 can be overlapped to form multiple sub-views within the output view 123.

In one use case, one of the multiple output views is automatically selected based on one or more sensed conditions relative to the vehicle. For example, for a vehicle equipped with ultrasonic proximity detection, the image conversion and multi-view output system 100 can be controlled to display one or more predefined views when an external object is detected in proximity. This allows the appropriate view of the external object to be automatically displayed to the driver of the vehicle.

Various types of image corrections can be performed in the lookup table 204. For example, the lookup table 204 can perform cylindrical viewing, which includes correction along one direction (e.g., vertical or horizontal) while ignoring loss of field of view in orthogonal directions. In another example, the lookup table 204 can perform rectilinear viewing, which includes distortion correction along two orthogonal axes but also a reduction in field of view along those axes, where any reduction in field of view is compensated for by generating multiple rectilinear view centers relative to different regions of the original image.

Coordinate mapping 205 may also include image processing operations, such as rotation operations and zoom operations (changing focal length).

The coordinate maps 205 are generated, for example, by the coordinate map generator 1000 ( FIG. 10 ), which may perform calibration of the imaging lens 112 and perspective and distortion correction in each coordinate map 205 , and then use the coordinate maps 205 in the lookup table 204 of the image conversion and multi-view output system 100 .

In one embodiment, the inverse mapper 206 uses a nearest neighbor algorithm to set the output view pixel intensity value to the original image pixel intensity of the pixel at coordinates (x _m , _yn ), where (x _m , _yn ) is the coordinate closest to the mapped original image coordinates (x _raw , y _raw ). In other embodiments, more accurate methods that approximate the intensity of the mapped original image coordinates may be used. For example, a multiple neighbor interpolation algorithm may be used, which includes the intensities of the three, four, or N pixels closest to the mapped original image coordinates (x _raw , y _raw ). These methods include, but are not limited to, interpolation algorithms such as bilinear interpolation and bicubic interpolation.

In one embodiment, the results of the interpolation algorithm are pre-calculated and stored in a coordinate map 205 in the lookup table 204. For example, each element in the coordinate map 205 may include an array identifying a pixel location corresponding to the original image and a weight value for each location. The inverse mapper 206 calculates a weighted average of the pixel intensities based on the identified pixel locations and the weight values stored in the coordinate map 205.

Figures 3 through 6 illustrate an example of the operation of the system 100 of Figures 1 and 2. Figures 3 through 6 are best viewed in conjunction with the following description.

FIG3 shows an exemplary wire grid 300 that is imaged by the camera 110 of FIG1 and processed by the image conversion and multi-view output system 100. To better illustrate the functionality of the image conversion and multi-view output system 100, a plane having an 8×8 square grid formed by the intersection of horizontal lines 301 and vertical lines 302 is used as the wire grid 300. The horizontal line 301(4) and the vertical line 302(4) intersect at an intersection 315, and the horizontal lines 301(2) and 302(4) intersect at an intersection 317.

FIG4 shows an imaging configuration 400 in which the camera 110 captures raw image data 117 of the wire grid 300, shown as a cross-section along line 434. The camera 110 is positioned such that the optical axis 432 of the imaging lens 112 intersects the wire grid 300 at intersection 315, and the camera 110 is perpendicular to the plane of the wire grid 300. Thus, the imaging lens 112 images the wire grid 300 onto the image sensor 116, capturing the raw image data 117. In the right-hand coordinate axis 430, the y-axis is perpendicular to the xz plane.

FIG5 shows the raw image data 117 captured by the camera 110 of the wire grid 300 when the wire grid 300 is configured as shown in FIG4 . The right-hand coordinate axis 534 represents a 90° rotation of the coordinate axis 430 about the x-axis, such that the z-axis of the coordinate axis 534 is perpendicular to the x-y plane. The coordinate axis 534 shows the orientation of the raw image 512 relative to the camera 110 and the wire grid 300.

In the original image data 117, the vertical line 502(4) is a mirror image of the center vertical line 302(4), and the horizontal line 501(4) is a mirror image of the horizontal line 301(4). The barrel distortion of the original image data 117 is a common feature of images captured using a wide-angle lens, which causes the horizontal lines 301(0)-(3) and 301(5)-(8) to bend, and the vertical lines 502(0)-(3) and 502(5)-(8) to bend. The vertical line 302(4) and the horizontal line 301(4) intersect at the center intersection 515 of the original image data 117, and because the barrel distortion at this location is minimal in the original image data 117, they can remain relatively straight. The intersection 515 is a mirror image of the intersection 315, and the intersection 517 is a mirror image of the intersection 317. As shown in FIG. 4 , the principal ray from the center of the wire grid 300 travels along the optical axis 432 of the imaging lens 112.

In the example of FIGS. 3-6 , the output view data 121 is centered at the intersection 317 of the line grid 300 , thereby generating a coordinate map 205 that maps the output view data 121 to the region 516 defined by the dashed lines of the original image data 117 .

6 shows an output view 123 corresponding to region 516 generated by the image conversion and multi-view output system 100. In the output view 123, vertical line 602(4) corresponds to vertical line 302(4) of the wire grid 300, and horizontal line 601(4) corresponds to horizontal line 301(2) of the wire grid 300. The intersection 617 of horizontal line 601(4) and vertical line 602(4) thus corresponds to intersection 317 of the wire grid 300. Since intersection 317 is not located on the optical axis 432 of the camera 110, the principal ray from intersection 317 to the camera 110 has a non-zero viewing angle 436 to the optical axis 432. Thus, the coordinate mapping 205 includes perspective and distortion correction, so that the inverse mapper 206 generates the output view data 121 from the region 516 of the original image data 117 with perspective and distortion correction. As shown, lines 601(2) and 602(4) of the output view data 121 are substantially straight. It should be noted that when the output view data 121 is rectilinear, the region 516 in the original image data 117 is not necessarily straight.

To generate the output view data 121, the inverse mapper 206 uses the lookup table 204 to map the pixel coordinates of the output view data 121 to locations in the original image data 117. For example, the pixels representing the intersection 617 are inverse mapped to the location of the intersection 517 in the original image data 117. Similarly, the pixels representing the horizontal line 601(4) and the vertical line 602(4) are mapped to corresponding locations within the curved horizontal line 501(2) and the curved vertical line 502(4) in the original image data 117.

FIG7 illustrates an exemplary image conversion and multi-view output method 700 for generating the output view data 121 of FIG1 from the raw image data 117 using the lookup table 204 of FIG2 . For example, the method 700 is performed in the inverse mapper 206 of the image conversion and multi-view output system 100 . Step 701 is optional. If included, in step 701, the method 700 determines the selection of an output view. In one example of step 701, the multi-view selection module 208 determines the selection of the output view data 121 from the multi-view selection switch 122, thereby selecting the coordinate mapping 205 in the lookup table 204. In step 702, the method 700 receives the raw image data. In one example of step 702, the image conversion and multi-view output system 100 receives the raw image data 117 from the image sensor 116 .

In step 706, method 700 de-maps each output pixel to a location in the original image data. In the example of step 706, de-mapper 206 de-maps the output pixel of output view data 121 to a location in original image data 117 using lookup table 204. In step 708, method 700 determines a pixel intensity value for the output view pixel based on the original image data at the mapped location. In the example of step 708, intensity setting module 216 of de-mapper 206 uses pixel interpolation to approximate the mapped location in original image data 117 to determine the value for the corresponding pixel in output view data 121. As indicated by dashed outline 704, steps 706 and 708 are repeated until all pixels of output view data 121 are mapped and assigned a value. Each output pixel can be mapped sequentially or in parallel, depending on the execution of de-mapper 206.

In step 710, method 700 outputs the output view data. In one example of step 710, image conversion and multi-view output system 100 sends output view data 121 to display 120. When applied to the processing of video frames, the steps of method 700 are repeated for each received raw image data (video frame).

FIG8 shows an exemplary original image 802, which represents the original image data 117, and three exemplary output views 804, 806, and 808, all generated by the image conversion and multi-view output system 100. In the example of FIG8, the output view 806 is formed into two exemplary sub-views 810(1) and 810(2), which are generated using a single coordinate mapping 205. The output views 804, 806, and 808 are generated from the original image 802 by the image conversion and multi-view output system 100 of FIG1, for example, using the method 700 of FIG7. Each sub-view 810(1) and 810(2) of the output view 804 and the output view 806 respectively shows a rectilinear correction of the center, left, and right views of the original image 802. The output view 808 shows a vertical correction of the original image 802, wherein only the vertical distortion in the original image 802 is corrected while maintaining a nearly complete horizontal viewing angle.

FIG9 shows a warped image 904 and an output view 902 generated by the image conversion and multi-view output system 100 using perspective and distortion correction in the lookup table 204 and according to the method 700 of FIG7 . The warped image 904 represents the original image data 117 of FIG1 , and the output view 902 represents the output view data 121. Using the method 700, the system 100 inversely maps the pixels corresponding to the sphere 908 in the output view 902 to the positions of the sphere 906 in the warped image 904.

Distorted image 904 exhibits barrel distortion, as seen in the image where ceiling line 909 curves away from the center of the image. Coordinate mapping 205 combines perspective and/or distortion correction so that in output view 902, ceiling line 907 appears straight. While only barrel distortion is shown, coordinate mapping 205 can also be applied to other types of distortion correction without departing from the scope of the present invention. For example, coordinate mapping 205 can be applied to correct for pincushion distortion, and any combination of barrel and pincushion distortion (i.e., compound distortion) can be corrected.

FIG10 is a schematic diagram illustrating an exemplary coordinate map generator 1000 for generating the coordinate map 205 for use in the lookup table 204 of FIG2 . The coordinate map generator 1000 is, for example, a computer and includes a processor 1002 and a memory 1010 storing software 1020, including a perspective correction algorithm 1014. Memory 1010 represents one or both of volatile and non-volatile memory as is known in the art. Software 1020 includes machine-readable instructions executed by processor 1002 to perform the functions of the coordinate map generator 1000 as described below. Memory 1010 also illustrates the function of storing imaging lens data 1012, including one or both of a distortion curve 1030 and a distortion center 1032 of the imaging lens 112 ( FIG1 ). The imaging lens data 1012 may also determine the field of view (e.g., field of view 113) of the imaging lens 112.

Memory 1010 also stores raw image data parameters 1038 that define characteristics of raw image data 117 and output view parameters 1040 that define characteristics of output view data 121. For example, raw image data parameters 1038 include pixel array dimensions of raw image data 117 corresponding to the pixel array of image sensor 116, and output view parameters 1040 define the pixel array dimensions of output view 123.

Output view parameters 1040 may also include the output view size, i.e., the digital zoom relative to the original image area. The geometric shape is defined by the angular range 436 ( FIG. 4 ) of the portion of vertical line 502 ( FIG. 5 ) across subview 516 in the x-z plane and the angular range of the portion of curved horizontal line 501 ( 2 ) across subview 516 in the y-z plane.

For example, perspective angle 1016 is defined using x-axis pitch, y-axis yaw, and z-axis roll rotations corresponding to a desired output view 123 relative to optical axis 432 of camera 110. One or both of imaging lens data 1012 and perspective angle 1016 are provided by a user via user input device 1018. Imaging lens data 1012 may be determined via measurements of imaging lens 112. As shown in FIG4 , x-axis pitch, y-axis yaw, and z-axis roll are rotations about the x-axis, y-axis, and z-axis, respectively.

Subviews 810(1) and 810(2) ( FIG. 8 ) show the perspective of output view 123. The vertical perspective (pitch) in each of subviews 810(1) and 810(2) is zero because it does not differ from the vertical perspective of original image 802. The horizontal perspective (yaw) of subviews 810(1) and 810(2) differs from each other and from the horizontal perspective (yaw) of original image 802.

The perspective correction algorithm 1014, when executed by the processor 1002, performs a mathematical inverse perspective correction on the output view parameters 1040 based on one or more of the original image data parameters 1038 and the perspective viewing angle 1016, thereby generating intermediate mapping data 1050 in the memory 1010. The processor 1002 then executes a distortion correction algorithm 1015 to generate the coordinate map 205 based on the intermediate mapping data 1050, the original image data parameters 1038, and one or both of the distortion curve 1030 and the distortion center 1032. The coordinate map 205 thus includes perspective correction and distortion correction to inversely map pixel coordinates of the output view data 121 to the original image data 117.

11 is a flow chart illustrating an exemplary method 1100 for generating the coordinate mapping 205 for use in the lookup table 204. For example, the method 1100 is executed in the software 1020 of FIG. 10 to generate the coordinate mapping 205 for each of the one or more desired output views 123 based on characteristics of the imaging lens 112 and the raw image data 117.

In step 1101 , the method 1100 receives imaging lens parameters. In an example of step 1101 , the coordinate map generator 1000 receives input from a user indicating a field of view of the imaging lens 112 .

In step 1102, method 1100 receives parameters of the raw image data. In one example of step 1102, coordinate map generator 1000 receives input from a user indicating attributes of raw image data 117, such as pixel array dimensions, number of pixels, and pixel pitch.

In step 1103, method 1100 determines a distortion curve for the lens. In one example of step 1103, coordinate map generator 1000 controls and/or instructs a calibration device to measure distortion curve 1030 for imaging lens 112. In another example of step 1103, coordinate map generator 1000 receives distortion curve 1030 from a third party, such as user input data provided by the manufacturer of imaging lens 112.

In step 1104, method 1100 determines a distortion center corresponding to the lens. In one example of step 1104, coordinate map generator 1000 controls and/or instructs a calibration device to measure and generate the distortion center 1032 of imaging lens 112. In another example of step 1104, coordinate map generator 1000 receives the distortion center 1032 from a third party, such as via user input data provided by the manufacturer of imaging lens 112.

In step 1106, method 1100 receives at least one perspective angle for use in generating an output view. Each of the at least one perspective angle represents an angle within the field of view of an imaging lens. In one example of step 1106, coordinate map generator 1000 receives an indication from a user representing perspective angles rotated about the x-axis, about the z-axis, and about the y-axis. In another example of step 1106, coordinate map generator 1000 receives an indication from a user representing specified values for one or both of the perspective angles rotated about the x-axis, about the z-axis, and about the y-axis, wherein unspecified perspective angles are set to zero.

At step 1107, method 1100 receives an output view size. In the example of step 1107, the output view size is defined and input by the user as two viewing angles in orthogonal planes relative to the axes defined by the perspective angle received in step 1106.

The information provided in steps 1106 and 1107 defines a sub-field of view of imaging lens 112 , where the sub-field of view is converted into output view 123 by system 100 .

In step 1108, method 1100 applies perspective correction to the intermediate mapping data using inverse-mapped pixel coordinates (e.g., pixel coordinates x _out , y _out ) of the selected output view data using mathematical calculations based on the at least one perspective angle received in step 1106. In one example of step 1108, processor 1002 executes instructions of perspective correction algorithm 1014 based on output view parameters 1040 to generate intermediate mapping data 1050.

In step 1112, method 1100 applies distortion correction to the original image data locations by inverse-mapping the intermediate map data generated in step 1108 using the distortion curves and/or distortion centers. In the example of step 1112, the distortion correction algorithm 1015 of the coordinate map generator 1000 processes the intermediate map data 1050 to combine one or both of the distortion curves 1030 and the distortion centers 1032, as determined in steps 1103 and 1104, respectively. In step 1114, method 1100 outputs the coordinate map. In the example of step 1114, the coordinate map generator 1000 outputs the coordinate map 205 for use in the lookup table 204. For example, where the image conversion and multi-view output system 100 is implemented as a single package (e.g., a single integrated circuit), the coordinate map generator 1000 may write the coordinate map 205 to the memory 202 of the image conversion and multi-view output system 100.

Steps 1106 through 1114 are repeated, as indicated by dashed outline 1105, to generate a coordinate map 205 for each output view. For example, to generate a coordinate map 205 corresponding to each output view 804, 806, and 808, steps 1106 through 1114 are repeated three times: first, a coordinate map 205 corresponding to output view 804 is generated; second, a coordinate map 205 corresponding to output view 806 is generated; and third, a coordinate map 205 corresponding to output view 808 is generated. For the second iteration, coordinate map 205 converts original image 802 into subviews 810(1) and 810(2), steps 1106 through 1112 are repeated for each subview 810, and step 1114 combines the multiple coordinate maps 205 to generate a single coordinate map 205 for use in lookup table 204.

The order of the steps in method 1100 may be changed without departing from the scope of the present invention. For example, step 1104 may precede step 1103, and step 1107 may precede step 1106.

The above methods and systems may be modified without departing from the scope of the present invention. It should be noted that the above description and the accompanying drawings are to be interpreted as illustrative and not restrictive. The following claims are intended to cover all generic and specific features described herein, as well as all statements of the scope of the methods and systems of the present invention, which, by definition, may be construed as falling within the scope thereof.

Claims (24)

1. A method for image conversion and multi-view output to a display device, comprising: Raw image data generated by an image sensor is received into the system's memory, the system including an inverse mapper and a lookup table stored in non-volatile memory; Using coordinate mapping to correct both (a) perspective and (b) distortion in the original image, and using the coordinate mapping module and lookup table of the inverse mapper, each pixel coordinate of the output view data stored in memory is mapped to its position in the original image data. Mathematical inverse perspective correction is performed on the output view parameters based on one or more of the original image data parameters and perspective angles to generate intermediate mapping data in the non-volatile memory. Then, a distortion correction algorithm is performed based on the intermediate mapping data, the original image data parameters, and one or both of the distortion curve and distortion center of the imaging lens to generate the coordinate mapping. Based on the information closest to the location in the original image data, the intensity setting module of the inverse mapper determines the intensity of each output pixel in the output view data; and Output view data to the display device. 2. The method of claim 1 further includes selecting the coordinate mapping from a plurality of coordinate mappings stored in the lookup table. 3. The method of claim 1, wherein in the mapping step, at least two subviews are generated in the output view data. 4. The method of claim 1, wherein the location in the original image data is a pixel in the original image data. 5. The method of claim 1, wherein the location in the original image data is a pixel group identified in the original image data. 6. The method of claim 1, wherein the mapping step is to correct distortion along the axis of the image plane. 7. The method of claim 1, wherein the mapping step is to correct distortion along two orthogonal axes of the image plane. 8. The method of claim 1, wherein the mapping step includes selecting an image transformation from the group consisting of scaling and rotation functions. 9. The method of claim 1, wherein the decision step includes an interpolation algorithm. 10. A system for image conversion and multi-view output, comprising: A lookup table, stored in non-volatile memory, contains coordinate mappings that map pixels in the output view data to their positions in the original image data; and An inverse mapper that uses the coordinate mapping to generate the output view data from the original image data, wherein mathematical inverse perspective correction is performed on the output view parameters based on one or more of the original image data parameters and perspective angles to generate intermediate mapping data in the non-volatile memory, and then a distortion correction algorithm is performed based on one or both of the intermediate mapping data, the original image data parameters, and the distortion curve and distortion center of the imaging lens to generate the coordinate mapping; The coordinate mapping includes both (a) perspective correction and (b) distortion correction. 11. The system of claim 10, wherein the inverse mapper includes an intensity setting module for interpolating between at least two input pixels in the original image data to generate an intensity value for each output pixel of the output view data. 12. The system of claim 10, wherein the inverse mapper is capable of correcting distortion along the axis of the image plane. 13. The system of claim 10, wherein the inverse mapper is capable of correcting distortion along two orthogonal axes of the image plane. 14. The system of claim 10, wherein the output view data includes two independent subviews of the original image data. 15. The system of claim 10, further comprising: At least one additional coordinate mapping, which is stored in the lookup table; and A multi-view selection module is provided for (a) receiving an input indication of a selective output view, and (b) selecting one of the coordinate mappings and the additional coordinate mappings for use by the inverse mapper to produce the output view data. 16. The system of claim 10, wherein the coordinate mapping is capable of performing one or more scaling and rotations. 17. The system of claim 10, wherein the lookup table and the inverse mapper are configured as a single integrated circuit. 18. The system of claim 10, wherein the output view data is sent to a display device for display to a user. 19. The system of claim 10, wherein the original image data is a frame in the input video sequence data, and wherein the output view data is displayed as a frame in the live video stream. 20. A method for generating coordinate mappings for use in a lookup table by an image transformation and multi-view output system, comprising: Receive parameters of the imaging lens, including the distortion curve and distortion center of the imaging lens; Parameters for receiving raw image data; Receive at least one perspective angle in the output view corresponding to the image conversion and multi-view output system; Inverse perspective correction is applied to the output view data based on at least one perspective angle to produce intermediate mapping data; Inverse distortion correction is applied to the intermediate mapping data based on at least one of the distortion curve and the distortion center to produce the coordinate mapping; and The coordinate mapping output is used in the image conversion and multi-view output system, wherein the coordinate mapping corrects both (a) perspective and (b) distortion. 21. The method of claim 20, wherein the parameter includes the field of view of the imaging lens. 22. The method of claim 20, wherein the original image data parameters include pixel positions. 23. The method of claim 20, wherein the coordinate mapping includes a scaling operation between the original image data and the output view data. 24. The method of claim 20, wherein the coordinate mapping is to map each pixel of the output view data to a position in the original image data.