JP2018174429A - Image processing apparatus, photographing apparatus, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To create a spherical image, a process of connecting two hemispherical images is required. Since each hemispherical image is obtained through a fisheye lens with an angle of view of about 210 °, conventionally, a special digital camera takes out an image of the joint region of each hemispherical image and performs distortion correction processing. , I was doing template matching. As the seam processing by this technique, each corresponding area is cut out as a reference image and a comparison image, and template matching is performed on the image after distortion correction. However, as the image resolution increases, if a method of acquiring a large reference image and a comparative image in accordance with the increase in resolution is adopted, the memory band in the image processing device in the digital camera becomes imminent. When an image processing device 10 is assembled in a photographing device 1, a test pattern TP (TPa, TPb) is photographed and stored in an external storage device 50 for use in pattern matching processing. [Selection diagram] FIG. 25

Description

  The present invention relates to an invention for performing composition by joining a plurality of images expressed by first and second image data.

  In recent years, there has been provided a special digital camera that obtains two hemispherical image data based on a 360-degree spherical image by one shooting (see Patent Document 1). This special digital camera creates one equirectangular projection image data based on two hemispherical image data, and transmits the equirectangular projection image data to a communication terminal such as a smartphone. The communication terminal that has obtained the equirectangular projection image data creates an omnidirectional image based on the equirectangular projection image data. However, since the image is curved as it is, it is difficult for the user to see. Therefore, by displaying a predetermined area image indicating a predetermined area of the omnidirectional image on the communication terminal, the user is photographed with a general digital camera. Can be viewed with the same feeling as a flat image.

  In addition, when creating an omnidirectional image, a process for joining two hemispherical images is required. Since each hemisphere image is obtained through a fisheye lens having an angle of view of 180 ° or more, a special digital camera has conventionally used a joint of each hemisphere image to create a 360 ° omnidirectional image. Image quality is improved by extracting a region image and performing distortion correction processing and then performing template matching (see Patent Document 2). As a joint process by this technique, corresponding regions are cut out as a reference image and a comparison image, and template matching is performed on the image after distortion correction.

  However, as the image resolution becomes higher, if the method of acquiring the reference image and the comparative image greatly in accordance with the higher resolution is adopted, the memory bandwidth in the image processing device in the special digital camera is imminent. The problem that it will be done arises.

  In order to solve the above-mentioned problem, an invention according to claim 1 is an image processing apparatus for joining a plurality of images expressed by first and second image data, wherein the first and second image data are connected. On the other hand, a first distortion correction unit that performs a process of correcting image distortion, a first image related to the first image data subjected to the process by the first distortion correction unit, and the first In the overlapping region when the second image related to the second image data that has been processed by the distortion correction unit is connected, a comparison image indicating a comparison region of the second image, A pattern matching unit that performs a pattern matching process on a reference image that represents a reference area of the first image, and a pattern matching process that is performed by the pattern matching unit calculates a shift amount of the overlapping area. An amount calculation unit, a parameter creation unit that creates a distortion correction parameter for correcting distortion of at least one of the first image and the second image according to the amount of deviation, and the parameter creation unit A second distortion correction unit that performs processing for correcting image distortion for at least one of the first and second image data using the generated distortion correction parameter, and The shift amount calculation unit is an image processing apparatus that performs the pattern matching using a test pattern used for the pattern matching process in at least the overlapping region of the first image and the second image. is there.

  According to the present invention, it is possible to avoid imminent memory bandwidth in an image processing apparatus in a photographing apparatus such as a special digital camera.

(A) is a right side view of the special photographing apparatus, (b) is a rear view of the special photographing apparatus, (c) is a plan view of the special photographing apparatus, and (d) is a bottom view of the special photographing apparatus. It is. It is a usage image figure of a special imaging device. (A) is a hemispheric image (front) photographed by a special photographing device, (b) is a hemispheric image (rear) photographed by a special photographing device, and (c) is an image represented by equirectangular projection. FIG. (A) is the conceptual diagram which showed the state which covered the sphere with an equirectangular projection image, (b) is the figure which showed the omnidirectional image. It is the figure which showed the position of a virtual camera and a predetermined area | region at the time of making a spherical image into a three-dimensional solid sphere. (A) is a three-dimensional perspective view of FIG. 5, (b) is a figure which shows the state in which the image of the predetermined area is displayed on the display of a communication terminal. FIG. 6 is a diagram illustrating a relationship between predetermined area information and an image of a predetermined area T. It is a hardware block diagram of the image processing apparatus in the imaging device concerning this embodiment. It is a functional block diagram of the image processing apparatus in the imaging device according to the present embodiment. It is a figure explaining the flow until a matching process is complete | finished. It is a figure explaining the overlapping area | region of each hemisphere image. It is a figure explaining the overlapping area | region of each hemisphere image. It is a figure at the time of dividing an overlap area in each hemisphere image, respectively. (A) Conceptual diagram of reference image, (b) Conceptual diagram of comparative image. It is a figure explaining the example of the calculation method of the deviation | shift amount by a matching process block. It is the figure which showed the image showing a to-be-photographed object. (A) is a diagram showing a basic image in an FHD image representing a subject, and (b) is a diagram showing a basic image in a 4K image representing the subject. (A) is a diagram showing a matching area in a comparison image in the case of an FHD image representing a subject, and (b) is a diagram showing a matching area in a comparison image in the case of a 4K image representing the subject. It is the figure which showed the data transfer image relevant to a memory band based on the block diagram of FIG. It is a conceptual diagram of the test pattern of a template. It is an enlarged view of the upper right part of FIG. FIG. 22 is a diagram illustrating an example when distortion correction is performed on the overlapping region of the reference image (hemisphere image A). FIG. 23 is a diagram exemplifying a case where the overlapping region of the comparative image (hemispheric image B) is corrected for distortion. FIG. 24 is an enlarged view when the area Aa shown in FIG. 22 and the area Ab shown in FIG. 23 are overlapped. It is the figure which set this embodiment and the conventional matching area | region.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<< Summary of Embodiment >>
First, the outline of this embodiment will be described with reference to FIGS. 1 to 7.

  First, the external appearance of the photographing apparatus 1 will be described with reference to FIG. The photographing apparatus 1 is a digital camera for obtaining a photographed image that is a source of the omnidirectional image. 1A is a right side view of the special photographing apparatus, FIG. 1B is a rear view of the special photographing apparatus, and FIG. 1C is a plan view of the special photographing apparatus. (D) is a bottom view of the special photographing apparatus.

  As shown in FIGS. 1 (a), 1 (b), 1 (c), and (d), a fish-eye lens is placed on the front side (front side) of the special photographing apparatus 1a. A fisheye lens 102b is provided on the rear side (rear side) 102a. Imaging devices (image sensors) 103a and 103b, which will be described later, are provided inside the special imaging device 1a, and a hemispherical image (angle of view 180 °) is obtained by shooting a subject or a landscape through the lenses 102a and 102b, respectively. Above). A shutter button 115a is provided on the surface opposite to the front side of the special imaging device 1a. Further, a power button 115b, a WiFi (Wireless Fidelity) button 115c, and a shooting mode switching button 115d are provided on the side surface of the special imaging device 1a. Each time the power button 115b and the WiFi button 115c are pressed, they are switched on and off. The shooting mode switching button 115d switches between a still image shooting mode and a moving image shooting mode each time the button is pressed. The shutter button 115a, the power button 115b, the WiFi button 115c, and the shooting mode switching button 115d are a kind of the operation unit 115, and the operation unit 115 is not limited to these buttons.

  Further, a tripod screw hole 151 for attaching the special imaging device 1a to the camera tripod or the general imaging device 3a is provided at the center of the bottom 150 of the special imaging device 1a. A micro USB (Universal Serial Bus) terminal 152 is provided on the left end side of the bottom 150. On the right end side of the bottom 150, an HDMI (High-Definition Multimedia Interface) terminal is provided. HDMI is a registered trademark.

  Next, the use situation of the special imaging device 1a will be described with reference to FIG. In addition, FIG. 2 is a use image figure of a special imaging device. As shown in FIG. 2, the special imaging device 1 a is used, for example, for a user (photographer) to take a subject around the user while holding it in his hand. In this case, two hemispherical images can be obtained by imaging the subject around the user by the imaging device 103a and the imaging device 103b shown in FIG.

  Next, an outline of processing from the image captured by the special imaging device 1a to the creation of the equirectangular projection image EC and the omnidirectional image CE will be described with reference to FIGS. 3A is a hemispheric image (front side) photographed by the special photographing apparatus 1a, FIG. 3B is a hemispheric image photographed by the special photographing apparatus (rear side), and FIG. It is the figure which showed the image (henceforth "an equirectangular projection image") represented by the cylindrical projection. FIG. 4A is a conceptual diagram showing a state where a sphere is covered with an equirectangular projection image, and FIG. 4B is a diagram showing an omnidirectional image.

  As shown in FIG. 3A, the image obtained by the image sensor 103a is a hemispherical image (front side) curved by a fish-eye lens 102a described later. Also, as shown in FIG. 3B, the image obtained by the image sensor 103b is a hemispherical image (rear side) curved by a fish-eye lens 102b described later. Then, the hemispherical image (front side) and the hemispherical image inverted by 180 degrees (rear side) are synthesized by the special imaging device 1a, and as shown in FIG. 3C, the equirectangular projection image EC. Is created.

  Then, by using OpenGL ES (Open Graphics Library for Embedded Systems) or the like, the equirectangular projection image is pasted so as to cover the spherical surface as shown in FIG. An omnidirectional image CE as shown in (b) is created. In this way, the omnidirectional image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. OpenGL ES is a graphics library used for visualizing 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the omnidirectional image CE may be a still image or a moving image.

  As described above, since the omnidirectional image CE is an image that is pasted so as to cover the spherical surface, it is uncomfortable when viewed by a human. Therefore, by displaying a predetermined area (hereinafter referred to as “predetermined area image”) of a part of the omnidirectional image CE as a flat image with little curvature, a display that does not give a sense of discomfort to humans can be achieved. This will be described with reference to FIGS.

  FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional solid sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the omnidirectional image CE displayed as a three-dimensional solid sphere. FIG. 6A is a three-dimensional perspective view of FIG. 5, and FIG. 6B is a diagram showing a predetermined area image when displayed on the display. In FIG. 6A, the omnidirectional image shown in FIG. 4 is represented by a three-dimensional solid sphere CS. Assuming that the omnidirectional image CE generated in this way is a three-dimensional sphere CS, the virtual camera IC is positioned inside the omnidirectional image CE as shown in FIG. The predetermined area T in the omnidirectional image CE is a shooting area of the virtual camera IC. The zoom of the predetermined region T can be expressed by expanding or contracting the range (arc) of the angle of view α. The zoom of the predetermined area T can also be expressed by moving the virtual camera IC closer to or away from the omnidirectional image CE. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the omnidirectional image CE (see FIG. 7).

  Then, the predetermined area image Q shown in FIG. 6A is displayed on the predetermined display as an image of the photographing area of the virtual camera IC as shown in FIG. 6B. The image shown in FIG. 6B is a predetermined area image represented by the predetermined (default) predetermined area information. The predetermined area T may be indicated not by the angle of view α and the distance f but by the imaging area (X, Y, Z) of the virtual camera IC that is the predetermined area T.

  Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. FIG. 7 is a diagram showing the relationship between the predetermined region information and the relationship between the images of the predetermined region T. As shown in FIG. 7, the center point CP when the diagonal field angle of the predetermined region T represented by the field angle α of the virtual camera IC is α is the (x, y) parameter of the predetermined region information. It becomes. f is the distance from the virtual camera IC to the center point CP. L is a distance between an arbitrary vertex of the predetermined region T and the center point CP (2L is a diagonal line). In FIG. 7, a trigonometric function represented by the following (formula 1) is generally established.

L / f = tan (α / 2) (Formula 1)
Hereinafter, the detailed content of the embodiment of the present invention will be described.

<< Details of Embodiment >>
<Hardware configuration>
Next, the hardware configuration of the image processing apparatus 1 will be described with reference to FIG. FIG. 8 is a hardware configuration diagram of the image processing apparatus in the photographing apparatus according to the present embodiment. As shown in FIG. 8, the image processing apparatus 10 in the photographing apparatus 1 includes input I / Fs 11a and 11b, ISPs (Image Signal Processors) 12a and 12b, and DMACs (Direct Memory Access Controllers) 13a to 13g. , A matching processing block 14, a distortion correction block 15, interconnects 20 and 30, a MEMC (Motion Estimation Motion Combination) 21, and a CPU 40. In addition to the image processing apparatus 10, the photographing apparatus 1 includes image pickup devices 103 a and 103 b, an external storage device 50, and a flash ROM (Read Only Memory) 60. The external storage device 50 is, for example, a DRAM (Dynamic Random Access Memory).

  Then, the frame image data as the data of the hemispherical images A and B respectively input from the imaging elements 103a and 103b passes through the ISP (Image Signal Processing) circuits 11a and 11b, so that the matching processing 0 block 14 and the distortion correction are performed. Image processing is performed in the block 15 and finally transferred to the external storage device 50 using the MEMC 21.

  Since both the matching processing block 14 and the distortion correction block 15 use an image on which distortion correction processing has been performed, the interconnect 30 preliminarily sets parameters for performing distortion correction for matching (hereinafter referred to as “parameter Pa”) to the flash ROM 60. After the power is turned on, the matching processing block 14 reads each hemispherical image data developed from the flash ROM 60 to the external storage device 50 by the interconnects 30 and 20, and also reads the parameter Pa from the fresh ROM 60. .

  Further, the matching processing block 14 performs distortion correction processing on the matching target region (overlapping region) of each hemispherical image according to the parameter Pa, and sets the hemispherical image A side as a reference image and the hemispherical image B side as a comparative image. Then, pattern matching processing is performed. At this time, the matching processing block 14 converts the fisheye images of the hemispherical images A and B that have a circular overlapping area into Equi images (equirectangular projection images), respectively. Note that the number of divided areas is arbitrary for the creation of each Equi image. This is because it is affected by the size of SRAM (Static Random Access Memory: image storage area after distortion correction) mounted in the matching processing block 14. By repeatedly performing the matching process of the number of divided areas, it is possible to calculate the amount of deviation of each overlapping region, and therefore the lens seed parameter stored in the flash ROM 60 and developed in the external storage device 50 is used. The CPU 40 shifts (interpolates) the distortion correction parameter Pa of the hemispheric image B stored in advance in the external storage device 50 based on the calculated amount of deviation, so that the final distortion correction parameter ( (Hereinafter referred to as “parameter Pb”) is created and stored in the external storage device 50.

  Then, the distortion correction block 15 stores the parameter Pa for the hemispheric image A that has been stored in advance in the external storage device 50 and the parameter Pb for the hemispheric image B that has been created and stored in the external storage device 50. Read from device 50. Further, the distortion correction block 15 reads the data of the hemispheric images A and B stored in the external storage device 50. Then, the distortion correction block 15 uses the distortion correction parameters Pa and Pb to perform joint processing on the distorted hemisphere images A and B as shown in FIGS. 3 (a) and 3 (b). Thus, one Equi image (an equirectangular projection image) shown in FIG. 3C is created. The created Equi image data is temporarily stored in the external storage device 50 and then output from the image processing device 10 via the interconnect circuit 20 and the DMAC 13g. The distortion correction block 15 may output the single Equi image data from the image processing apparatus 10 via the interconnect circuit 20 and the DMAC 13g without storing the Equi image data in the external storage apparatus 50.

<Functional configuration>
Next, the functional configuration of the image processing apparatus 10 will be described with reference to FIG. FIG. 9 is a functional block diagram of the image processing apparatus in the photographing apparatus according to the present embodiment.

  As shown in FIG. 9, the matching processing block 14 can realize the functions of the first distortion correction unit 24, the pattern matching unit 25, the deviation amount calculation unit 26, and the parameter creation unit 27. Further, the distortion correction block 15 can realize the function of the second distortion correction unit 28. The external storage device 50 is an example of a storage unit, and stores image data of a test pattern obtained by photographing a predetermined test pattern after being assembled to the photographing device 1 at a factory. This test pattern will be described in detail later.

  The first distortion correction unit 24 reads hemispherical images A and B (image data) from the external storage device 50 as a storage unit via the interconnect circuit 30, and converts the circular image into a rectangular image (Equi image). Distortion correction to be deformed. The distortion correction is performed using a parameter Pa prepared in advance (matching distortion correction). Each image after the distortion correction is stored in the external storage device 50 via the interconnect circuit 20. Since the parameter Pa prepared in advance is matched with the optical system of the photographing apparatus 1, it is fixed. The first distortion correction unit 24 is a function realized by the matching processing block 24 of FIG.

  Hereinafter, an image related to the image data after the hemispheric image A has been subjected to distortion correction by the first distortion correction unit 24 will be referred to as “deformed image A”, and the hemispheric image B will be subjected to distortion correction by the first distortion correction unit 24. An image related to the image data after being processed is shown as “deformed image B”.

  The pattern matching unit 25 is a reference indicating the reference region of the modified image A with respect to the comparison image indicating the comparison region of the deformed image B in the overlapping region when the deformed image A and the deformed image B are connected. Perform pattern matching on the image.

The deviation amount calculation unit 26 performs pattern matching processing performed by the pattern matching unit 25.
This is a function for calculating (specifying) the shift amount of the overlapping area.

  The parameter creation unit 27 creates a (distortion correction) parameter Pb for correcting the distortion of at least one of the deformed image A and the deformed image B in accordance with the shift amount calculated by the shift amount calculating unit 26. This parameter Pb is for performing distortion correction of the deformed image B so that the deformed image B excluding the overlapped portion after correction is continuous with the deformed image A with a natural feeling. Thereby, the parameter correction unit 17 uses the difference between the deviation amount assumed by the parameter Pa prepared in advance (for example, the width of the overlapping portion) and the actual deviation amount, and performs interpolation processing of the parameter Pa prepared in advance. Thus, the parameter Pb to be used for distortion correction is created for each region.

  The second distortion correction unit 28 creates a deformed image B ′ by performing distortion correction in units of regions on the deformed image B using the parameter Pb for each region after being determined. The data of the deformed image B ′ is stored in the external storage device 50 via the interconnect circuit 20. The data of the deformed image B 'is obtained by excluding, for example, image data of overlapping portions. The data of the deformed image A and the data of the deformed image B ′ are output from the external storage device 50 to the outside via the interconnect circuit 20 and the DMAC 13g. The data of the deformed image A and the data of the deformed image B ′ are not temporarily stored in the external storage device 50, but may be directly output to the outside from the distortion correction block 15 via the interconnect circuit 20 and the DMAC 13g. Good.

<< Process of Embodiment >>
Next, the processing of this embodiment will be described with reference to FIGS. FIG. 10 is a diagram illustrating a flow until the matching process is completed.

  In FIG. 10, “Start” indicates that the imaging apparatus 1 in which the image processing apparatus 10 is mounted is turned on. Since a product such as the photographing apparatus 1 is assembled with various parts, a shift including an assembly error occurs in each product. When the joint (matching distortion correction) parameter Pa is first created, it is necessary to create it using a template of a predetermined test pattern (or test chart or guide) before shipping the product. The amount of deviation (horizontal direction and vertical direction) between the hemispherical images A and B is calculated from the frame image obtained by photographing the test pattern, and a parameter Pa is created (step S11). The created parameter Pa is stored in the data of the flash ROM so that it can be used in the product (step S12). Then, when the power is turned on when using the photographing apparatus 1, the parameter Pa stored in the flash ROM 60 by software processing is expanded in the external storage device (step S13). Then, the matching processing block 14 reads out the parameter Pa and the like via the respective DMACs 13e and 13f (step S14). The matching processing block 14 creates Equi images from the hemispherical images A and B based on the parameter Pa (step S15). The Equi image of the hemispherical image A is a reference image, and the Equi image of the hemispherical image B is a comparative image. Further, the matching processing block 14 performs pattern matching of the reference image with respect to the comparison image (step S16). Note that which of the hemispherical image A and the hemispherical image B is the reference image depends on how the parameter Pa is created. Then, after the pattern matching, the matching processing block 14 obtains the deviation amount, thereby creating the parameter Pb (step S17). Thereby, a series of processes until the matching process is completed. Hereinafter, the above-described pattern matching will be described in more detail.

  First, in describing pattern matching, overlapping regions of hemispherical images (fisheye images) A and B will be described with reference to FIGS. 11 and 12. FIG. 11 and FIG. 12 are diagrams for explaining overlapping regions of the respective hemispherical images.

  The outermost circles of the hemispherical images A and B shown in FIG. 11 indicate the range where the image area is arranged. In the case of a wide-angle lens having an angle of view exceeding 180 °, the portion indicated by the black line is an overlapping region. In order to match the hemispherical images A and B, distortion correction is performed as shown in FIG. 12, and matching processing is performed on the overlapping regions of the hemispherical images A and B. The parameter Pa (distortion correction for matching) is used to generate the image shown in FIG.

  Next, a template matching method will be described with reference to FIGS. FIG. 13 is a diagram when the overlapping area is divided in each hemisphere image.

  In FIG. 13, in order to perform pattern matching (template matching) on the overlapping region (blacked portion), for each overlapping region of the hemispherical image A as the reference image and the hemispherical image B as the comparison image, It shows that the area is divided by an arbitrary number of divisions. The idea of the number of divided areas is to match the images in the hemispherical image A and hemispherical image B, and those that appear far away, those that appear closer, or those that repeat like a brick tone It becomes matching for. At this time, if the matching area is too small, the accuracy of the repeated pattern matching will be reduced, but if the matching area is enlarged, a dynamic object enters the joint area, and the matching process in the far and near areas A large amount of shift and a small amount of shift occur at, and it is necessary to determine which should be given priority. Therefore, it is difficult to determine the matching size in general.

  FIG. 14A is a conceptual diagram of a reference image, and FIG. 14B is a conceptual diagram of a comparative image. Compared to the reference image, the comparison image requires an area for matching search in the vertical and horizontal directions during the matching process. Assuming that one area of the reference image is “A area 0”, XSIZE and YSIZE of A area 0 are set by the number of pixels and the number of lines. Further, in the comparison image, if the area to be compared with “A area 0” of the reference image is “B area 0”, it has a region width of ± α in the horizontal direction and ± β in the vertical direction. In order to increase the accuracy, matching over a wider range should be advanced over time, but increasing the size of the matching region requires time for matching processing of the previous area. In order to shorten the processing time, there is an idea that parallel processing may be performed. However, the reference image and the comparison image must be retained for each matching area until the processing is completed. In consideration of this design, the capacity of the SRAM built in the matching processing block 14 is also increased.

FIG. 15 is a diagram for explaining an example of a method for calculating a deviation amount by the matching processing block. In FIG. 15, the overlapping area is divided into N, and processing is performed for each area, and N shift amounts are output.
It is necessary to generate an image area including the ± α and ± β regions of the hemispherical image B with the matching distortion correction parameter Pa. That is, the matching processing block 14 calculates the amount of deviation for each region. “Α” and “β” in FIG. 5 are symbols indicating the width in the X direction and the Y direction when searching for the amount of deviation.

  The above is the processing content along the flow shown in FIG.

  Subsequently, the image resolution will be described with reference to FIGS. 16 to 18. FIG. 16 is a diagram illustrating an image representing a subject. FIG. 17A is a diagram showing a basic image in an FHD (Full High Definition) image representing a subject, and FIG. 17B is a diagram showing a basic image in a 4K image representing the subject. FIG. 18A is a diagram showing a matching area in a comparison image in the case of an FHD image representing a subject, and FIG. 18B is a diagram showing a matching area in a comparison image in the case of a 4K image representing the subject. FIG.

  First, as shown in FIG. 16, it is assumed that there is an image representing a subject and shows one of the matching areas described above. Image resolution is increasing year by year, and changes from an era when FHD images were mainstream to an era where 4K images are mainstream, there are problems like the image shown in FIG. Occur. In other words, when it is simply considered that the matching area needs to include all of the designated image area in the case of an FHD image, in the case of a 4K image, the number of pixels is four times as many (the image format is arbitrary) The problem arises that is necessary. Further, as shown in FIG. 18, similarly, there is a problem that the number of pixels is four times as large as that of the comparative image. Furthermore, increasing the range of the search width (± α, ± β) also causes a problem that if the resolution is increased, the image size required for one area process is increased.

  Subsequently, the memory area will be described with reference to FIG. FIG. 19 is a diagram showing a data transfer image related to the memory band based on the configuration diagram of FIG. Note that memory access from other functions such as a CPU is omitted because it is determined that the image transfer amount is very small.

  FIG. 19 shows processing in units of frames, and the input image is output three frames later. In the first frame, the input hemispherical images A and B are stored in the memory. In the second frame, matching processing is performed. In the third frame, distortion correction is performed. In the fourth frame, the created parameter Pb is output to the external storage device 50.

  In the frame image transfer from the input I / Fs 11a and 11b, data is transferred within the frame period. Therefore, the memory access processing for each line is not shown in FIG. 19, but the memory access is performed at regular intervals. Yes.

  As shown in FIG. 19, a system in which memory access commands are issued from seven DMACs 13a to 13g is considered as a normal system. As the image resolution becomes higher and the frame rate becomes faster, the memory bandwidth is imminent. Although it is possible to widen the applicable range by increasing the operating frequency of the memory and setting a wide data width, the entire photographing apparatus 1 such as an increase in power consumption and an increase in mounting area as a product of the photographing apparatus 1 is possible. It is impossible to find any merit for. On the other hand, since the memory bandwidth reduction related to the distortion correction processing is being studied and implemented, the present embodiment can reduce the memory bandwidth by setting an optimal matching area without lowering the matching accuracy. . Therefore, a method for reducing the memory bandwidth is shown below.

  FIG. 20 is a conceptual diagram of a template test pattern. FIG. 21 is an enlarged view of the upper right part of FIG. The test pattern TP has a plurality of radial straight lines from the same center on concentric circles. First, the range indicated by the arrow is an area range for template matching. For this reason, FIG. 20 shows that the number of areas for template matching is eight.

  Although the size needs to be suitable for the circuit mounted on the photographing apparatus 1, the number of divisions may be arbitrary. Here, for convenience of explanation, it is divided into eight. A circle-shaped thick black line portion is a test pattern for determining the current matching region. As shown in FIG. 20, test patterns TP in which black line portions appear are prepared for the hemispherical images A and B, respectively. The point where the black line portion crosses is also arbitrary, and the number is not limited.

  The matching processing block 14 performs distortion correction on each of the input hemispherical images A and B. When parameters Pa are used for images input using the same test pattern TP, the results are as shown in FIGS.

  FIG. 22 is a diagram illustrating an example when distortion correction is performed on the overlapping region of the reference image (hemisphere image A). FIG. 23 is a diagram exemplifying a case where the overlapping region of the comparative image (hemispheric image B) is corrected for distortion. If it remains as it is, template matching is performed between the area Aa filled in gray of the basic image and the area Ab filled in gray of the comparison image, but the test pattern TPa of the basic image and the test pattern TPb of the comparison image It is possible to check the amount of deviation in advance by the method of calculating the amount of deviation by the matching processing block 14 described above. Here, a method of calculating the deviation amount using the test patterns TPa and TPb will be described with reference to FIGS.

  FIG. 24 is an enlarged view when the region Aa shown in FIG. 22 and the region Ab shown in FIG. 23 are overlapped. Ideally, one pixel Pia in the area Aa and one pixel Pib in the area Ab overlap each other. However, the pixels Pia and Pib may be shifted as shown in FIG. The main cause of the shift is due to an assembly error of the image processing apparatus 1 to the image pickup apparatus 1, and a deviation amount of 3 pixels is assumed originally. In addition, when the user uses the image pickup apparatus 1, the amount of shift may be larger than that of three pixels due to an accidental drop, and thus pattern matching needs to be performed in a wider range. Furthermore, as shown in FIG. 18, in the case of shooting with a 4K image, not only the pixels are four times that of an FHD image, but also considering the amount of displacement, pattern matching must be performed over a wider range of pixels. I must. On the other hand, in the present embodiment, in the assembly process of the photographing apparatus 1, as shown in FIGS. 21 to 23, the test pattern TP (TPa, TPb) is photographed and stored in the external storage device 50. By using the pattern matching process, the matching area can be reduced as shown in FIG.

  FIG. 25 is a diagram in which a matching area according to the present embodiment and the prior art is set. Here, the reference image that is the hemispherical image A is set to Xa × Ya size, while the comparative image that is the hemispherical image B is set to Xb × Yb size. In FIG. 25, one divided area width W2, which is a region required this time, can be made smaller than one divided area width W1, which is a region previously required for template matching.

<< Effects of Embodiment >>
As described above, according to the present embodiment, the compression of the memory band can be reduced. Further, as shown in FIG. 25, the matching processing block 14 does not need to read the useless areas W3 and W4 on both sides of the area width W1 excluding the area width W2 from the external storage device. There is an effect that the processing speed of template matching is improved.

DESCRIPTION OF SYMBOLS 1 Image pick-up device 10 Image processing device 14 Matching processing block 15 Distortion correction block 24 1st distortion correction part 25 Pattern matching part 50 External storage device

JP, 2014-120878, A Japanese Patent Laid-Open No. 2006-027744

Claims (6)

An image processing apparatus for joining a plurality of images represented by first and second image data,
A first distortion correction unit that performs a process of correcting image distortion for the first and second image data;
The first image related to the first image data processed by the first distortion correction unit and the second image data related to the second image data processed by the first distortion correction unit. Pattern matching for performing a pattern matching process on a reference image indicating a reference area of the first image with respect to a comparison image indicating a comparison area of the second image in an overlapping area when the images are joined together And
A shift amount calculation unit that calculates a shift amount of the overlapping region by pattern matching processing by the pattern matching unit;
A parameter creating unit that creates a distortion correction parameter for correcting distortion of at least one of the first image and the second image according to the amount of deviation;
A second distortion correction unit that performs a process of correcting image distortion on at least one of the first and second image data using the distortion correction parameter generated by the parameter generation unit;
Have
The shift amount calculation unit performs the pattern matching using a test pattern used for the pattern matching process at least in the overlapping region of the first image and the second image. .   The image processing apparatus according to claim 1, wherein the test pattern includes a plurality of radial straight lines from the same center on concentric circles.   The image processing apparatus according to claim 1, wherein the test pattern is stored in a storage unit in the imaging apparatus when the image processing apparatus is assembled into a predetermined imaging apparatus.   The first and second images are hemispherical images, and the image created after the correction by the second distortion correction is an equirectangular projection image that is the source of the omnidirectional image. The image processing apparatus according to any one of claims 1 to 3. An image processing apparatus according to any one of claims 1 to 4,
A first imaging unit that sends the first image data obtained by photographing a subject to the image processing device;
A second imaging unit for sending the second image data obtained by photographing a subject to the image processing device;
A photographing apparatus having An image processing method for joining a plurality of images expressed by first and second image data,
A first distortion correction step for performing a process of correcting image distortion for each of the first and second image data;
The first image related to the first image data processed by the first distortion correction step and the second image data related to the second image data processed by the first distortion correction step. By performing a pattern matching process on the reference image indicating the reference region of the first image with respect to the comparison image indicating the comparison region of the second image in the overlapping region when connecting the images. , A deviation amount calculating step for calculating a deviation amount of the overlapping area;
A parameter creating step for creating a distortion correction parameter for correcting the distortion of at least one of the first image and the second image in accordance with the shift amount;
A second distortion correction step for performing a process of correcting image distortion for at least one of the first and second image data using the distortion correction parameter created by the parameter creation step;
Run
The shift amount calculating step includes a process of performing the pattern matching using a test pattern used for the pattern matching process in at least the overlapping region of the first image and the second image. Image processing device.