JP4324271B2 - Image processing apparatus and method - Google Patents

Description

とおき、Ｎ点の３次元座標（Ｘi，Ｙi，Ｚi）（ｉ＝１，…，Ｎ）を用いて、上記平面方程式の４つのパラメータ（ａ，ｂ，ｃ，ｄ）を、次式で示す最小自乗法により算出する。
【００５０】
【数２】

画像歪み補正手段１１は、平面計測手段１０が出力した被写体面の向きを表す３つのパラメータ（ａ，ｂ，ｃ）を用いて、被写体面を正面から撮影した画像に変換することにより、斜め撮影による画像の歪みを補正する。本実施の形態では図４のように、平面計測手段１０が計測した被写体面に平行な平面を被投影面として、該被投影面に向かって歪んだ被写体像を持つ画像を投影することにより、画像の歪みを補正する。このようにして歪みが補正された画像を、以下歪み補正画像と呼ぶ。以下、歪み補正画像を作成する方法について、詳細に説明する。
【００５１】
まず、画像座標（ｘ，ｙ）に対応する視線の向きを定める。撮像手段２の光学系は図５のように、ｘ軸：画像面右向きを正ｙ軸：画像面下向きを正ｚ軸：光軸方向；対象に向かう向きを正原点０：撮像手段２の光学中心ｆ：焦点距離とする中心射影モデルであると仮定し、以後このｘｙｚ座標系を装置座標系と呼ぶ。すると、（ｘ，ｙ）に対応する視線の向きは、装置座標系を基準にした次の３次元ベクトルＰで表すことができる。
【００５２】
【数３】

次に図６のように、装置座標系のz軸を、被投影面の単位法線ベクトルに変換するような回転行列Ｒを求める。この場合、次の関係式が成立する。
【００５３】
【数４】

(4)式を満たす回転行列Ｒは多数存在するが、ここでは回転行列Ｒを次式のように定義する。
【００５４】
【数５】

これは図７のように、以下の順序で装置座標系を回転させることに相当する。
(1) 装置座標系を、ｙ軸回りにβだけ回転する。この回転によりできる座標系をｘ1ｙ1ｚ1座標系とする。
(2) 装置座標系を、ｘ1軸回りにαだけ回転するこの時、ａ2＋ｂ2＋ｃ2＝１（ｃ＞０）、及び(4)〜(6)式を用いると、回転角α，βは次式のように導出される。
【００５５】
【数６】

以上で求められた回転角α，βを(5)式に代入することにより、Ｒを一意に定めることができる。
【００５６】
最後に、これまでの手順で求められた３次元ベクトルｐ、回転行列Ｒ、及び被写体面の向きを表す３つのパラメータ（ａ，ｂ，ｃ）を用いて、画像座標（ｘ，ｙ）を被投影面上に座標変換する。すなわち図４において、画像面の１点に対応する３次元ベクトルＰを延長した時に被投影面と交差する点ｐを、座標変換後の画像座標とする。装置座標系を基準とした、点ｐに対応する３次元ベクトルをＰとすると、Ｐは次式で表される。
【００５７】
【数７】

ここでｋは、撮像手段２の光学中心０から被投影面までの距離を表す拡大係数であるが、前述したように該被投影面に向かって画像を投影することにより歪みを補正するので、ｋは作成される歪み補正画像の大きさを表している。また、Ｐは点ｐを画像撮影時の装置座標系を基準に表したベクトルであるが、Ｐを回転行列Ｒを用いて次式のように座標変換することにより、画像処理装置１を被写体面と正対させた時の３次元ベクトルＰ´に変換する。
【００５８】
【数８】

したがって、(10)式の３次元ベクトルＰ´のｘ座標、ｙ座標を、点（ｘ，ｙ）の座標変換後の画像座標とすることにより、補正変換後の画像座標（ｘ，ｙ）を得る。整理すると次式のようになる。
【００５９】
【数９】

以上の手順によって、(11)式を用いて画像面上の任意の座標（ｘ，ｙ）を（Ｘ，Ｙ）に座標変換することにより、歪み補正画像を作成することができる。歪み補正画像の画像座標（Ｘ，Ｙ）に対応する変換前の画像座標（ｘ，ｙ）は一般に整数にならないが、双一次補間法やB-スプライン補間法などの既存の方法を用いて補間することにより、画像を作成すればよい。
【００６０】
上記したような画像処理の方法について、図８のフローチャートにより説明する。まず、撮像手段２により被写体面を撮影すると同時に、撮影時における被写体面の向きを表す３つのパラメータを求める（ステップS1）。次に、得られた（ａ，ｂ，ｃ）を基に、先に説明した手順で回転行列Ｒを算出する（ステップS2）。以上の手順で求めた（ａ，ｂ，ｃ）及び回転行列Ｒを基に、(11)式の座標変換を施すことにより歪み補正画像を作成し（ステップS3）、処理を終了する。
【００６１】
［発明の実施の形態２］以下に説明する発明の実施の形態２〜４では、図９のように、画像処理装置１を用いて、異なる場所から同一の被写体面の静止画像を、画像の一部が重複するように、視点1から視点KまでのK箇所から撮影した場合の動作について説明する。視点1で撮影した画像をIm 1、視点2で撮影した画像をIm2などと呼ぶことにする。また、説明を簡単にするため、Im 1とIm 2、Im 2とIm 3のように、隣り合う順序で撮影したImjとIm j+1との間で重複領域を持つと仮定する。
【００６２】
図１０は、この発明の実施の形態２に係る画像処理装置１の構成図である。図１０において、図２と同一符号の部材は、発明の実施の形態１と同様の部材であるので、詳細な説明は省略する。
【００６３】
画像処理装置１は、互いに重複する領域を持つ画像間の特徴点及び対応点を抽出する対応検出手段３１と、対応検出手段３１が出力した特徴点と対応点の関係を基に、座標変換により互いに重複する領域を持つ画像を貼り合わせる画像貼り合わせ手段３２と、画像貼り合わせ手段３２が作成した貼り合わせ画像を記憶する貼り合わせ画像記憶手段３３と、平面計測手段１０の出力した前記被写体面の向きを基に、貼り合わせ画像記憶手段３３に蓄積された貼り合わせ画像の斜め撮影による歪みを補正する合成画像歪み補正手段３４とを有している。
【００６４】
対応検出手段３１、画像貼り合わせ手段３２、貼り合わせ画像記憶手段３３、及び合成画像歪み補正手段３４の構成・動作を、以下説明する。対応検出手段３１は、互いに重複した領域を持つ２枚の画像において同一の部分を撮影した箇所を検出するものである。ここでは、相関演算を用いた方法について説明する。対応検出手段３１は、図１１のように、特徴点設定手段３５、相関演算手段３６より構成される。図１０のフレームメモリ７には、先に撮影した画像（基準画像）と、基準画像と重複した領域を持つ参照画像が蓄積されている。特徴点設定手段３５は基準画像において、特徴点の位置を決定し、特徴点を中心とする濃淡パターンを抽出し、相関窓を作成する。特徴点の位置は、角（corner）のように画像の濃度パターンが特徴的である箇所を抽出することにより決定される。相関演算手段３６は、基準画像で作成した相関窓の濃淡パターンとほぼ一致する箇所を、参照画像において相関演算により検出し、これを対応点と決定する。ここで、相関演算によるブロックマッチングにより対応点を検出する一例について説明する。図１２のように、（２Ｎ＋１）（２Ｐ＋１）の相関窓を用いたブロックマッチングで、基準画像におけるｉ番目の特徴点（ｘi0，ｙi0）と、参照画像における点（ｘi0＋ｄｘi，ｙi0＋ｄｙi）の相互相関値Ｓiは、次式により計算される。
【００６５】
【数１０】

各特徴点に対して、相互相関値Ｓiの最大値が予め定められた閾値以上である点を求めることにより、参照画像における対応点（ｘi0＋ｄｘi，ｙi0＋ｄｙi）が求められる。Ｓiの最大値が閾値以下ならば、対応点は存在しないとする。
【００６６】
画像貼り合わせ手段３２は、対応検出手段３１より出力された特徴点と対応点との関係を基に、基準画像を参照画像に座標変換することにより画像を貼り合わせる。ここでは、座標変換に射影変換を用いると仮定して説明する。射影変換とは、図１３のようにある被写体平面を２枚撮影し、片方を基準画像、もう片方を参照画像とした時に、図１４のように基準画像に写った被写体像を参照画像撮影時の見え方に変換することを指す。これを数式で表すと、基準画像における特徴点（ｘs，ｙs）と参照画像における対応点（ｘr，ｙr）の間に、次式の関係が成立する。
【００６７】
【数１１】

(13)式においては８つの未知パラメータが存在するので、特徴点と対応点の組が４つ以上得られれば、最小自乗法等の計算法により係数ｂ1〜ｂ8を求めることができる。以上の手順で、係数ｂ1〜ｂ8を計算することにより貼り合わせた画像を、射影変換画像と呼ぶことにする。
以上では２視点から撮影した場合の射影変換画像の作成法を説明したが、反復処理により射影変換画像を順次作成して３枚以上の画像を貼り合わせることができる。以下、Im 1〜Im j-1を、Im jに対して射影変換した画像を、PIm j（ 1≦j≦K 、但しPIm 1 = Im 1とする）と呼ぶことにする。PIm jにおいて、Im 1〜Imj-1に写った被写体像はそれぞれ、Imj撮影時の見え方に変換されて、Im jに貼り付けられている。PIm jを作成する反復処理を、以下に示す。
【００６８】
j= 2, … , Kについて、以下の処理を繰り返す。
（ｉ）PIm j-1を基準画像、Im jを参照画像とする。
（ｉｉ）基準画像において特徴点を抽出し、参照画像において対応点を検出する。
（ｉｉｉ）（ｉｉ）で検出した4組以上の特徴点と対応点を用いて、(13)式の係数ｂ1〜ｂ8を求め、PIm j-1をIm jに対して射影変換してPIm jを作成する。
（ｉｖ）jを１つ加算して、（ｉ）に戻る。
【００６９】
以上の反復処理によって、最終的にはIm 1〜Im K-1をIm Kに射影変換した画像PIm Kが得られる、すなわちIm 1〜Im Kから１枚の貼り合わせ画像を作成することができる。こうして作成された貼り合わせ画像は、貼り合わせ画像記憶手段３３に保存される。
【００７０】
合成画像歪み補正手段３４は、前記画像貼り合わせ手段３２が作成した貼り合わせ画像PIm Kにおける歪みを補正するものである。画像貼り合わせ手段３２が作成した貼り合わせ画像PIm Kは既に説明したように、参照画像撮影時の見え方に変換して貼り合わされた画像なので、参照画像において被写体像が歪んでいた場合、当然貼り合わせ画像も歪みを含んでいる。そこで図１５において、参照画像（＝Im K）を撮影した時に、前記平面計測手段１０の出力した被写体面の向きを表す３つのパラメータ（ａ，ｂ，ｃ）を用いて、発明の実施の形態１と同様に、(3)式〜(11)式の処理によって、貼り合わせ画像PIm Kの歪みを補正する。
【００７１】
上記したような画像処理の方法について、図１６のフローチャートにより説明する。まず、撮像手段２により被写体面を撮影し、この画像を基準画像として記憶すると同時に、撮影時における被写体面の向きを表す３つのパラメータ（ａ，ｂ，ｃ）を求める（ステップS11）。続いて、基準画像と一部が重複するように視点及び画像処理装置１の姿勢を変化させて撮像手段２により被写体面を撮影し、この画像を参照画像として記憶すると同時に、撮影時における被写体面の向きを表す３つのパラメータ（ａ，ｂ，ｃ）を求める（ステップS12）。基準画像において特徴点を抽出すると共に、参照画像において該特徴点と同じ箇所を示す対応点を検出する（ステップS13）。こうして得られた特徴点と対応点の関係を基に、先に説明した方法で基準画像を参照画像に対して座標変換した画像を作成する（ステップS14）。その後、さらに画像を撮影して合成する画像を増やす場合には（ステップS15）、以上の処理で作成した貼り合わせ画像を基準画像に置き換えて（ステップS16）、ステップS2からステップS14の動作を繰り返す。そうでない場合には、最後に作成した貼り合わせ画像を、参照画像撮影時の被写体面のパラメータ（ａ，ｂ，ｃ）を用いて、(11)式の座標変換を行うことにより貼り合わせ画像の歪みを補正して（ステップS17）、処理を終了する。
【００７２】
［発明の実施の形態３］図１７は、この発明の実施の形態３である画像処理装置１の構成図である。図１７において、図２と同一符号の部材は、発明の実施の形態１と同様の部材であるので、詳細な説明は省略する。
【００７３】
画像処理装置１は、平面計測手段１０の出力した被写体面の向きを基に、各々の画像の斜め撮影による歪みを補正する画像歪み補正手段４１を備えている。
【００７４】
画像処理装置１による処理の手順を中心に、構成・動作を説明する。平面計測手段１０は、各々の画像を撮影する時の、画像処理装置１に対する被写体面の向きを計測する。画像歪み補正手段４１は、前記平面計測手段１０が出力した被写体面の向き（ａ，ｂ，ｃ）を用いて、先に説明した手順で、各画像の斜め撮影による歪みを補正する。対応検出手段３１は、前記画像歪み補正手段４１で歪みを補正され、互いに重複した領域を持つ２枚の画像において、同一の部分を撮影した箇所を検出する。例えば、前述した相関演算を用いて対応付けを行う場合、本実施の形態では斜め撮影による歪みを補正した画像間の相関をとるので、さらに容易かつ精度の高い対応付けが可能となる。以上で得られた歪みを補正した基準画像、参照画像および特徴点と対応点の関係により、(13)式の係数ｂ1〜ｂ8を求めて基準画像を参照画像に貼り合わせる。
【００７５】
上記したような画像処理の方法について、図１８のフローチャートにより説明する。まず、撮像手段２により被写体面を撮影し、この画像を基準画像として記憶すると同時に、撮影時における被写体面の向きを表す３つのパラメータ（ａ，ｂ，ｃ）を求め（ステップS21）、このパラメータ（ａ，ｂ，ｃ）を用いて、(3)〜(11)式の処理手順で基準画像における歪みを補正し、基準画像を歪みを補正したものに置き換える（ステップS22）。続いて、基準画像と一部が重複するように視点及び画像処理装置１の姿勢を変化させて、撮像手段２により被写体面を撮影し、この画像を参照画像として記憶すると同時に、撮影時における被写体面の向きを表す３つのパラメータ（ａ，ｂ，ｃ）を求め（ステップS23）、このパラメータ（ａ，ｂ，ｃ）を用いて、(3)〜(11)式の処理手順で参照画像における歪みを補正し、参照画像を歪みを補正したものに置き換える（ステップS24）。基準画像において特徴点を抽出すると共に、参照画像において該特徴点と同じ箇所を示す対応点を検出する（ステップS25）。こうして得られた特徴点と対応点の関係を基に、先に説明した方法で基準画像を参照画像に対して座標変換した画像を作成する（ステップS26）。その後、さらに画像を撮影して合成する画像を増やす場合には（ステップS27）、以上の処理で作成した貼り合わせ画像を基準画像に置き換えて（ステップS28）、ステップS23からステップS26の動作を繰り返す。そうでない場合には、処理を終了する。
【００７６】
なお本実施の形態においては、斜め撮影による画像の歪みを補正した画像同士を貼り合わせるので、(13)式で表される射影変換の代わりに、次式のアフィン変換を用いても貼り合わせ画像を得ることができる。
【００７７】
【数１２】

画像貼り合わせ手段３２が、(13)式の射影変換または(14)式のアフィン変換に基づいた構成・動作に限定されないことは言うまでもない。
【００７８】
［発明の実施の形態４］この発明の実施の形態４は、発明の実施の形態１〜３において、被写体面までの距離を利用して、作成される歪み補正画像の大きさを、各被写体像の大きさが概ね等しくなるように調整することに関するものである。例えば、図１９(a)のように基準画像と参照画像を、被写体面までの距離や撮像手段２の光軸（z軸）の向きが異なった状態で撮影すると、図１９(b)のようにそれぞれの歪み補正画像における被写体像の大きさも異なる場合がある。そこで、平面計測手段１０の出力を利用して歪み補正画像の大きさを調整するように、画像歪み補正手段４１を構成する。その一例を以下説明する。
【００７９】
平面計測手段１０は、発明の実施の形態１と同様の構成をとり、被写体面の方程式を表すパラメータ（ａ，ｂ，ｃ，ｄ）、及び装置座標系を基準にした各点の３次元座標を出力する。画像歪み補正手段４１は、前記平面計測手段１０が出力した（ａ，ｂ，ｃ，ｄ）及び各点の３次元座標より、(3)〜(11)式の手順で各画像の歪みを補正すると共に、(11)式の座標変換に用いる最適な拡大係数kを算出する。例えば、図１９(a)において、基準画像撮影時に平面計測手段１０が出力した被写体面のパラメータを（ａ，ｂ，ｃ，ｄ）、また参照画像撮影時に平面計測手段１０が出力した被写体面のパラメータを（ａ´，ｂ´，ｃ´，ｄ´）とする。基準画像、参照画像の歪み補正画像作成に用いる拡大係数をそれぞれk、k´(k＞０，k´＞０)とおくと、
【００８０】
【数１３】

とすれば、図１９(b)のように各々の被写体像の大きさが概ね等しい歪み補正画像が得られる。
【００８１】
［発明の実施の形態５］この発明の実施の形態５は、発明の実施の形態１〜４において、撮影時における被写体面の向きを表す３つのパラメータ（ａ，ｂ，ｃ）を用いて画像処理装置１を被写体面に正対させて撮影した画像に変換することにより、各々の画像の歪みを補正するが、これは撮像手段２の光軸（z軸）を被写体面の向きと一致させるような座標変換を施した結果得られる画像を擬似的に生成することを意味する。しかし、この座標変換は被写体面の法線ベクトルを回転軸とする１軸回りの自由度を残している。すなわち、撮影時の画像処理装置１の姿勢によっては、図２０(a)のように、画像面上で被写体像が傾いた歪み補正画像が作成される。本実施の形態では、発明の実施の形態１〜４に傾き補正手段５１を設置し、図２０(b)のように傾きの殆ど無い歪み補正画像を得ることを目的とする。
【００８２】
傾き補正手段５１は、図２１のように、回転角設定手段５２、回転中心設定手段５３、及び回転演算手段５４より構成される。回転角設定手段５２と回転中心設定手段５３はスイッチ等により構成され、例えば画像処理装置１の筐体上に表示手段８と隣り合う位置に配置される。回転角設定手段５２は画面上での回転角θを、また回転中心設定手段５３は画面上での回転中心座標（ｘ0，ｙ0）を指定する。回転演算手段５４は筐体内部に内蔵された演算回路等により構成され、回転角θ及び回転中心座標（ｘ0，ｙ0）を用いて、以下の座標変換式により歪み補正画像の画面上での傾きを補正する。
【００８３】
【数１４】

(16)式の座標変換を、歪み補正画像に適用して傾きを補正した画像を、表示手段８上で確認することができる。
【００８４】
なお、いうまでもなく、前記の各発明の実施の形態の説明は、本発明の一例を説明したにすぎず、この発明を前記の各発明の実施の形態通りに限定・縮小すべきではない。例えば、平面計測手段１０の構成は前記実施の形態で説明した範囲に限定されず、波動（光、電波、音波）を対象に照射し、対象からの反射波の伝播時間や位相差を計測することにより対象までの距離を求める、また被写体面の焦点ぼけの量から距離を求めるなど、別の構成でも構わない。また、対応検出手段３１において相関法による濃度マッチングにより対応点を検出すると説明したが、時空間微分法など別の手法で行っても構わない。
【発明の効果】
【００８５】
請求項１に記載の発明によれば、任意の被写体面を任意の視点・姿勢で撮影した場合でも、平面を計測することにより、簡便に画像の歪みを補正することができる。
【００８６】
請求項２に記載の発明によれば、任意の被写体面を任意の視点・姿勢で撮影した複数の画像を貼り合わせた画像を、計測した被写体面の向きを用いて座標変換することにより、簡便に歪みのない合成画像を作成することができる。
【００８７】
請求項３に記載の発明によれば、任意の被写体面を任意の視点・姿勢で撮影した場合でも、平面を計測することにより、簡便に各画像の歪みを補正することができる。また、歪みを補正した画像を用いて対応を検出するので、請求項２に記載の画像処理装置と比較して、さらに対応付けの精度が上がり、その結果、一層高精細な合成画像を簡便に得ることができる。
【００８８】
請求項４に記載の発明によれば、請求項１乃至３のいずれか１に記載の画像処理装置において、撮像する手段の光軸の被写体面の法線ベクトルに対する回転角を算出することにより、簡便に画像の歪みを補正することができる。
【００８９】
請求項５に記載の発明によれば、請求項１乃至３のいずれか１に記載の画像処理装置において、撮像された画像を被写体面の法線ベクトルの方向から撮像された画像に補正することにより、簡便に画像の歪みを補正することができる。
【００９１】
請求項６に記載の発明によれば、請求項１乃至５のいずれか１に記載の画像処理装置において、撮影した各々の画像や貼り合わせた画像における被写体像の歪み・傾きを補正することができる。また、請求項３に記載の発明に本請求項に記載の発明を適用すれば、被写体像の傾きがほぼ等しい歪み補正画像を用いて対応を検出するので、さらに対応付けの精度が上がり、その結果、一層高精細な合成画像を簡便に得ることができる。
【００９２】
請求項７に記載の発明によれば、任意の被写体面を任意の視点・姿勢で撮影した場合でも、平面を計測することにより、簡便に画像の歪みを補正することができる。
【００９３】
請求項８に記載の発明によれば、任意の被写体面を任意の視点・姿勢で撮影した複数の画像を貼り合わせた画像を、計測した被写体面の向きを用いて座標変換することにより、簡便に歪みのない合成画像を作成することができる。
【００９４】
請求項９に記載の発明によれば、任意の被写体面を任意の視点・姿勢で撮影した場合でも、平面を計測することにより、簡便に各画像の歪みを補正することができる。また、歪みを補正した画像を用いて対応を検出するので、請求項８に記載の発明と比較して、さらに対応付けの精度が上がり、その結果、一層高精細な合成画像を簡便に得ることができる。
【００９５】
請求項１０に記載の発明によれば、請求項７乃至９のいずれか１に記載の画像処理方法において、撮像する手段の光軸の被写体面の法線ベクトルに対する回転角を算出することにより、簡便に画像の歪みを補正することができる。
【００９６】
請求項１１に記載の発明によれば、請求項７乃至９のいずれか１に記載の画像処理方法において、撮像された画像を前記被写体面の法線ベクトルの方向から撮像された画像に補正することにより、簡便に画像の歪みを補正することができる。
【００９８】
請求項１２に記載の発明によれば、請求項７乃至１１のいずれか１に記載の画像処理方法において、撮影した各々の画像や貼り合わせた画像における被写体像の歪み・傾きを補正することができる。また、請求項９に記載の発明に本請求項に記載の発明を適用すれば、被写体像の傾きがほぼ等しい歪み補正画像を用いて対応を検出するので、さらに対応付けの精度が上がり、その結果、一層高精細な合成画像を簡便に得ることができる。
【図面の簡単な説明】
【図１】被写体面を撮影した時に生じる画像の歪みの補正を説明する概念図である。
【図２】この発明の実施の形態１である画像処理装置の構成を示すブロック図である。
【図３】前記画像処理装置の平面計測手段の構成を示すブロック図である。
【図４】前記画像処理装置の動作を説明する概念図である。
【図５】前記画像処理装置の動作を説明する概念図である。
【図６】前記画像処理装置の動作を説明する概念図である。
【図７】前記画像処理装置の動作を説明する概念図である。
【図８】前記画像処理装置の動作を説明するフローチャートである。
【図９】画像処理装置を用いて、異なる場所から同一の被写体面の静止画像を、画像の一部が重複するように、視点1から視点KまでのK箇所から撮影する場合の概念図である。
【図１０】この発明の実施の形態２に係る画像処理装置の構成を説明するブロック図である。
【図１１】前記画像処理装置の対応検出手段の構成を説明するブロック図である。
【図１２】前記画像処理装置の動作を説明する概念図である。
【図１３】前記画像処理装置の動作を説明する概念図である。
【図１４】前記画像処理装置の動作を説明する概念図である。
【図１５】前記画像処理装置の動作を説明する概念図である。
【図１６】前記画像処理装置の動作を説明するフローチャートである。
【図１７】この発明の実施の形態３である画像処理装置の構成を説明するブロック図である。
【図１８】前記画像処理装置の動作を説明するフローチャートである。
【図１９】この発明の実施の形態４である画像処理装置の動作を説明する概念図である。
【図２０】この発明の実施の形態５である画像処理装置の動作を説明する概念図である。
【図２１】前記画像処理装置の動作を説明する概念図である。
【符号の説明】
１ 画像処理装置
２ 撮像手段
１０ 平面計測手段
１１ 画像歪み補正手段
３１ 対応検出手段
３２ 画像貼り合わせ手段
３４ 合成画像歪み補正手段
５１ 傾き補正手段[0001]
BACKGROUND OF THE INVENTION
[0002]
  The present invention relates to an image processing apparatus that is used in a digital still camera or the like and performs distortion correction by oblique photographing of an image obtained by photographing a subject surface, and a method thereof.
[Prior art]
[0003]
  With the rapid advancement of computer networks, the way of business has diversified, and there is a need to quickly acquire important information in all aspects. Along with this, there is an increasing demand for simple and highly accurate input of planar documents and photographs using a portable input device. At present, there is an application example in which image information obtained by photographing a subject surface such as a paper document having a size of about A4 is used by using an image input device such as a digital still camera.
[0004]
  However, in this application example, there is a problem that when the image is taken in a state where the imaging surface of the apparatus is not parallel to the paper surface, distortion due to the perspective effect occurs in the taken image.
[0005]
  On the other hand, in Japanese Patent Laid-Open No. 3-94383, a known fixed shape is assumed, an input image is arranged in the fixed shape, and the distortion of the fixed shape is examined, so that on the original plane. A technique is disclosed in which a conversion parameter having the following shape is discovered and image distortion caused by oblique photographing is corrected.
[0006]
  In Japanese Patent Application Laid-Open No. 5-101221, an orthogonal grid is introduced on the object plane, the spatial coordinates of each grid point are obtained, and the image is projected by performing orthogonal transformation on the coordinate system provided on the imaging plane. A technique for correcting distortion is disclosed.
[0007]
  In Japanese Patent Laid-Open No. 9-289600, an angle setting means for inputting an inclination angle formed by an imaging surface of a camera and a subject surface, and a distance measuring means for measuring a distance to the subject are provided. A technique is disclosed in which a subject is photographed with an angle input, and the subject is corrected to an image viewed from the front based on the tilt angle and the subject distance detected by the distance measuring means.
[0008]
  In Japanese Patent Laid-Open No. 10-79878, in a composite image obtained by connecting divided images obtained by periodically scanning a subject by scanning an imaging surface by two-dimensional coordinate transformation processing such as affine transformation, for each same connected component. Label and classify the connected components of the labeled image into lines, characters, and figures according to their sizes, evaluate the slope of the classified line or character base line, and maintain the distance between the circumscribed rectangle of the character and the line Thus, there is disclosed a still image capturing apparatus that obtains a high-resolution image by rotating and correcting a straight line by the evaluated inclination.
[Problems to be solved by the invention]
[0009]
  However, in the technique disclosed in Japanese Patent Application Laid-Open No. 3-94383, it is necessary to arrange the photographing object in a fixed figure whose shape is known, and a troublesome work is necessary for photographing the paper surface.
[0010]
  In the technique disclosed in Japanese Patent Laid-Open No. 5-101221, it is necessary to superimpose an orthogonal grid on the subject surface and manually input the two-dimensional coordinates of each lattice point. It is extremely difficult to shoot.
[0011]
  Furthermore, in the technique disclosed in Japanese Patent Laid-Open No. 9-289600, it is very difficult to manually input the inclination angle formed between the imaging surface and the subject surface by operating the angle setting means, and as a result, high accuracy. Therefore, it becomes difficult to correct image distortion.
[0012]
  In addition, the application example has a problem that it is extremely difficult to read information on a large area such as newspaper or information drawn on a panel or a wall with a high resolution using the above-described apparatus.
[0013]
  Therefore, there is a technique in which these large subjects are divided and photographed with a portable image input device, and a plurality of obtained images are combined to create a single composite image to obtain a high-resolution image. . An example thereof is the technique disclosed in Japanese Patent Laid-Open No. 10-79878.
[0014]
  However, since the posture of the image pickup apparatus with respect to the document is generally different when each divided image is picked up, the inclination of the straight line or the character base line varies depending on the position of the composite image on the screen. Therefore, it is necessary to perform the labeling, the inclination evaluation, and the rotation correction everywhere in the composite image, and the processing becomes very complicated.
[0015]
  An object of the present invention is to correct distortion of an image shot in a state where an imaging surface and a subject surface are inclined with respect to each other, and to easily create a composite image with corrected distortion from these images.
[Means for Solving the Problems]
[0016]
  The invention according to claim 1 includes means for capturing the subject surface as an image;When imaging with the imaging means, illuminate the subject surface with light,Based on the direction of reflected light from the subject surface,Of 3 or more pointsMeans for calculating a three-dimensional coordinate based on the means for imaging;By means of calculatingBased on the calculated 3D coordinates of each point,For the imaging meansMeans for determining the orientation of the subject surface;Determined by the means for determiningBased on the orientation of the subject surface,The captured image is changed to an image captured from the front of the subject surface.Convert distortion of the captured imageAnd an image processing apparatus.
[0017]
  Therefore, even when an arbitrary subject surface is photographed from an arbitrary viewpoint / posture, image distortion can be easily corrected by measuring the plane.
[0018]
  The invention according to claim 2 is the first imaging.CriteriaPlace the subject surface so that part of the image overlapsAs a reference imageMeans for imaging;When imaging with the imaging means,Irradiate light onto the subject surface, and based on the direction of reflected light from the subject surface,Of 3 or more pointsMeans for calculating a three-dimensional coordinate based on the means for imaging;By means of calculatingBased on the calculated 3D coordinates of each point,For the imaging meansMeans for determining the orientation of the subject surface; and the reference imageFromA plurality of feature points are extracted, and the reference imageFromBased on correspondence detection means for detecting corresponding points corresponding to the feature points, and feature points and corresponding points detected by the correspondence detection means, the base image is subjected to coordinate conversion on the reference image.The reference image and the reference image;Means for pasting together,With the means to findObtained reference imageThe imaging means for imagingBased on the orientation of the subject surface,The pasted image is an image taken from the front of the subject surface.Convert distortion of the captured imageAnd an image processing apparatus.
[0019]
  Therefore, a composite image can be easily created without distortion by coordinate-transforming an image obtained by pasting together a plurality of images obtained by photographing an arbitrary subject surface from an arbitrary viewpoint and posture using the measured orientation of the subject surface. be able to.
[0020]
  The invention according to claim 3 was imaged first.StandardPlace the subject surface so that part of the image overlapsAs a reference imageMeans for imaging;When imaging with the imaging means, the subject surface is irradiated with light, and based on the direction of reflected light from the subject surface,On the subject surfaceOf 3 or more pointsMeans for calculating a three-dimensional coordinate based on the means for imaging;By means of calculatingBased on the calculated 3D coordinates of each point,For the imaging meansMeans for determining the orientation of the subject surface;With the means to findBased on the obtained orientation of the subject surface,The captured image is changed to an image captured from the front of the subject surface.Convert distortion of the captured imageMeans for correcting, andBy means of correctioncorrectionWasThe reference imageFromExtract multiple feature points,Corrected by the correcting meansReference imageFromBased on correspondence detection means for detecting corresponding points corresponding to the feature points, the feature points detected by the correspondence detection means, and the corresponding points,CorrectedThe reference image isCorrectedAn image processing apparatus comprising: means for pasting images by converting coordinates on a reference image.
[0021]
  Therefore, even when an arbitrary subject surface is photographed from an arbitrary viewpoint / posture, distortion of each image can be easily corrected by measuring the plane.
[0022]
  The invention according to claim 4 is the invention according to claim 1.The image processing apparatus according to any one of 1 to 3InThe means for measuring the orientation of the subject plane calculates a rotation angle of the optical axis of the imaging means with respect to the normal vector of the subject plane.
[0023]
  Therefore,By calculating the rotation angle of the optical axis of the imaging means with respect to the normal vector of the subject surface, image distortion can be easily corrected.
[0024]
  The invention according to claim 5 is the invention according to claim 1.The image processing apparatus according to any one of 1 to 3InThe correcting means corrects the captured image to an image captured from the direction of the normal vector of the subject surface.
[0025]
  Therefore,By correcting the captured image to an image captured from the direction of the normal vector of the subject surface, it is possible to easily correct image distortion.
[0028]
  Claim6The invention described in claim 1 to claim 15The image processing apparatus according to any one of the above, further includes an inclination correction unit that corrects an inclination on the screen of the image corrected by the correction unit.
[0029]
  Therefore,It is possible to correct the distortion and inclination of the subject image in each photographed image and the pasted image.
[0030]
  Claim7The invention described in (1), the step of imaging the subject surface as an image by means of imaging,When imaging in the imaging step, illuminate the subject surface with light,Based on the direction of reflected light from the subject surface,Of 3 or more pointsCalculating three-dimensional coordinates based on the imaging means; andIn the process of calculatingBased on the calculated 3D coordinates of each point,For the imaging meansDetermining the orientation of the subject surface;Determined in the determining stepBased on the orientation of the subject surface,The captured image is changed to an image captured from the front of the subject surface.Convert distortion of the captured imageAnd a correcting step. An image processing method comprising:
[0031]
Therefore, even when an arbitrary subject surface is photographed from an arbitrary viewpoint / posture, image distortion can be easily corrected by measuring the plane.
[0032]
  Claim8According to the invention described in (1), the imaging is first performed by the imaging means.CriteriaPlace the subject surface so that part of the image overlapsAs a reference imageImaging step;When imaging in the imaging step,Irradiate light onto the subject surface, and based on the direction of reflected light from the subject surface,Of 3 or more pointsCalculating a three-dimensional coordinate based on the imaging step;In the process of calculatingBased on the calculated 3D coordinates of each point,For the imaging meansObtaining the orientation of the subject surface; and the reference imageFromA plurality of feature points are extracted, and the reference imageFromBased on the correspondence detecting step for detecting the corresponding point corresponding to the feature point, and the feature point and the corresponding point detected by the correspondence detecting step, the base image is coordinate-transformed on the reference image.The reference image and the reference image;A process of pasting together,In the step of findingObtained reference imageThe imaging means for imagingBased on the orientation of the subject surface,The pasted image is an image taken from the front of the subject surface.Convert distortion of the captured imageAnd a correcting step.
[0033]
Therefore, a composite image can be easily created without distortion by coordinate-transforming an image obtained by pasting together a plurality of images obtained by photographing an arbitrary subject surface from an arbitrary viewpoint and posture using the measured orientation of the subject surface. be able to.
[0034]
  Claim9In the invention described in the above, the image is taken first by the means for taking an image.StandardPlace the subject surface so that part of the image overlapsAs a reference imageImaging step;When imaging in the imaging step, the subject surface is irradiated with light, and based on the direction of reflected light from the subject surface,On the subject surfaceOf 3 or more pointsCalculating three-dimensional coordinates based on the imaging means; andIn the process of calculatingBased on the calculated 3D coordinates of each point,For the imaging meansDetermining the orientation of the subject surface;In the step of findingBased on the obtained orientation of the subject surface,The captured image is changed to an image captured from the front of the subject surface.Convert distortion of the captured imageCorrecting, andIn the process of correctingcorrectionWasThe reference imageFromExtract multiple feature points,Corrected by the correcting meansReference imageFromBased on the correspondence detection step for detecting the corresponding point corresponding to the feature point, the feature point and the corresponding point detected by the correspondence detection step,CorrectedThe reference image isCorrectedAnd a step of pasting the images together by converting the coordinates on the reference image.
[0035]
Therefore, even when an arbitrary subject surface is photographed from an arbitrary viewpoint / posture, distortion of each image can be easily corrected by measuring the plane.
[0036]
  Claim10The invention described in claim7Thru9In the image processing method according to any one of the above, the step of determining the orientation of the subject surface calculates a rotation angle of the optical axis of the imaging unit with respect to a normal vector of the subject surface.
[0037]
Therefore, by calculating the rotation angle of the optical axis of the imaging means with respect to the normal vector of the subject surface, image distortion can be easily corrected.
[0038]
  Claim11The invention described in claim7Thru9In the image processing method according to any one of the above, the correcting step corrects the captured image to an image captured from the direction of the normal vector of the subject surface.
[0039]
Therefore, by correcting the captured image to an image captured from the direction of the normal vector of the subject surface, it is possible to easily correct image distortion.
[0042]
  Claim12The invention described in claim7Thru11The image processing method according to any one of the above, further includes an inclination correction step of correcting an inclination on the screen of the image corrected by the correction step.
[0043]
Therefore, it is possible to correct the distortion and inclination of the subject image in each captured image and the combined image.
DETAILED DESCRIPTION OF THE INVENTION
[0044]
  [Embodiment 1] The configuration and operation of an image processing apparatus 1 for correcting image distortion that occurs when a subject surface as shown in FIG. 1 is photographed will be described below. FIG. 2 is a block diagram showing the configuration of the image processing apparatus 1 according to the first embodiment of the present invention.
[0045]
  That is, the image processing apparatus 1 includes an imaging unit 2, a signal processing unit 3 that processes an image signal obtained by the imaging unit 2, a memory control unit 4 to which an output of the signal processing unit 3 is supplied, and a main control unit 5. And an interface (hereinafter abbreviated as I / F) 6, a frame memory 7 for accumulating image signals according to instructions from the memory control means 4, display means 8 for displaying image signals via the I / F 6, and I / F 6 And external storage means 9 for reading and writing various signals including image signals. In addition, a plane measuring unit 10 that measures the orientation of the subject surface with respect to the imaging unit 2 and an image that corrects distortion caused by oblique shooting in the captured image based on the orientation of the subject surface output by the plane measuring unit 10. It also has distortion correction means 11.
[0046]
  Details of each block will be described below. The imaging unit 2 includes a lens 12, a diaphragm 13, a shutter 14, a photoelectric conversion element 15, and a preprocessing unit 16. For example, a CCD (charge coupled device) is used for the photoelectric conversion element 15. The preprocessing means 16 includes analog signal processing such as a preamplifier and AGC (auto gain controll) and an analog-digital converter (hereinafter abbreviated as A / D converter), and the analog output from the photoelectric conversion element 15. After preprocessing such as amplification and clamping on the video signal, the analog video signal is converted into a digital video signal.
[0047]
  The signal processing means 3 is constituted by a digital signal processor (DSP processor) or the like, and performs various image processing such as color separation, white balance adjustment, and γ correction on the digital video signal obtained from the imaging means 2. Apply. The memory control means 4 stores the image signal processed in this way in the frame memory 7, and conversely reads out the image signal stored in the frame memory 7. The frame memory 7 can store at least one image, and generally VRAM, SRAM, DRAM or the like is used. The recording of the image signal read from the frame memory 7 is performed by performing signal processing such as image compression on the image signal in the signal processing unit 3 and then storing it in the external storage unit 9 via the I / F 6. Done. The external storage means 9 can use an IC memory card, a magneto-optical disk, or the like, but may transmit an image signal directly to a remote recording medium via a network using a modem card or an ISDN card. . Conversely, the reading of the image signal recorded in the external storage means 9 is performed by transmitting the image signal to the signal processing means 3 via the I / F 6 and performing image expansion in the signal processing means 3. On the other hand, display of the image signal read from the external storage means 9 and the frame memory 7 is signal processing such as digital-analog conversion (hereinafter abbreviated as D / A conversion) or amplification for the image signal in the signal processing means 3. Is performed by transmitting to the display means 8 through the I / F 6. The display unit 8 is configured by a liquid crystal display device installed in the housing of the image processing apparatus 1, for example.
[0048]
  The plane measuring unit 10 measures the orientation of the subject surface with respect to the image processing apparatus 1 when each image is captured. As shown in FIG. 3, the plane measuring unit 10 includes a spot light source 21, a light receiving element 22, a three-dimensional coordinate calculating unit 23, and a plane calculating unit 34 as an example. The spot light source 21 includes a light source 25 such as a light emitting diode or a semiconductor laser, a scanning mirror 26 such as a polygon mirror, and a driving unit 27 that controls the movement of the scanning mirror 26 so that spot light generated from the light source 25 strikes the subject surface. This is a part for controlling the direction of light, and the direction of the light is detected by a signal output from the driving means 27. The light receiving element 22 is composed of a photoelectric conversion element such as a PSD (position sensitive detector) or a CCD installed in a place where the positional relationship with the spot light source 21 is known, and detects the direction of the reflected light returning from the subject surface. . The three-dimensional coordinate calculation means 23 is an image based on the principle of triangulation from the direction of the spot light irradiated by the spot light source 21, the positional relationship between the spot light source 21 and the light receiving element 22, and the direction of the reflected light detected by the light receiving element 22. The three-dimensional coordinates (X, Y, Z) of the subject surface with respect to the processing device 1 are calculated. The plane calculation unit 34 estimates a plane equation using three or more three-dimensional coordinates that are not on the same straight line and calculated by the three-dimensional coordinate calculation unit 23. For example, the plane equation to be calculated is
[0049]
[Expression 1]

  Then, using the three-dimensional coordinates (Xi, Yi, Zi) of N points (i = 1,..., N), the four parameters (a, b, c, d) of the plane equation are expressed by the following equations: It is calculated by the least square method shown.
[0050]
[Expression 2]

  The image distortion correction unit 11 uses the three parameters (a, b, c) representing the orientation of the subject surface output from the plane measuring unit 10 to convert the subject surface into an image photographed from the front, thereby obliquely photographing. Correct image distortion caused by. In the present embodiment, as shown in FIG. 4, by projecting an image having a distorted subject image toward the projection surface using a plane parallel to the subject surface measured by the plane measuring unit 10 as a projection surface, Correct image distortion. An image whose distortion has been corrected in this way is hereinafter referred to as a distortion corrected image. Hereinafter, a method for creating a distortion corrected image will be described in detail.
[0051]
  First, the direction of the line of sight corresponding to the image coordinates (x, y) is determined. As shown in FIG. 5, the optical system of the image pickup means 2 is x-axis: rightward image plane is positive y-axis: downward image plane is positive z-axis: optical axis direction; Assuming that the center projection model has a center f: focal length, this xyz coordinate system is hereinafter referred to as a device coordinate system. Then, the direction of the line of sight corresponding to (x, y) can be expressed by the following three-dimensional vector P with reference to the apparatus coordinate system.
[0052]
[Equation 3]

  Next, as shown in FIG. 6, a rotation matrix R that converts the z-axis of the apparatus coordinate system into a unit normal vector of the projection surface is obtained. In this case, the following relational expression is established.
[0053]
[Expression 4]

  There are many rotation matrices R satisfying the equation (4). Here, the rotation matrix R is defined as the following equation.
[0054]
[Equation 5]

  As shown in FIG. 7, this corresponds to rotating the apparatus coordinate system in the following order.
(1) Rotate the device coordinate system by β around the y axis. A coordinate system formed by this rotation is defined as an x1y1z1 coordinate system.
(2) When the device coordinate system is rotated by α around the x1 axis, using a2 + b2 + c2 = 1 (c> 0) and equations (4) to (6), the rotation angles α and β are Is derived as follows.
[0055]
[Formula 6]

R can be uniquely determined by substituting the rotation angles α and β obtained above into the equation (5).
[0056]
  Finally, using the three parameters (a, b, c) representing the three-dimensional vector p, the rotation matrix R, and the orientation of the subject surface obtained in the above procedure, the image coordinates (x, y) are received. Transform coordinates on the projection plane. That is, in FIG. 4, a point p that intersects the projection surface when a three-dimensional vector P corresponding to one point on the image plane is extended is set as an image coordinate after coordinate conversion. When a three-dimensional vector corresponding to the point p with reference to the apparatus coordinate system is P, P is expressed by the following equation.
[0057]
[Expression 7]

Here, k is an enlargement coefficient representing the distance from the optical center 0 of the image pickup means 2 to the projection surface, and as described above, distortion is corrected by projecting an image toward the projection surface. k represents the size of the distortion-corrected image to be created. P is a vector representing the point p with reference to the apparatus coordinate system at the time of image capture. By converting the coordinates of P using the rotation matrix R as shown in the following equation, the image processing apparatus 1 is moved to the object plane. Is converted into a three-dimensional vector P ′.
[0058]
[Equation 8]

  Therefore, the image coordinate (x, y) after correction conversion is obtained by setting the x coordinate and y coordinate of the three-dimensional vector P ′ in the equation (10) as the image coordinate after coordinate conversion of the point (x, y). obtain. When organized, the following formula is obtained.
[0059]
[Equation 9]

  With the above procedure, a distortion-corrected image can be created by converting an arbitrary coordinate (x, y) on the image plane into (X, Y) using the equation (11). The image coordinates (x, y) before conversion corresponding to the image coordinates (X, Y) of the distortion corrected image are not generally integers, but are interpolated using existing methods such as bilinear interpolation and B-spline interpolation. By doing so, an image may be created.
[0060]
  The image processing method as described above will be described with reference to the flowchart of FIG. First, at the same time that the subject surface is photographed by the imaging means 2, three parameters representing the orientation of the subject surface at the time of photographing are obtained (step S1). Next, based on the obtained (a, b, c), the rotation matrix R is calculated by the procedure described above (step S2). Based on (a, b, c) and the rotation matrix R obtained by the above procedure, a distortion-corrected image is created by performing coordinate transformation of equation (11) (step S3), and the process is terminated.
[0061]
  [Embodiment 2] In Embodiments 2 to 4 to be described below, as shown in FIG. 9, using an image processing apparatus 1, still images of the same subject surface are displayed from different locations. The operation when shooting from K locations from viewpoint 1 to viewpoint K so as to partially overlap will be described. An image taken from viewpoint 1 is called Im 1, and an image taken from viewpoint 2 is called Im 2. For the sake of simplicity, it is assumed that there is an overlapping area between Imj and Im j + 1 taken in the adjacent order, such as Im 1 and Im 2 and Im 2 and Im 3.
[0062]
  FIG. 10 is a block diagram of an image processing apparatus 1 according to Embodiment 2 of the present invention. 10, members having the same reference numerals as those in FIG. 2 are the same members as those in the first embodiment of the invention, and thus detailed description thereof is omitted.
[0063]
  The image processing apparatus 1 performs coordinate conversion on the basis of the relationship between the feature points and the corresponding points output from the correspondence detection unit 31 by extracting the feature points and the corresponding points between the images having overlapping areas. An image pasting unit 32 for pasting images having areas overlapping each other, a pasted image storage unit 33 for storing a pasted image created by the image pasting unit 32, and the subject surface output by the plane measuring unit 10 Based on the orientation, a composite image distortion correction unit 34 that corrects distortion caused by oblique photographing of the composite image stored in the composite image storage unit 33 is provided.
[0064]
  The configurations and operations of the correspondence detecting unit 31, the image combining unit 32, the combined image storage unit 33, and the composite image distortion correcting unit 34 will be described below. The correspondence detection means 31 detects a place where the same part is photographed in two images having overlapping areas. Here, a method using correlation calculation will be described. As shown in FIG. 11, the correspondence detection unit 31 includes a feature point setting unit 35 and a correlation calculation unit 36. In the frame memory 7 of FIG. 10, a previously captured image (standard image) and a reference image having a region overlapping with the standard image are stored. The feature point setting means 35 determines the position of the feature point in the reference image, extracts a gray pattern centered on the feature point, and creates a correlation window. The position of the feature point is determined by extracting a portion where the density pattern of the image is characteristic, such as a corner. Correlation calculation means 36 detects a location that substantially matches the shading pattern of the correlation window created in the standard image by correlation calculation in the reference image, and determines this as a corresponding point. Here, an example of detecting corresponding points by block matching based on correlation calculation will be described. As shown in FIG. 12, the cross-correlation value between the i-th feature point (xi0, yi0) in the base image and the point (xi0 + dxi, yi0 + dyi) in the reference image by block matching using the (2N + 1) (2P + 1) correlation window. Si is calculated by the following equation.
[0065]
[Expression 10]

  For each feature point, a corresponding point (xi0 + dxi, yi0 + dyi) in the reference image is obtained by obtaining a point where the maximum value of the cross-correlation value Si is equal to or greater than a predetermined threshold value. If the maximum value of Si is not more than the threshold value, it is assumed that there is no corresponding point.
[0066]
  The image pasting unit 32 pastes the images by performing coordinate conversion of the standard image into a reference image based on the relationship between the feature points output from the correspondence detecting unit 31 and the corresponding points. Here, description will be made assuming that projective transformation is used for coordinate transformation. The projective transformation means that when two subject planes are photographed as shown in FIG. 13 and one image is taken as a reference image and the other image is taken as a reference image, the subject image shown in the reference image is photographed as a reference image as shown in FIG. It refers to converting to the appearance of. When this is expressed by a mathematical expression, the following relationship is established between the feature point (xs, ys) in the base image and the corresponding point (xr, yr) in the reference image.
[0067]
## EQU11 ##

  Since there are eight unknown parameters in equation (13), if four or more pairs of feature points and corresponding points are obtained, the coefficients b1 to b8 can be obtained by a calculation method such as the least square method. The image that is pasted by calculating the coefficients b1 to b8 by the above procedure is called a projective transformation image.
In the above description, the method of creating the projection conversion image when photographing from two viewpoints has been described. However, it is possible to sequentially create the projection conversion image by iterative processing and paste three or more images. Hereinafter, an image obtained by projectively transforming Im 1 to Im j−1 with respect to Im j is referred to as PIm j (1 ≦ j ≦ K, where PIm 1 = Im 1). In PIm j, the subject images captured in Im 1 to Imj-1 are converted into the way they are seen at the time of Imj shooting and pasted on Im j. The iterative process for creating PIm j is shown below.
[0068]
  The following processing is repeated for j = 2,…, K.
(I)Let PIm j-1 be a standard image and Im j be a reference image.
(Ii)Feature points are extracted from the reference image, and corresponding points are detected from the reference image.
(Iii) (ii)Using the four or more sets of feature points and corresponding points detected in step (1), the coefficients b1 to b8 of equation (13) are obtained, and PIm j-1 is projectively transformed to Im j to create PIm j.
(Iv)Add one j,(I)Return to.
[0069]
  By the above iterative process, finally, an image PIm K obtained by projective conversion of Im 1 to Im K-1 to Im K is obtained, that is, one composite image can be created from Im 1 to Im K. . The composite image created in this way is stored in the composite image storage means 33.
[0070]
  The composite image distortion correcting unit 34 corrects distortion in the composite image PIm K created by the image combining unit 32. As described above, the combined image PIm K created by the image combining unit 32 is an image that has been converted and combined into the appearance at the time of capturing the reference image. Therefore, if the subject image is distorted in the reference image, it is naturally pasted. The combined image also includes distortion. Therefore, in FIG. 15, when a reference image (= Im K) is photographed, the three parameters (a, b, c) representing the orientation of the subject surface output from the flat surface measuring means 10 are used. Similar to 1, the distortion of the composite image PIm K is corrected by the processing of equations (3) to (11).
[0071]
  The above-described image processing method will be described with reference to the flowchart of FIG. First, a subject surface is photographed by the imaging means 2, and this image is stored as a reference image, and at the same time, three parameters (a, b, c) representing the orientation of the subject surface at the time of photographing are obtained (step S11). Subsequently, the subject plane is photographed by the imaging means 2 while changing the viewpoint and the posture of the image processing apparatus 1 so that the reference image partially overlaps the reference image, and this image is stored as a reference image. Three parameters (a, b, c) representing the direction of are obtained (step S12). A feature point is extracted from the reference image, and a corresponding point indicating the same location as the feature point is detected in the reference image (step S13). Based on the relationship between the feature points and the corresponding points obtained in this way, an image is generated by coordinate-converting the reference image with respect to the reference image by the method described above (step S14). Thereafter, when more images are to be captured and combined (step S15), the combined image created by the above processing is replaced with a reference image (step S16), and the operations from step S2 to step S14 are repeated. . If this is not the case, the last created composite image is subjected to coordinate transformation of equation (11) using the subject surface parameters (a, b, c) at the time of capturing the reference image. The distortion is corrected (step S17), and the process ends.
[0072]
  [Third Embodiment] FIG. 17 is a block diagram of an image processing apparatus 1 according to a third embodiment of the present invention. In FIG. 17, members having the same reference numerals as those in FIG. 2 are the same members as those in the first embodiment of the invention, and detailed description thereof will be omitted.
[0073]
  The image processing apparatus 1 includes an image distortion correction unit 41 that corrects distortion caused by oblique photographing of each image based on the orientation of the subject surface output from the plane measurement unit 10.
[0074]
  The configuration and operation will be described focusing on the processing procedure by the image processing apparatus 1. The plane measuring unit 10 measures the orientation of the subject surface with respect to the image processing apparatus 1 when each image is captured. The image distortion correction unit 41 corrects distortion caused by oblique photographing of each image in the procedure described above using the direction (a, b, c) of the object surface output from the plane measurement unit 10. Correspondence detection means 31 detects a portion where the same portion is photographed in two images that have been subjected to distortion correction by image distortion correction means 41 and have overlapping areas. For example, when the association is performed using the above-described correlation calculation, the correlation between the images corrected for the distortion caused by the oblique photographing is obtained in the present embodiment, so that the association can be performed more easily and accurately. Based on the relationship between the standard image, the reference image, the feature point, and the corresponding point obtained by correcting the distortion obtained as described above, the coefficients b1 to b8 in Expression (13) are obtained and the standard image is pasted to the reference image.
[0075]
  The image processing method as described above will be described with reference to the flowchart of FIG. First, the subject surface is photographed by the imaging means 2, and this image is stored as a reference image, and at the same time, three parameters (a, b, c) representing the orientation of the subject surface at the time of photographing are obtained (step S21). Using (a, b, c), the distortion in the reference image is corrected by the processing procedure of equations (3) to (11), and the reference image is replaced with the distortion corrected (step S22). Subsequently, the viewpoint and the posture of the image processing apparatus 1 are changed so that a part of the reference image overlaps, and the subject surface is photographed by the imaging unit 2, and this image is stored as a reference image, and at the same time, the subject at the time of photographing Three parameters (a, b, c) representing the orientation of the surface are obtained (step S23), and the parameters (a, b, c) are used in the reference image in the processing procedure of equations (3) to (11). The distortion is corrected, and the reference image is replaced with a distortion-corrected image (step S24). A feature point is extracted from the reference image, and a corresponding point indicating the same location as the feature point in the reference image is detected (step S25). Based on the relationship between the feature points and the corresponding points obtained in this way, an image is generated by coordinate-converting the reference image with respect to the reference image by the method described above (step S26). Thereafter, when more images are to be captured and combined (step S27), the combined image created by the above processing is replaced with a reference image (step S28), and the operations from step S23 to step S26 are repeated. . Otherwise, the process ends.
[0076]
  In the present embodiment, since the images corrected for the distortion of the image due to the oblique shooting are pasted together, the pasted image can also be used by using the following affine transformation instead of the projective transformation represented by the equation (13). Can be obtained.
[0077]
[Expression 12]

  It goes without saying that the image pasting means 32 is not limited to the configuration / operation based on the projective transformation of equation (13) or the affine transformation of equation (14).
[0078]
  [Embodiment 4] Embodiment 4 of the present invention is the same as Embodiments 1 to 3 in that the size of a distortion-corrected image created using the distance to the subject surface is set for each subject. The present invention relates to adjustment so that the image sizes are approximately equal. For example, as shown in FIG. 19 (a), when the standard image and the reference image are photographed in a state where the distance to the subject surface and the direction of the optical axis (z-axis) of the image pickup means 2 are different, In addition, the size of the subject image in each distortion-corrected image may be different. Therefore, the image distortion correction unit 41 is configured to adjust the size of the distortion correction image using the output of the plane measurement unit 10. One example will be described below.
[0079]
  The plane measuring means 10 has the same configuration as that of the first embodiment of the invention, and parameters (a, b, c, d) representing the object plane equations and the three-dimensional coordinates of each point based on the apparatus coordinate system. Is output. The image distortion correction means 41 corrects the distortion of each image according to the procedures of equations (3) to (11) from (a, b, c, d) output from the plane measurement means 10 and the three-dimensional coordinates of each point. At the same time, the optimum enlargement coefficient k used for the coordinate conversion of equation (11) is calculated. For example, in FIG. 19A, the parameters of the subject surface output by the plane measuring unit 10 at the time of capturing the reference image are (a, b, c, d), and the object plane output by the plane measuring unit 10 at the time of capturing the reference image. Let the parameters be (a ′, b ′, c ′, d ′). If the enlargement coefficients used for creating the distortion-corrected image of the reference image and the reference image are k and k ′ (k> 0, k ′> 0), respectively,
[0080]
[Formula 13]

Then, as shown in FIG. 19B, distortion-corrected images in which the sizes of the subject images are approximately equal are obtained.
[0081]
  [Embodiment 5] Embodiment 5 of the present invention is an image obtained by using three parameters (a, b, c) representing the orientation of a subject surface at the time of photographing in the first to fourth embodiments of the present invention. By converting the processing device 1 into a photographed image facing the subject surface, the distortion of each image is corrected. This makes the optical axis (z-axis) of the imaging means 2 coincide with the orientation of the subject surface. This means that an image obtained as a result of such coordinate conversion is generated in a pseudo manner. However, this coordinate conversion leaves a degree of freedom around one axis with the normal vector of the subject surface as the rotation axis. That is, depending on the posture of the image processing apparatus 1 at the time of shooting, a distortion corrected image in which the subject image is inclined on the image plane is created as shown in FIG. The purpose of this embodiment is to provide the tilt correction means 51 in the first to fourth embodiments of the invention and obtain a distortion-corrected image having almost no tilt as shown in FIG.
[0082]
  As shown in FIG. 21, the inclination correction unit 51 includes a rotation angle setting unit 52, a rotation center setting unit 53, and a rotation calculation unit 54. The rotation angle setting unit 52 and the rotation center setting unit 53 are configured by switches or the like, and are disposed, for example, on the casing of the image processing apparatus 1 at a position adjacent to the display unit 8. The rotation angle setting means 52 designates the rotation angle θ on the screen, and the rotation center setting means 53 designates the rotation center coordinates (x0, y0) on the screen. The rotation calculation means 54 is configured by an arithmetic circuit or the like built in the housing, and uses the rotation angle θ and the rotation center coordinates (x0, y0) to tilt the distortion corrected image on the screen by the following coordinate conversion formula. Correct.
[0083]
[Expression 14]

  An image obtained by applying the coordinate transformation of equation (16) to the distortion corrected image and correcting the tilt can be confirmed on the display means 8.
[0084]
  Needless to say, the description of the embodiment of each invention described above is merely an example of the present invention, and the present invention should not be limited or reduced according to the embodiment of each invention described above. . For example, the configuration of the plane measuring unit 10 is not limited to the range described in the above embodiment, and the object is irradiated with a wave (light, radio wave, sound wave), and the propagation time and phase difference of the reflected wave from the object are measured. Thus, another configuration may be employed, such as obtaining the distance to the object, or obtaining the distance from the amount of defocus on the subject surface. Further, the correspondence detection unit 31 has been described as detecting the corresponding points by density matching using the correlation method, but another method such as a spatiotemporal differentiation method may be used.
【The invention's effect】
[0085]
  Invention of Claim 1According toEven when an arbitrary subject plane is photographed from an arbitrary viewpoint / posture, distortion of the image can be easily corrected by measuring the plane.
[0086]
  Invention of Claim 2According toCreate a composite image easily without distortion by converting the coordinates of multiple images obtained by photographing an arbitrary subject surface from an arbitrary viewpoint and orientation using the measured orientation of the subject surface. Can do.
[0087]
  Invention of Claim 3According toEven when an arbitrary subject plane is photographed from an arbitrary viewpoint / posture, distortion of each image can be easily corrected by measuring the plane. Further, since the correspondence is detected by using the image whose distortion is corrected, the accuracy of the correspondence is further improved as compared with the image processing apparatus according to claim 2, and as a result, a higher-definition composite image can be easily obtained. Obtainable.
[0088]
  Invention of Claim 4According to, Claim 1The image processing apparatus according to any one of 1 to 3InBy calculating the rotation angle of the optical axis of the imaging means with respect to the normal vector of the subject surface, image distortion can be easily corrected.
[0089]
  Invention of Claim 5According to, Claim 1The image processing apparatus according to any one of 1 to 3InBy correcting the captured image to an image captured from the direction of the normal vector of the subject surface, it is possible to easily correct image distortion.
[0091]
  Claim6According to the invention described in claim 1, the claims 1 to5In the image processing apparatus according to any one of the above, it is possible to correct distortion and inclination of the subject image in each photographed image or a pasted image. Further, if the invention according to the present invention is applied to the invention according to claim 3, since the correspondence is detected by using the distortion corrected image in which the inclination of the subject image is substantially equal, the accuracy of the correspondence further increases, As a result, a higher-definition composite image can be easily obtained.
[0092]
  Claim7According to the invention described in (1), even when an arbitrary subject plane is photographed from an arbitrary viewpoint / posture, distortion of the image can be easily corrected by measuring the plane.
[0093]
  Claim8According to the invention described in (4), an image obtained by pasting together a plurality of images obtained by photographing an arbitrary subject surface with an arbitrary viewpoint and posture is subjected to coordinate conversion using the measured orientation of the subject surface, so that distortion can be easily performed. No composite image can be created.
[0094]
  Claim9According to the invention described in (1), even when an arbitrary subject plane is photographed from an arbitrary viewpoint / posture, distortion of each image can be easily corrected by measuring the plane. Further, since correspondence is detected using an image whose distortion has been corrected,8As compared with the invention described in (1), the accuracy of the association is further improved, and as a result, a higher-definition composite image can be easily obtained.
[0095]
  Claim10According to the invention described in claim7Thru9In the image processing method according to any one of the above, the image distortion can be easily corrected by calculating the rotation angle of the optical axis of the imaging means with respect to the normal vector of the subject surface.
[0096]
  Claim11According to the invention described in claim7Thru9In the image processing method according to any one of the above, the distortion of the image can be easily corrected by correcting the captured image to an image captured from the direction of the normal vector of the subject surface.
[0098]
  Claim12According to the invention described in claim7Thru11In the image processing method according to any one of the above, it is possible to correct the distortion and inclination of the subject image in each captured image and the combined image. Claims9If the invention described in the present invention is applied to the invention described in, the correspondence is detected by using the distortion corrected image in which the inclination of the subject image is substantially equal, so that the accuracy of the correspondence is further improved, and as a result, the higher definition A simple composite image can be easily obtained.
[Brief description of the drawings]
FIG. 1 is a conceptual diagram illustrating correction of image distortion that occurs when a subject surface is imaged.
FIG. 2 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 1 of the present invention.
FIG. 3 is a block diagram illustrating a configuration of a plane measuring unit of the image processing apparatus.
FIG. 4 is a conceptual diagram illustrating the operation of the image processing apparatus.
FIG. 5 is a conceptual diagram illustrating the operation of the image processing apparatus.
FIG. 6 is a conceptual diagram illustrating the operation of the image processing apparatus.
FIG. 7 is a conceptual diagram illustrating the operation of the image processing apparatus.
FIG. 8 is a flowchart for explaining the operation of the image processing apparatus.
FIG. 9 is a conceptual diagram when shooting still images of the same subject surface from different locations from K locations from viewpoint 1 to viewpoint K using an image processing apparatus so that part of the images overlap. is there.
FIG. 10 is a block diagram illustrating a configuration of an image processing apparatus according to Embodiment 2 of the present invention.
FIG. 11 is a block diagram illustrating a configuration of correspondence detection means of the image processing apparatus.
FIG. 12 is a conceptual diagram illustrating the operation of the image processing apparatus.
FIG. 13 is a conceptual diagram illustrating the operation of the image processing apparatus.
FIG. 14 is a conceptual diagram illustrating the operation of the image processing apparatus.
FIG. 15 is a conceptual diagram illustrating the operation of the image processing apparatus.
FIG. 16 is a flowchart illustrating the operation of the image processing apparatus.
FIG. 17 is a block diagram illustrating a configuration of an image processing apparatus according to a third embodiment of the present invention.
FIG. 18 is a flowchart illustrating the operation of the image processing apparatus.
FIG. 19 is a conceptual diagram illustrating the operation of the image processing apparatus according to the fourth embodiment of the present invention.
FIG. 20 is a conceptual diagram for explaining the operation of an image processing apparatus according to Embodiment 5 of the present invention;
FIG. 21 is a conceptual diagram illustrating the operation of the image processing apparatus.
[Explanation of symbols]
1 Image processing device
2 Imaging means
10 Planar measuring means
11 Image distortion correction means
31 Correspondence detection means
32 Image pasting means
34 Composite image distortion correction means
51 Tilt correction means

Claims (12)

Means for imaging the subject surface as an image;
When imaging with the imaging means, the object surface is irradiated with light, and based on the direction of reflected light from the subject surface, the imaging means of three or more points on the subject surface is used as a reference. Means for calculating three-dimensional coordinates;
Based on the three-dimensional coordinates of each point calculated in the means for calculating comprises means for determining the orientation of the object plane with respect to said means for imaging,
Means for correcting the distortion of the determined based on the orientation of the object plane determined by means the imaged converted to the captured image an image in the image imaged from the object plane front,
An image processing apparatus comprising: Means for capturing a subject surface as a reference image so that a part of the reference image captured previously overlaps;
When imaging with the imaging means, the object surface is irradiated with light, and based on the direction of reflected light from the subject surface, the imaging means of three or more points on the subject surface is used as a reference. Means for calculating three-dimensional coordinates;
Based on the three-dimensional coordinates of each point calculated in the means for calculating comprises means for determining the orientation of the object plane with respect to said means for imaging,
Correspondence detection means for extracting a plurality of feature points from the reference image and detecting corresponding points corresponding to the feature points from the reference image;
Based on the feature points and corresponding points detected by the correspondence detection means, means for coordinate-transforming the standard image on the reference image and pasting the standard image and the reference image ;
Based on the orientation of the object plane with respect to said means for imaging of imaging the reference image obtained by said obtaining means, the imaging converts the bonded image in the image imaged from the object plane front Means for correcting distortion of the captured image ;
An image processing apparatus. Means for capturing a subject surface as a reference image so that a part of the reference image captured previously overlaps;
When imaging with the imaging means, the object surface is irradiated with light, and based on the direction of reflected light from the subject surface, the imaging means of three or more points on the subject surface is used as a reference. Means for calculating three-dimensional coordinates;
Based on the three-dimensional coordinates of each point calculated in the means for calculating comprises means for determining the orientation of the object plane with respect to said means for imaging,
Means for correcting the distortion of the based on the orientation of the object surface obtained by the means for obtaining said imaged converted to the captured image an image in the image imaged from the object plane front,
A corresponding detection means for detecting a corresponding point the extracted plurality of feature points from the corrected the reference image correction to means, corresponding to the feature point from the corrected the reference image in the means for correcting,
Means for attaching an image by the corresponding detecting means based on said corresponding points and the feature points detected, coordinate transformation of the corrected reference image on said corrected reference image,
An image processing apparatus. The means for obtaining the orientation of the subject surface is:
The image processing apparatus according to claim 1, wherein a rotation angle of the optical axis of the imaging unit with respect to a normal vector of the subject surface is calculated. The means for correcting is
The image processing apparatus according to claim 1, wherein the captured image is corrected to an image captured from a direction of a normal vector of the subject surface.   The image processing apparatus according to claim 1, further comprising an inclination correction unit that corrects an inclination of the image corrected by the correction unit on a screen.   Imaging the subject surface as an image by means for imaging;
  When imaging in the imaging step, the object surface is irradiated with light, and based on the direction of reflected light from the object surface, the means for imaging three or more points on the object surface is used as a reference. Calculating three-dimensional coordinates;
  Obtaining the orientation of the subject surface relative to the imaging means based on the three-dimensional coordinates of each point calculated in the calculating step;
  Converting the captured image into an image captured from the front of the subject surface based on the orientation of the subject surface determined in the determining step, and correcting distortion of the captured image;
An image processing method comprising:   Imaging the subject surface as a reference image so that a part of the reference image previously captured by the imaging means overlaps;
  When imaging in the imaging step, light is applied to the subject surface, and the imaging step of three or more points on the subject surface is based on the direction of reflected light from the subject surface. Calculating three-dimensional coordinates;
  Obtaining the orientation of the subject surface relative to the imaging means based on the three-dimensional coordinates of each point calculated in the calculating step;
  A correspondence detection step of extracting a plurality of feature points from the reference image and detecting corresponding points corresponding to the feature points from the reference image;
  Based on the feature points and corresponding points detected by the correspondence detection step, the step of performing coordinate transformation on the reference image on the reference image and pasting the reference image and the reference image;
  Based on the orientation of the subject surface with respect to the imaging means when imaging the reference image obtained in the obtaining step, the combined image is converted into an image taken from the front of the subject surface and the imaging Correcting distortion of the generated image;
An image processing method comprising:   Imaging the subject surface as a reference image so that a part of the reference image previously captured by the imaging means overlaps;
  When imaging in the imaging step, the object surface is irradiated with light, and based on the direction of reflected light from the object surface, the means for imaging three or more points on the object surface is used as a reference. Calculating three-dimensional coordinates;
  Obtaining the orientation of the subject surface relative to the imaging means based on the three-dimensional coordinates of each point calculated in the calculating step;
  Converting the captured image into an image captured from the front of the subject surface based on the orientation of the subject surface determined in the determining step, and correcting distortion of the captured image;
  A correspondence detecting step of extracting a plurality of feature points from the reference image corrected in the correcting step and detecting corresponding points corresponding to the feature points from the reference image corrected by the correcting means;
  Based on the feature points detected by the correspondence detection step and the corresponding points, pasting the corrected reference image on the corrected reference image to coordinate the images; and
An image processing method comprising:   The step of obtaining the orientation of the subject surface includes
  The image processing method according to claim 7, wherein a rotation angle of the optical axis of the imaging unit with respect to a normal vector of the subject surface is calculated.   The correcting step includes
  The image processing method according to claim 7, wherein the captured image is corrected to an image captured from a direction of a normal vector of the subject surface.   The image processing method according to claim 7, further comprising an inclination correction step of correcting an inclination on the screen of the image corrected by the correction step.

Images