JP3679512B2 - Image extraction apparatus and method - Google Patents

Description

Ｉ（ｖ（ｓ））はｖ（ｓ）上の輝度レベルを表し、α（ｓ）、β（ｓ）、ｗ0は、ユーザが適宜定める。輪郭の変形、移動は時系列的に逐次更新しながら行われる。各ステップでは輪郭線上の各点において移動可能範囲の点の集合（予め定義される近傍領域）から上記関数が最小となる点を選択したり、あるいは上記評価関数を最小にする輪郭を変分法を用いてＥｕｌｅｒの方程式を解くことなどにより処理される。
【００９６】
このように、マスクデータ上で動的輪郭処理を行うことにより、被写体画像上の被写体近傍にある背景パターンに誤って収束することを回避しつつ、領域成長後のマスク形状の平滑化、未成長領域の輪郭形状の修整などを行うことができる。特に、修整機能としては形状の欠損部分に主観的輪郭線に相当する滑らかな連続形状を生成する場合などに発揮される。
【００９７】
エネルギ関数が最小値に収束した後の輪郭形状は多くの場合、被写体の形状に十分に近いと考えられる。そこで、これを初期輪郭として設定し（ステップＳ４０３）、被写体画像上で上述した動的輪郭処理を同様に適用し（ステップＳ４０４）、最終的に収束した輪郭内を切り出しマスク領域とする（ステップＳ４０５）。
【００９８】
尚、マスクデータ上での動的輪郭処理は必須でない場合があるが、このようなときはステップＳ４０２、Ｓ４０３の処理を省略しても構わない。
【００９９】
また、初期マスク領域抽出後、領域成長処理を省いてマスク領域を包囲するように初期輪郭を設定し、マスクデータに対し上述した動的輪郭処理を行ってもよい。この場合、概略収束した輪郭形状を初期輪郭として更に被写体画像上で輪郭整形処理して細部の形状を抽出する。また、条件によっては初期マスク領域抽出後、被写体画像上で上記動的輪郭処理を行ってもよい。
【０１００】
［第３の実施の形態］
図１３は第３の実施の形態における撮像システムの構成を示すブロック図である。本実施の形態では、切り出したい被写体が写った被写体画像とその被写体を除いた背景画像との２つが入力画像として用いられる。
【０１０１】
撮像装置１は、レンズ、絞りおよびそれらのレンズ駆動制御部１ｅを含む結像光学系１ａ、イメージセンサ１ｂ、映像信号処理回路（ガンマ特性、ホワイトバランス制御、露光条件制御、フォーカシング特性制御などを行う）１ｃ、画像記録部１ｄ、インターフェース部１ｆなどを主な構成要素とする。
【０１０２】
切り出し処理装置２は、被写体画像一時記憶部３ａおよび背景画像一時記憶部３ｂからなる画像一時記憶部３、差分画像データの閾値処理を行う初期マスク領域抽出部５、領域成長部６、被写体画像出力部８、インターフェース部１１などから構成される。切り出し処理装置２の外部には画像表示装置９および端末装置１０が接続されている。
【０１０３】
図１４および図１５は被写体切り出し処理手順を示すフローチャートである。尚、切り出し処理装置としては上記構成の他、図１４および図１５のフローチャートで示すプログラムとしてコンピュータの記憶装置上に存在するもの、同様にゲートアレーとして存在するもの、さらに撮像装置１に内蔵されているものなど種々の形態をとることができる。
【０１０４】
まず、被写体画像および背景画像を撮像装置１から入力する（ステップＳ１１１）。この際、後続の処理スピードを上げるために、適当な縮尺率で画像データの間引きを行い（ステップＳ１１２）、被写体画像上で被写体存在範囲を含むように処理領域の設定を行う（ステップＳ１１３）。尚、ステップＳ１１２の間引き処理およびステップＳ１１３の処理領域の設定処理は省いてもよい。
【０１０５】
図１６、図１７は被写体切り出し処理の主な途中結果が表示された画面を示す写真である。本実施の形態では、前記第１の実施の形態と同様に、図３の（ａ）、（ｂ）をそれぞれ入力被写体画像、背景画像とする。
【０１０６】
つづいて、領域成長の核を抽出するための初期核データ抽出処理を行う（ステップＳ１１６、Ｓ１１７、Ｓ１１８）。この処理は背景画像および被写体画像間での各画素について色成分（ＲＧＢ値または色相、彩度）の差異に基づき、例えば各成分の差分絶対値を閾値処理することにより初期核抽出を主として行う。ここで、閾値を比較的高く設定することにより、ノイズや撮影条件の差異に起因する画素値の変動の影響を排除し、かつ淡い陰影などを除去することができる。
【０１０７】
但し、三脚等に撮像装置１を固定せず手持ち状態で撮影を行い、また、背景画像、被写体画像をそれぞれを露光条件およびフォーカシングの自動設定により撮影する場合、閾値処理を行う前に以下の処理を行う必要がある。
【０１０８】
即ち、被写体画像および背景画像間の対応する点間での幾何学的またはＲＧＢ成分の変換パラメータ群を抽出し（ステップＳ１１４）、２つの画像間での位置合わせ（シフト、回転、倍率変換）、および各色成分（ＲＧＢ値）のレベル合わせ（最小二乗法などを用いた補正用非線形関数の推定などを行う）を抽出した変換パラメータ群を用いて行う（ステップＳ１１５）。
【０１０９】
また、他の処理法として各点を中心として定義される所定サイズのブロック間の相関係数または画素値の平均値、標準偏差値などの統計量を閾値処理することにより抽出してもよい。初期核抽出の結果得られるマスク領域を図１６の（ａ）に示す（被写体領域を黒で示す）。上記処理は図１３の切り出し処理装置２の初期マスク領域抽出部５で行われる。初期マスク領域抽出部５で行う代わりに、端末装置１０のコンピュータ内部のプログラムにより実行されるようにしてもよい。
【０１１０】
一般的な背景と被写体の組み合わせでは、この段階で切り出すべき被写体の領域が完全に抽出されることはあり得ない。即ち、被写体画像および背景画像間において同一箇所で部分的にＲＧＢレベル、あるいはそれらの局所的統計量（平均値、標準偏差など）などが類似した領域がある場合、その部分領域が閾値処理後に未抽出領域として残る。したがって、次の領域成長処理では、それらの復元が行われる。
【０１１１】
領域成長処理では、初期マスクの各核領域の境界上の点において近傍画素（または領域）との画像特徴量の類似度を求め、これが所定の閾値により高い場合、同一被写体領域とみなしてマスク領域に併合する。上記処理は図１３の切り出し処理装置２の中の初期領域成長部６で行われる。初期領域成長部６で行う代わりに、端末装置１０のコンピュータ内部のプログラムにより実行されるようにしてもよい。
【０１１２】
本実施の形態では、まず領域成長を行う前に被写体画像のエッジ強度分布抽出処理（ステップＳ１１９）、およびその閾値処理による２値化処理（ステップＳ１２０）、および初期核の成長範囲を制限するための最大領域成長範囲の設定処理（ステップＳ１２１）を初期核の領域データに基づいて行う。
【０１１３】
具体的には初期核の存在領域の横（Ｘ）方向各点でのＹ（縦）成分最小値、最大値（ＭｉｎＹ（Ｘ），ＭａｘＹ（Ｘ））、および縦（Ｙ）方向各点でのＸ（横）成分最小値、最大値（ＭｉｎＸ（Ｙ），ＭａｘＸ（Ｙ））を求め、各データに対して平滑化（ローパスフィルタ）を行う。
【０１１４】
本実施の形態では、所定サイズのメディアンフィルタを適用することにより突発的な変動が抑えられ、被写体の輪郭形状にほぼ沿った滑らかな成長範囲を得ることができる（図１６の（ｂ）参照）。
【０１１５】
つぎに、類似度判定に必要な特徴パラメータとして、輝度レベル、色相の近傍との差異の初期閾値Ｉｔｈ，Ｈｔｈの値を入力する（ステップＳ１２２）。これらは背景画像および被写体画像間のそれぞれのパラメータに関する差分（差分絶対値）データの平均値、標準偏差などの統計量に基づいて自動設定してもよい。更に、最大成長範囲およびエッジ強度分布に基づき各閾値の値を画像中の位置によって以下のように可変に設定する（ステップＳ１２３）。
【０１１６】
最大成長範囲（ＭｉｎＹ（Ｘ），ＭａｘＹ（Ｘ））、（ＭｉｎＸ（Ｙ），ＭａｘＸ（Ｙ））の中では閾値を高く、各範囲外では低く設定する。具体的には点（Ｘ，Ｙ）が各成長範囲のいずれの中にもある場合、高い閾値（例えば、初期閾値）に設定し、それ以外の場合、低い閾値（例えば、初期閾値の１割）に設定する。
【０１１７】
また、閾値の分布関数としては最大成長範囲の境界線からの距離が大きくなるにつれて減少する任意の関数を用いてもよい。この関数の他の値の取り方としては離散値とし、所定範囲の境界からの距離については同じ値を与えるようなものでもよい。さらに、別の閾値の与え方としては、領域成長の閾値を縦方向と横方向それぞれで設定し（Ｉｔｈｙ，Ｉｔｈｘ）、縦方向成長範囲（ＭｉｎＹ（Ｘ），ＭａｘＹ（Ｘ））内ではＩｔｈｙを高く、横方向成長範囲（ＭｉｎＸ（Ｙ），ＭａｘＸ（Ｙ））ではＩｔｈｘが高くなるように設定してもよい。
【０１１８】
本実施の形態では、さらに被写体画像のエッジ強度分布を所定の閾値で２値化し、エッジが存在する位置およびその近傍で閾値を低くする。閾値の分布としてはエッジの位置で最も低く（例えば、値０とするとエッジと交差する領域成長は完全に抑止される）、その近傍ではエッジからの距離に対して若干の増加を呈するもの、あるいはエッジとその近傍で一様に低い値をとるものなどが用いられる。
【０１１９】
図１８はエッジ分布および最大領域成長範囲に基づいて設定された近傍画素との特徴量差異に基づく領域成長の閾値分布を示す説明図である。横軸Ｘ（Ｙ）は縦方向の位置Ｙにおける被写体画像の断面上の横方向位置を表し、最大成長範囲ＭｉｎＸ（Ｙ），ＭａｘＸ（Ｙ）も同様にその断面上での横方向の成長範囲の最小値、最大値を示す。また、閾値Ｔｈｈ，Ｔｈｌはそれぞれ閾値の最大値、最小値を表す。尚、最大領域成長範囲の境界線を入力画像に重畳させてディスプレイに表示し、ユーザがこれに基づいて適正な平滑化フィルタサイズを設定できるようにしてもよい。
【０１２０】
上記閾値設定方法により初期閾値（Ｉｔｈ，Ｈｔｈ）の差異に対する領域成長結果のロバスト化、安定化が達せられる（被写体の輪郭形状に沿った形状の変動が少ない）。また、最大成長範囲が被写体の外側輪郭形状と異なる場合でも領域成長の結果、これが一致するように閾値（Ｉｔｈ，Ｈｔｈ）を設定する際、大まかに行うことができる。
【０１２１】
つぎに、特徴量に基づき、近傍画素について類似度判定処理（ステップＳ１２４）を行う。特に、本実施の形態では近傍画素（その時点で非マスク領域）とのＲＧＢそれぞれの差分絶対値がいずれも閾値以下となるか、または色相の差分絶対値が閾値以下であることを類似度判定、即ち成長条件に用い、いずれか一方が成立するとき当該近傍画素を同一被写体領域として併合する（ステップＳ１２５、図１７の（ａ）参照）。
【０１２２】
類似度判定に用いる特徴量としてはこれらに限定されるものではなく、例えば、彩度などの低次特徴量、あるいは高次特徴量としてエッジを含む局所的ラインセグメントの部分形状（セグメントの向きなど）、局所空間周波数、さらにはＲＧＢなどの低次特徴量を統計処理して得られる特徴量などが好適に用いられる。
【０１２３】
また、領域成長の併合処理は、必ずしも８近傍画素について行われるように限定されるものではなく、他の方法で求めた近傍領域を用いてもよい。さらに、領域成長後のマスクデータについて所定サイズ以下の穴を自動的に埋める穴埋め処理（ステップＳ１２６）を行う。
【０１２４】
この穴埋め処理は被写体画像データ、即ち、画像特徴量の近傍領域との類似度、均質性などとは一切無関係に行われ、２値化マスクデータに対して行われるものである。以上の結果、得られる成長済み領域を被写体切り出しマスク領域とし、該当する領域を被写体画像から抽出し（ステップＳ１２７）、ディスプレイに画像データを（または画像ファイルとして）出力する（ステップＳ１２８）ことにより切り出し処理は完了する（図１７の（ｂ）参照）。
【０１２５】
尚、被写体画像抽出処理（ステップＳ１２７）の際、マスク領域の境界線の平滑化処理、あるいは境界線の補正処理などを行ってからマスクに対応する被写体画像を抽出してもよい。切り出し画像出力処理（ステップＳ１２８）は図１３の切り出し処理装置２の被写体画像出力部８で行われる。被写体画像出力部８で行う代わりに、コンピュータ上のプログラムとして実行されるようにしてもよい。
【０１２６】
［第４の実施の形態］
図１９は第４の実施の形態における撮像システムの構成を示すブロック図である。図において、５１は画像入力装置であり、撮像装置または画像データベース部を示す。撮像装置としてはビデオカメラ、あるいは複眼タイプのカメラなど特に限定されない。
【０１２７】
５２は画像中の１次特徴データを抽出する一次特徴抽出部、５３は特徴データの均質度を評価する特徴均質度評価部である。５４は画像記憶部、５５は特徴データの均質度に基づき領域を分割する領域分割部、５６は分割画像生成部、５７は二次特徴に基づく領域成長部である。５８は表示装置である。５９は分割領域の選択を指示する分割領域指示装置であり、マウスなどのポインティングデバイスを表す。
【０１２８】
尚、本実施の形態では、領域成長は画像中の所定領域内の特徴量の均質度を評価して領域の分割、併合を行うことを意味する。本実施の形態における特徴は、画像中の一次特徴データを抽出し、その分布に基づいて画像の領域分割を行うことにより抽出可能性のある画像領域を概略的に抽出する。次のステップとしてより精細な抽出のために候補領域を核とする二次特徴量（一次特徴と種別、属性などが異なる特徴量）に基づく領域成長を前記第３の実施の形態と同様に行う。
【０１２９】
また、一次特徴データに基づいて抽出された領域には、二次特徴に基づく領域成長を行うために必要な画像情報が含まれているものとする。本実施の形態では二次特徴とは、基本的に一次特徴と異なるものであればよく、必ずしも幾何学的な構造を有するもの、あるいは輝度分布、色成分を処理して得られる高次の特徴を有するものなどのような限定を意味せず、例えば色成分情報などが好適に用いられる。但し、一次特徴と同じ種別（本実施の形態では動きベクトルまたは視差ベクトル）であっても、領域成長による精細な画像分割が可能であれば問題ない。
【０１３０】
一次特徴データの検出の目的は領域成長処理を行う際の成長核となる領域の抽出を高速に行うことで、実際に特定対象の画像領域を抽出する際のオペレータによる大まかな選択と指示を可能としたり、あるいは自動抽出処理を容易にすることである。
【０１３１】
一次特徴データとしては画像入力装置５１より時系列画像データを入力する場合、画面上の各点の動きベクトルを、また複眼カメラから画像を入力する場合、左右画像間の対応点の視差ベクトルの分布を用いる。一次特徴抽出部５２の抽出精度は後で用いる二次特徴抽出部の精度（分解能）より多少劣っていても構わないが、高速に抽出可能であることが望ましい。このため、一次特徴の抽出、均質領域の統合を行うための専用ハードウェアを設定してもよい。
【０１３２】
図２０は図１９の撮像システムの具体的構成を示すブロック図である。尚、視差ベクトル（動きベクトル）の検出アルゴリズムについては本発明の主眼ではないので、ここでの説明を省略する（「ロボットビジョン」谷内田 著、昭晃堂など参照）。
【０１３３】
本実施の形態では、特徴均質度の尺度として、所定領域内の一次特徴量（動きベクトル、視差ベクトルなど）の大きさと方向に関する分散で表すこととする。評価する領域（ブロック）のサイズを対象の画面内サイズに基づいて適切に設定することにより、処理時間の短縮と効率化が達成される。即ち、対象の画面内縦横サイズと比較して数分の１から１０分の１程度の分割領域程度が典型的に用いられる。この値はオペレータ側で予め適宜設定可能にしてもよい。
【０１３４】
領域分割部５５は、均質度の値（その領域内の分散値など）が所定閾値より小さい場合、その領域は均質であるとして均質度代表値が所定許容範囲内にある領域同士を連結し許容範囲外の領域を分離することにより、一定値の一次特徴量（視差ベクトル、動きベクトル等）で代表することができる均質な塊状領域（複数可）を形成する。この処理はいわゆる領域成長処理の１つとみなすこともできるが、この段階では、前記第３の実施の形態と同様の領域成長の条件（制約）を付与しない。
【０１３５】
図２１は複眼カメラで撮像された入力画像の例（ａ）、視差ベクトルの大きさに基づいて行った大まかな領域分割結果（ｂ）、および色成分などの二次特徴に基づく領域成長（及び分割）による画像切り出し結果（ｃ）を示す説明図である。
【０１３６】
尚、前述したように図２１の（ｂ）に示す分割後の領域（初期核）は第１の実施の形態と同様に必ずしも実際の対象物の形状を正確に反映するものではなくてもよいが、分割後の領域（初期核）内に背景領域が混在しないことが望ましい。
【０１３７】
このため、領域分割後（初期核抽出後）、一定割合で各領域を縮小したもの、あるいはｍｏｒｐｈｏｌｏｇｉｃａｌ演算子の作用によるマスクの細らせ（ｅｒｏｓｉｏｎ；縮退）処理を行った結果を分割領域（初期核）としてもよい。
【０１３８】
また、背景領域が部分的に混在する場合、まずその初期核領域を色成分などに基づいて分割し、初期核の輪郭線を含む小領域を削除してから領域成長を行うことなどにより、被写体形状のより精密な切り出しが可能である。
【０１３９】
分割画像（初期核画像）生成部５６では、一次特徴データに基づいて分割された複数の領域に対応してそれぞれの異なる属性の画像データ（例えば、異なる色、異なるハッチングパターンなど）を付与し、表示装置５８上に入力画像データと重なるように表示する。具体的には分割された領域にラベル付けを行い、同じラベル領域を固有のパターン（または色）で塗りつぶすこなどの処理を行う。
【０１４０】
このようにすることにより、オペレータは指示すべき候補領域の区別を目視により容易に行うことができ、かつ分割領域指示装置５９による抽出対象の指定が容易となる。分割領域指示装置５９としては典型的にマウスなどが用いられるが、タッチパネル型表示装置を用いる場合、不要である。画像中に動きのある物体、あるいは視差値の所定範囲の領域が１つしかない場合も指示選択は不要である。また、抽出された複数の初期核全てを領域成長核としてもよい。
【０１４１】
領域成長部５７は、一次特徴データに基づき抽出されて選択された初期核領域の近傍から二次特徴（例えば、色相、ＲＧＢ値など）を抽出する二次特徴抽出部５７１、二次特徴に基づく領域成長部５７２、およひ成長後の各領域を連結する領域連結部（穴埋め手段）５７３、最大領域成長範囲設定部５７４、二次特徴量の近傍画素との類似度評価のための閾値分布設定部５７５から構成される。
【０１４２】
各設定部５７４、５７５での処理内容は前記第３の実施の形態と同様であるが、本実施の形態における閾値分布設定の固有の方法としては、例えば指示選択（あるいは自動処理）により統合された初期核領域内の一次特徴量（視差ベクトル、動きベクトルなど）の不連続部を二次特徴量（ＲＧＢ値、色相など）の不連続部と同等に扱い、これらの上（および近傍）で類似度評価の閾値を低く設定してもよい。
【０１４３】
図２２は閾値分布の設定を示す説明図である。上記のような場合を含めた閾値（近傍画素との特徴量差異の許容値）分布の設定例を示し、視差エッジとは視差ベクトルの変化率がその点を含む近傍で所定閾値より大となる所を意味する。ここでは、画像中のいわゆるエッジ強度分布（強度分布または各色成分の強度分布などに対してＳＯＢＥＬ演算子を代表とする微分オペレータなどを作用させて得られる）と視差エッジが同一点で観測される場合、閾値を最も低く（Ｔｈｌ）、いずれか一方のみが観測される場合、中程度の閾値（Ｔｈｍ）、いずれのエッジも観測されず、さらにどのエッジ近傍にも属さない領域には最も高い閾値（Ｔｈｈ）を与えている。
【０１４４】
【発明の効果】
本発明の画像抽出方法によれば、抽出すべき被写体が記録された第１画像と該被写体を除く背景が記録された第２画像との間の差分データに基づき、抽出すべき領域の初期マスクを生成する第１ステップと、該生成された初期マスクの領域を近傍領域との特徴量類似度に基づき成長させる第２ステップと、該領域成長後のマスク領域に基づいて前記第１画像から前記被写体を抽出する第３ステップとを備えたので、ノイズや撮影条件の変動の影響を排除し、かつ淡い陰影部分を自動除去した被写体抽出が可能である。また、被写体内部において背景画像と類似する画像特性を有する領域を含めた被写体領域の抽出が可能である。
また前記第２ステップは、前記第１および第２画像からそれぞれ抽出される前記第１および第２エッジ強度画像間の差分エッジデータに基づいて得られるエッジ密度が成長方向において所定の閾値以下である場合、成長を抑止するので、領域成長条件を緩和したり、あるいは大まかに設定しても被写体外への領域成長を抑制し、高精度な被写体抽出が可能となる。また、初期マスク領域が被写体以外の領域（陰影部など）を含む場合でもそのような領域からの成長を抑制することが可能である。
【図面の簡単な説明】
【図１】第１の実施の形態における撮像システムの構成を示すブロック図である。
【図２】被写体切り出し処理手順を示すフローチャートである。
【図３】被写体画像および背景画像が表示された中間調画像を示す写真である。
【図４】被写体切り出し処理の主な途中結果が表示された中間調画像を示す写真である。
【図５】被写体切り出し処理の主な途中結果が表示された中間調画像を示す写真である。
【図６】被写体切り出し処理の主な途中結果が表示された中間調画像を示す写真である。
【図７】被写体切り出し処理の主な途中結果が表示された中間調画像を示す写真である。
【図８】被写体切り出し処理の結果が表示された中間調画像を示す写真である。
【図９】ステップＳ３０における領域成長処理手順を示すフローチャートである。
【図１０】輪郭整形処理手順を示すフローチャートである。
【図１１】エッジ選択処理を示す説明図である。
【図１２】動的輪郭整形処理手順を示すフローチャートである。
【図１３】第３の実施の形態における撮像システムの構成を示すブロック図である。
【図１４】被写体切り出し処理手順を示すフローチャートである。
【図１５】図１４につづく被写体切り出し処理手順を示すフローチャートである。
【図１６】被写体切り出し処理の主な途中結果が表示された中間調画像を示す写真である。
【図１７】被写体切り出し処理の主な途中結果が表示された中間調画像を示す写真である。
【図１８】エッジ分布および最大領域成長範囲に基づいて設定された近傍画素との特徴量差異に基づく領域成長の閾値分布を示す説明図である。
【図１９】第４の実施の形態における撮像システムの構成を示すブロック図である。
【図２０】図１９の撮像システムの具体的構成を示すブロック図である。
【図２１】複眼カメラで撮像された入力画像の例（ａ）、視差ベクトルの大きさに基づいて行った大まかな領域分割結果（ｂ）、および色成分などの二次特徴に基づく領域成長（及び分割）による画像切り出し結果（ｃ）を示す説明図である。
【図２２】閾値分布の設定を示す説明図である。
【符号の説明】
１ 撮像装置
２ 切り出し処理装置
３ 画像一時記憶部
４ 正規化エッジ強度抽出部
５ 初期化マスク領域抽出部
６、５７ 領域成長部
７ 輪郭整形処理部
５１ 画像入力装置
５２ 一次特徴抽出部
５３ 特徴均質度評価部
５５ 領域分割部
５６ 分割画像生成部[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image extraction apparatus and method for extracting a subject from a background image and a subject image.
[0002]
[Prior art]
Conventionally, as a general technique for image extraction (extraction), a video mat that generates a key signal by chroma key using a specific color background or image processing (histogram processing, difference, differentiation processing, contour enhancement, contour tracking, etc.) Techniques such as the Television Society Technical Report, vol.12, pp.29-34, 1988) are known.
[0003]
Further, the method of performing image extraction based on the difference from the background image is a classic one. For example, in Japanese Patent Laid-Open No. 4-216181, a mask image (a specific processing region) is added to the difference data between the background image and the processing target image. A method for extracting or detecting target objects in a plurality of specific regions in an image is disclosed.
[0004]
Further, Japanese Examined Patent Publication No. 7-16250 uses the color model of the extraction target to determine the existence establishment distribution of the extraction target from the color conversion data of the current image including the background and the difference data of the brightness between the background image and the current image. A technique is disclosed.
[0005]
The difference method with the background image generally represents a difference such as a luminance level or a color component between each pixel of the background image and the subject image by a predetermined evaluation function, and the difference level is equal to or higher than a predetermined value by performing threshold processing. The region is extracted. For example, cross-correlation between blocks of a predetermined size centered on each point as an evaluation function (Digital Picture Processing (2nd ed.), By Rosenfeld, A. and Kak, AC, Academic Press, 1982), normalized principal component features Amount (Academic Journal of the Institute of Electronics, Information and Communication Engineers Vol. J74-D-II, pp. 1731-1740), weighted addition value of standard deviation value and difference value (Journal of Television Society, Vol. 45, pp. 1270-1276) 1991), and the distance between local histograms regarding the hue and the luminance level (Journal of Television Society, Vol. 49, pp. 673-680, 1995).
[0006]
Japanese Patent Application Laid-Open Nos. 4-328689 and 7-31248 disclose methods for extracting only moving objects by extracting motion vectors or inter-frame difference data from moving images. Japanese Patent Publication No. 7-66446, Japanese Patent Publication No. 6-14358 and Japanese Patent Publication No. 4-48030 disclose a method of extracting a moving object based on a difference from a background image. Further, a method of extracting an object from a background based on a disparity distribution (that is, a distance distribution from an imaging unit) by extracting a disparity amount distribution from images from different left and right viewpoint positions obtained by using a compound eye imaging system ( The 1995 IEICE Information and Systems Society Conference, pp. 138) is known.
[0007]
[Problems to be solved by the invention]
However, among the above-mentioned conventional examples, the chroma key method has problems such as the fact that the background is limited and it cannot be used outdoors, and that it is discolored. In addition, the video mat method has a problem that it is necessary for humans to accurately perform contour specifying work in units of pixels, which requires labor and skill.
[0008]
Also, the difference method with the background image described above makes it difficult to determine whether the subject is a background or a subject in a partial region similar to the background on the subject, and is affected by fluctuations in imaging conditions between the background image and the subject image. There are problems such as being easy and removing shadows caused by the subject is difficult. Furthermore, in order to faithfully extract the boundary line between the background and the subject, the image characteristics (pixel values, etc.) in the vicinity of the contour of the background image and the subject region must be significantly different, which is generally difficult. .
[0009]
Japanese Patent Publication No. 7-16250 is unsuitable for extracting an image of an arbitrary unknown object in that a color model to be extracted is required separately.
[0010]
Furthermore, it is generally difficult to extract a subject with high accuracy regardless of the contrast of the boundary with the background or the like in any method of moving object extraction from moving images or subject extraction from parallax distribution. .
[0011]
Therefore, an object of the present invention is to provide an image extraction apparatus and method that can extract a stable subject image without assuming a remarkable difference in image characteristics between the background and the subject.
[0012]
Another object of the present invention is to provide an image extraction apparatus and method that can ensure a large area of a subject area before area growth with a small number of processing steps and can extract details of contour shapes.
[0013]
Furthermore, the present invention provides an image extraction apparatus and method for performing processing for matching the contour of the mask after region growth with the contour of the actual subject without being affected by the pattern of the background near the contour of the subject. To do other purposes.
[0014]
Another object of the present invention is to provide an image extraction apparatus and method for growing an initial mask stably only in a subject region without depending on region growth conditions, that is, fluctuations in the allowable value of a feature amount difference from a neighboring region. Objective.
[0015]
Furthermore, the present invention suppresses fluctuations in the edge intensity distribution due to differences in shooting conditions between the background image and the subject image or noise, and accurately corrects the contour shape of the subject and the edge of the background portion existing in the subject area. It is another object of the present invention to provide an image extracting apparatus and method for extracting.
[0016]
In addition, the present invention provides an image extraction apparatus and method capable of stably extracting a subject image even when the edge strength forming the boundary between the subject and the background is small and the subject includes a comparatively thin partial shape. For other purposes.
[0017]
It is another object of the present invention to provide an image extraction apparatus and method that stably extract the contour shape of a subject without being affected by the edge distribution of the background portion existing in the vicinity of the subject.
[0018]
Another object of the present invention is to provide an image extraction apparatus and method for automatically restoring an incomplete partial shape after region growth based on the condition of shape continuity and smoothing shape data.
[0019]
Furthermore, the present invention provides an image extraction apparatus and method capable of extracting a stable subject image without being affected by a background pattern and without assuming a significant difference in image characteristics between the background and the subject. The purpose.
[0020]
It is an object of the present invention to provide an image extraction apparatus and method that enable stable and highly accurate extraction of a subject image when performing segmentation by region growth.
[0021]
Further, the present invention provides an image extraction apparatus capable of obtaining a cutout image with stable accuracy without assuming a remarkable difference in image characteristics between the background and the subject when cutting out based on a difference from the background image. Another object is to provide a method.
[0022]
It is another object of the present invention to provide an image extraction apparatus and method capable of subject extraction based on region growth that can faithfully reproduce the contour shape of an object to be extracted.
[0023]
Another object of the present invention is to provide an image extraction apparatus and method that suppresses endless growth of region growth and enables extraction of a cutout region closest to the subject.
[0024]
Furthermore, the present invention provides an image extraction apparatus and method that can obtain stable cutout accuracy even for a subject having a complicated contour shape by suppressing region growth across the edge and region growth from the edge. The purpose.
[0025]
In addition, the present invention provides an image extraction apparatus and method that can obtain stable clipping performance even with respect to noise components such as shadows existing outside the subject (in the background) or the presence of an unclear portion of the contour of the subject. The other purpose is to provide.
[0026]
Furthermore, the present invention provides an image extraction device that realizes region growth in which the contour of the cut out subject shape can closely approximate the true subject shape even when the shape of the partial region extracted in advance does not match the subject contour shape. And another object is to provide a method.
[0027]
Another object of the present invention is to provide an image extraction apparatus and method that realizes automatic extraction of a specific subject from a moving image with high accuracy.
[0028]
Furthermore, another object of the present invention is to provide an image extraction apparatus and method that realizes automatic extraction of a specific subject with high accuracy using a plurality of images from different viewpoint positions.
[0029]
[Means for Solving the Problems]
  In order to achieve the above object, an image extraction method according to claim 1 of the present invention provides a difference between a first image in which a subject to be extracted is recorded and a second image in which a background excluding the subject is recorded. A first step of generating an initial mask of an area to be extracted based on the data, and the generated initial mask area is grown based on a feature amount similarity with a neighboring areaIn addition, if the edge density obtained based on the difference edge data between the first and second edge intensity images respectively extracted from the first and second images is less than a predetermined threshold in the growth direction, the growth is suppressed. DoThe method includes a second step and a third step of extracting the subject from the first image based on the mask region after the region growth.
[0030]
In the image extraction method according to claim 2, in the image extraction method according to claim 1, the first step is a color difference kernel extracted based on a color component difference at each point between the first and second images. Alternatively, an initial nucleus is extracted based on an edge difference nucleus extracted based on difference edge data between first and second edge intensity images respectively extracted from the first and second images.
[0031]
In the image extraction method according to claim 3, in the image extraction method according to claim 1, the third step includes a difference edge between the first and second edge intensity images extracted from the first and second images, respectively. A shape correction process of the mask region after the region growth is performed based on the data, and the subject image in the first image corresponding to the mask region after the correction is extracted.
[0033]
  Claim4In the image extraction method according to claim 2, in the image extraction method according to claim 2, the first and second edge intensity images in the first step are normalized by a predetermined normalization coefficient. .
[0034]
  Claim5In the image extraction method according to claim 2, in the image extraction method according to claim 2, the first and second edge intensity images in the first step are normalized by a predetermined normalization coefficient, and the normalization coefficient Is the maximum value of the first edge intensity image.
[0035]
  Claim6The image extraction method related toA first step of generating an initial mask of an area to be extracted based on difference data between a first image in which a subject to be extracted is recorded and a second image in which a background excluding the subject is recorded; And a third step of extracting the subject from the first image based on the mask region after the growth of the region. The first step includes a color difference nucleus extracted based on a color component difference at each point between the first and second images, or first and second extracted from the first and second images, respectively. Based on the edge difference nucleus extracted based on the difference edge data between the two edge intensity images, the initial nucleus is extracted;A value at a predetermined point of the first and second edge intensity images in the first step is a value normalized by a predetermined normalization coefficient, and the normalization coefficient includes the first edge before normalization. A local edge intensity maximum value in a region of a predetermined size centered on the point of the intensity image is used for the first and second edge intensity images.
[0036]
  Claim7The image extraction method related toA first step of generating an initial mask of an area to be extracted based on difference data between a first image in which a subject to be extracted is recorded and a second image in which a background excluding the subject is recorded; And a third step of extracting the subject from the first image based on the mask region after the growth of the region. The third step performs shape correction processing of the mask region after the region growth based on differential edge data between the first and second edge intensity images extracted from the first and second images, respectively. Extracting the subject image in the first image corresponding to the mask region after the correction;In the shape correction process, a predetermined processing area is set at each point on the contour line of the mask area after the area growth, and the difference edge data or the edge data of the first image is equal to or larger than a predetermined threshold in the processing area. Edge selection processing for extracting a predetermined evaluation function value consisting of a shape continuity evaluation term and a continuity evaluation term of an image feature amount among edge candidates having a value of A smoothing process for smoothing a correction mask area including the correction mask area is provided.
[0037]
  Claim8The image extraction apparatus according to the present invention includes an input unit that inputs a subject image and a background image, an image temporary storage unit that stores the input subject image and a background image, and a difference between the stored subject image and the background image An initial mask generating means for generating initial data of the extraction area based on the data; an area growing means for growing the initial mask area based on a feature amount similarity with a neighboring area; and a mask area after the area growth. Subject image cutout processing means for extracting a subject from the subject image based on the subject imageThe subject image cut-out processing means includes difference edge data extraction means for extracting difference edge data between the subject image and the background image, and contour shaping means for shaping a subject outline based on the difference edge data, The contour shaping means includes threshold processing means for performing threshold processing on the difference edge data or the edge data of the subject image, first continuity evaluation means for evaluating shape continuity with respect to edge candidates remaining after the threshold processing, Second continuity evaluating means for evaluating the continuity of image feature amounts with respect to edge candidates; edge selecting means for selecting edge candidates based on outputs of the first and second continuity evaluating means; and the extracted contour line Or a smoothing means for a correction mask region including the sameIt is characterized by that.
[0038]
  Claim9In the image extracting apparatus according to claim8In the image extracting apparatus according to the present invention, the initial mask generating means extracts color difference nucleus extracting means for performing threshold processing of color component difference at each point between the subject image and the background image, and extracts from the subject image and the background image, respectively. First and second edge intensity images to be extracted, and an edge difference nucleus extracting means for performing threshold processing of difference data between the edge intensity images, and an output of the color difference nucleus extracting means and the edge difference nucleus extracting means And an initial nucleus extracting means for generating an initial nucleus based on the first nucleus.
[0041]
  Claim10In the image extracting apparatus according to claim8In the image extracting apparatus according to the above, the region growing means includes a growth suppression condition determining means and a similarity determining means.
[0042]
  Claim11In the image extraction method according to claim 3, in the image extraction method according to claim 3, the shape correction processing includes setting an initial contour based on the mask region after the initial mask extraction or after the region growth, and minimizing a predetermined evaluation function. And a dynamic contour process for deforming or moving the contour shape.
[0043]
  Claim12In the image extraction method according to claim11In the image extraction method according to the above, the dynamic contour processing is performed on the subject image after being performed on the mask data after the region has been grown.
[0044]
  Claim13The image extraction method according to the present invention includes a partial region extraction step of extracting a partial region that is a part of a subject to be cut out from an input image, and processing a similarity with a neighboring region using the extracted partial region as a core with a threshold value. In the image extraction method, comprising: a region growing step for growing the partial region by step; and a clipping step for extracting the subject image based on the region after the region growth. The threshold value in the region growing step is a value of the input image. It is characterized in that it is set based on the feature amount distribution at each point.
[0045]
  Claim14In the image extraction method according to claim13In the image extraction method according to claim 1, the partial region extracting step extracts the partial region based on a difference between a background image excluding the subject and a subject image including the subject.
[0046]
  Claim15In the image extraction method according to claim13In the image extraction method according to claim 1, the feature amount distribution is an edge distribution of the subject.
[0047]
  Claim16In the image extraction method according to claim13In the image extraction method according to claim 1, the feature amount distribution is a distribution of a maximum growth range set based on the partial region.
[0048]
  Claim17In the image extraction method according to claim13In the image extraction method according to claim 1, the threshold value is set so as to take a lower value at a position where the edge exists than at a position where the edge does not exist.
[0049]
  Claim18In the image extraction method according to claim13In the image extraction method according to claim 1, the threshold value is set to take a high value in an area within the maximum growth range and a low value outside the growth range.
[0050]
  Claim19In the image extraction method according to claim13In the image extraction method according to claim 1, the maximum growth range is obtained as an output when the shape of the partial region is smoothed by a smoothing filter of a predetermined scale.
[0051]
  Claim20In the image extraction method according to claim13In the image extraction method according to claim 1, the input image is a time-series image, and the partial region extraction step is performed based on difference data between image frames at different times of the input image.
[0052]
  Claim21In the image extraction method according to claim13In the image extraction method according to the above, the input image is a plurality of images from a plurality of different viewpoint positions, and the partial region extraction step extracts based on a parallax distribution between the input images.
[0053]
  Claim22The image extraction apparatus according to the present invention includes a partial area extraction unit that extracts a partial area that is a part of a subject to be cut out from an input image, and processes a similarity with a neighboring area using the extracted partial area as a core with a threshold value. In the image extraction apparatus having a region growing means for growing the partial region by means of, and a clipping means for extracting the subject image based on the region after the region growth, the threshold in the region growing means is the threshold value of the input image. It is characterized in that it is set based on the feature amount distribution at each point.
[0054]
DETAILED DESCRIPTION OF THE INVENTION
An embodiment of an image extraction apparatus of the present invention will be described. The image extraction apparatus in the present embodiment is applied to an imaging system.
[0055]
[First Embodiment]
FIG. 1 is a block diagram illustrating a configuration of an imaging system according to the first embodiment. The imaging apparatus 1 performs an imaging optical system 1a including a lens, a diaphragm, and a lens drive control unit 1e, an image sensor 1b, a video signal processing circuit (gamma characteristic, white balance control, exposure condition control, focusing characteristic control, etc. ) 1c, the image recording unit 1d, and the like are main components.
[0056]
The cutout processing device 2 includes an image temporary storage unit 3 including a subject image temporary storage unit 3a and a background image temporary storage unit 3b, a normalized edge strength extraction unit 4, an initial mask region extraction unit 5 that performs threshold processing of difference image data, The area growth unit 6, the contour shaping processing unit 7, the subject image output unit 8, the interface unit 11, and the like are included. An image display device 9 and a terminal device 10 are connected to the outside of the cutout processing device 2.
[0057]
In addition to the configuration shown in FIG. 2, the cut-out processing device 2 exists on a computer storage device as a program shown in the flowchart of FIG. 2 to be described later, or exists as a gate array shown in the same flowchart. Various forms can be taken.
[0058]
FIG. 2 is a flowchart showing the subject cutout processing procedure. First, a subject image and a background image are input from the imaging device 1 (step S11). At this time, in order to increase the subsequent processing speed, the image data is thinned out at an appropriate scale (step S12), and the processing area is set so as to include the subject existence range on the subject image (step S13). Note that the thinning process in step S12 and the process area setting process in step S13 may be omitted.
[0059]
3, 4, 5, 6, 7, and 8 are photographs showing screens on which main results of subject extraction processing are displayed. 3A and 3B are an input subject image and a background image, respectively.
[0060]
Next, initial data extraction processing for extracting the nucleus of region growth is performed. In the present embodiment, a subject edge strength image and a background edge strength image are extracted (step S14), and edge strength normalization processing is performed (step S15). A differential edge nucleus extraction process is performed from the normalized background edge image and subject edge image (step S16).
[0061]
A color difference nucleus extraction process based on a difference in color components (RGB value or hue) is performed for each pixel of the background image and the subject image (step S17). An initial mask area extraction process from the color difference kernel and the edge kernel is performed (step S18).
[0062]
Note that the initial mask area extraction processing is not limited to the above-described flowchart. For example, the correlation coefficient between blocks of a predetermined size defined around each point, the average value of pixel values, the standard deviation value, etc. May be extracted by threshold processing. The contents of each process will be described.
[0063]
In the edge extraction process (step S14), operators such as Sobel, Prewitt, Roberts (Image recognition basics (II), Chapter 15, Mori, Sakakura, Ohmsha, 1990), as well as Canny edge detection operators ( IEEE, Trans. Pattern Analysis and Machine Intelligence. Vol. PAMI-8, pp. 679-698, 1986), Marr-Holdreth edge detection operator (Proc. -217, 1980) may be used.
[0064]
In the edge intensity normalization process (step S15), the intensity maximum value of the subject edge image is used as a common denominator for normalization and applied to each of the subject edge image and the background edge image. However, when the edge that forms the boundary between the subject and the background, that is, when the subject outline is densely distributed (for example, when there are many relatively small partial shapes such as flowers), The maximum edge strength value in a block of a predetermined size centered on each point of the subject edge image is used for each edge strength image at the corresponding location.
[0065]
Note that the maximum value of each of the subject edge image and the background edge image (entire or local) can be used for the purpose of reducing the effect of fluctuations in the imaging conditions even if each normalized denominator is used. 4A and 4B respectively show a subject edge image and a background edge image as a result of edge extraction using the Sobel operator and normalization.
[0066]
In the difference edge nucleus extraction process (step S16), a point that is equal to or larger than a threshold value where the difference absolute value between the normalized edge intensity images of the subject image and the background image is left as an edge nucleus. This threshold value may be appropriately changed according to the image. FIG. 5 shows a result obtained by setting a point having a difference absolute value between normalized edge intensity images equal to or greater than a given threshold as an edge nucleus.
[0067]
In the color difference nucleus extraction process (step S17), a point where any of the similarly defined difference absolute values of the RGB color pixel values is equal to or greater than a predetermined threshold is left as a color difference nucleus. Here, by setting the threshold value relatively high, it is possible to eliminate the influence of fluctuations in pixel values due to noise and differences in shooting conditions, and to remove light shadows and the like.
[0068]
In the initial mask area extraction process (step S18), the color difference nucleus and the edge difference nucleus are merged. However, if the edge is rich at this stage, the remaining background edge pattern can be removed. In other words, once the extracted initial mask area, a point where the normalized edge intensity value of the background image is equal to or greater than a predetermined threshold value may be removed. The processing result is shown in FIG.
[0069]
It is determined whether or not the initial mask area and the subject area almost coincide with each other (step S19). If they almost coincide, the mask extraction process is terminated, and the subject image output process (step S50) or the contour shaping process (to be described later) The process proceeds to step S40). On the other hand, if it is determined that the initial mask area is incomplete, the process proceeds to the next area growth process (step 30).
[0070]
FIG. 9 is a flowchart showing the region growth processing procedure in step S30. In this process, the similarity of the image feature quantity with the neighboring pixel (or area) is determined at a point on the boundary of each nucleus area of the initial mask, and if the similarity is higher than a predetermined threshold, it is regarded as the same subject area. Merge with the mask area.
[0071]
First, at the time of region growth, brightness level and hue threshold values Ith and Hth are input as parameters necessary for similarity determination (step S31). In the present embodiment, the value of each RGB component and the absolute difference value (reciprocal number) of the hue value are used as the feature amount, and similarity determination is performed for each of the eight neighboring pixels (step S32). In particular, in the present embodiment, similarity determination is performed that the absolute difference values of RGB with respect to neighboring pixels are all equal to or smaller than the threshold value Ith, or that the absolute difference value of hue is equal to or smaller than the threshold value Hth. That is, when one of the conditions is used for the growth condition, the same subject area is merged (step S33).
[0072]
Further, when the distribution density of the edge nuclei in the predetermined region in the growth direction is smaller than the predetermined threshold, the growth condition may be set so as to stop the growth in that direction (step S34). For example, in an ungrown region existing in the growth direction (one of the eight neighboring directions), the processing region is set to about 10 pixels from the nearest pixel, and there is no edge difference nucleus in the region or values 1 to 2 When the similarity between the nearest pixel and the target pixel satisfies the growth condition, the region growth is set to the nearest pixel, and the subsequent growth is stopped.
[0073]
Such a deterrent condition may be automatically added when the edge is rich, or may be added at the user's discretion. The results of region growth are shown in FIG.
[0074]
7A and 7B show the result of executing the hole filling process (step S35) for automatically filling the hole of the subject area existing in the ungrown portion after the area growth process, and the intermediate cutout images. .
[0075]
The maximum size of such a hole is given in advance as a standard, or may be determined by the user based on the region growth result. If the contour shape of the subject is incomplete at this stage, contour shaping processing (step S40) is performed.
[0076]
FIG. 10 is a flowchart showing the contour shaping processing procedure. The purpose of this processing is to stably extract the correct shape without being influenced by the background contour pattern in the vicinity of the subject. Here, the difference data of the normalized edge intensity distribution between the subject image and the background image is used.
[0077]
In addition, when a background edge pattern exists inside a subject as in the case of differential edge nucleus extraction, an edge selection process described later is affected thereby, which is a factor that hinders extraction of a correct subject outline. In order to avoid this, it is assumed that the point (region) where the edge intensity value becomes negative by subtracting the background edge image from the subject edge image is removed from the differential edge data. However, the above-described purpose may be removed and the same processing as described below may be performed using the edge intensity data of the subject image.
[0078]
The contents of the contour shaping process will be described in detail. First, the difference data between the normalized subject edge image and the normalized background edge image is removed from the difference data below a predetermined threshold (positive value), and only the subject edge data with high reliability is left (step S41).
[0079]
The contour line tracking process of the mask after the region growth is performed (step S42), and the processing region is orthogonal to the contour tracking direction (the tracking direction is always determined so that the right hand side of the tracking direction is inside the subject region) at each point on the contour line. The direction is set (step S43). The tracked contour data is represented as a function of the track path length (arc length).
[0080]
If there are a plurality of candidate edge points in the local region set in the process of step S43, an edge selection process (step S44) is performed. FIG. 11 is an explanatory diagram showing edge selection processing. Black circle points Qs2, Qs1, Qs, etc. represent points selected by the edge selection process, and white circle points P1, P2, P3, etc. represent edge candidate points to be selected.
[0081]
The processing area includes a point to be subjected to edge selection and indicates an area set in a direction orthogonal to the contour tracking direction. Here, an energy function F for determining two of a feature amount (for example, RGB value) continuity P and contour shape continuity S on the subject side of a candidate edge (white circle point) is set, and energy is evaluated by evaluating this. Select the candidate edge with the smaller function value.
[0082]
In this embodiment, the feature amount continuity P is a subject of a contour line including candidate edges (however, it is contour data of about a four-point sequence consisting of already selected edge data and candidate edges connected thereto). It is expressed as a variation amount (value between inner neighboring pixels) in the contour tracking direction of the inner feature amount (RGB value), and is defined as, for example, Equation 1.
[0083]
[Expression 1]
P = dR / MR + dG / MG + dB / MB
Here, dR, dG, and dB are pixels adjacent to the inner side of the RGB value (for example, points Qs, Qs-1 and Pi when each point is a point sequence on the subject outline, and are orthogonal to the outline tracking direction. , MR, MG, and MB each represent a maximum variation (difference between the maximum value and the minimum value) in each region. Therefore, here, P is the sum of the normalized fluctuation amounts of the RGB components. However, such normalization processing is not a necessary condition for feature continuity evaluation.
[0084]
The shape continuity S is evaluated by the curvature of the contour line in the local region including the candidate edge. Since the contour line is given by the arc length s, it is expressed as Equation 2.
[0085]
[Expression 2]
S = (x_ss+ Y_ss)^0.5
Here, xss and yss represent second-order differential values with respect to the arc length s of the contour coordinate value data. Note that S may be given as a primary differential value related to a contour sampling point sequence including candidate edges of contour data. By discretizing Equation 2 using selected and unselected data, S is specifically given by Equation 3.
[0086]
[Equation 3]
S = {(ex-3 * EdgeX (s) + 3 * EdgeX (s-1) -EdgeX (s-2))²+ (Ey-3 * EdgeY (s) + 3 * EdgeY (s-1) -EdgeY (s-2))²}^0.5
Here, EdgeX (s), EdgeY (s), etc. represent contour data that has already been determined (or as set initial values), and ex, ey are candidate edges (that is, points P1, P2, P3), respectively. 1) X coordinate and Y coordinate are shown. If the contour tracking direction is up or down (Y-axis direction), the search area for the candidate edge is set in the direction orthogonal to the tracking direction, so ex becomes the edge candidate coordinate value to be determined (either left or right) Ey is a variable factor).
[0087]
The energy function F is F = P + aS, a is a factor that gives a weight between P and S (0 ≦ a ≦ 1), and can be regarded as a normalization coefficient of a kind of normalization processing.
[0088]
On the other hand, when there is no candidate edge in the local region set in step S43, that is, when there is no subject edge candidate with high reliability, a contour edge on the mask or a predicted edge position in the contour tracking direction (for example, tracking) If the direction is up, the X coordinate is fixed and the Y coordinate value is increased by one pixel, etc.) as an edge candidate point, and the energy function is evaluated to select the lower value or simply The point on the mask is the selected edge.
[0089]
After the process in step S44, the extracted mask data is smoothed (step S45). Here, median filter processing of contour data (one-dimensional) and two-dimensional mask data is performed. As a result, when the above-described edge selection process leaves an incomplete partial shape (for example, as a result of the process in the case where there is no highly reliable candidate edge, discontinuous irregularities around the subject contour line or around it, or When the dither pattern remains, etc.), it can be smoothed to improve the degree of approximation of the subject contour shape.
[0090]
Note that processing such as a median filter may be applied recursively. Needless to say, the smoothing filtering process is not limited to the median filter. Based on the mask data obtained by the contour shaping process (step S40), the subject image output process (step S50) is performed, and the result is output to the image display device 9 or a printer as the subject image output unit 8. A subject image is shown in FIG.
[0091]
[Second Embodiment]
In the imaging system according to the second embodiment, after initial mask region extraction and region growth processing, contour shaping processing is performed by dynamic contour method (M. kass et al. “Snakes: Active Contour Models,” International Journal of Computer Vision , vlo. 1, pp. 321-331, 1987).
[0092]
The dynamic contour is an initial contour set in the vicinity of the object that moves and deforms so as to minimize the evaluation function as shown below, and finally converges to the contour of the object or its envelope. is there. In the present embodiment, dynamic contour shaping processing (step S400) is performed on the mask image (value 0/1 data), and contour shaping processing (step S40) is further performed on the subject image.
[0093]
FIG. 12 is a flowchart showing a dynamic contour shaping process procedure. Specifically, the initial contour is set based on the mask area obtained after the initial mask area extraction and the area growth process as in the first embodiment (step S401). Examples of the initial contour include a mask boundary line enlarged by a predetermined number of times around its center of gravity, or a mask boundary line set around a mask region using an instruction selection device such as a mouse.
[0094]
Next, the following active contour processing is performed on the mask data (step S402). For a contour line v (s) = (x (s), y (s)) expressed using a parameter s (typically arc length) for describing the coordinates of each point on the contour, Equation 4 A contour line shape v (s) that minimizes the evaluation function E shown in FIG.
[0095]
[Expression 4]

I (v (s)) represents the luminance level on v (s), and α (s), β (s), and w0 are appropriately determined by the user. The deformation and movement of the contour are performed while sequentially updating in time series. At each step, a point that minimizes the above function is selected from a set of points within a movable range (predetermined neighboring area) at each point on the contour line, or a contour that minimizes the above evaluation function is selected by a variational method. It is processed by solving Euler's equation using.
[0096]
In this way, by performing dynamic contour processing on the mask data, the mask shape after region growth is smoothed and not grown while avoiding erroneous convergence to the background pattern near the subject on the subject image. The contour shape of the region can be modified. In particular, the correction function is exhibited when a smooth continuous shape corresponding to a subjective contour line is generated in a defective portion of the shape.
[0097]
The contour shape after the energy function has converged to the minimum value is considered to be sufficiently close to the shape of the subject in many cases. Therefore, this is set as an initial contour (step S403), and the above-described dynamic contour processing is similarly applied to the subject image (step S404), and the finally converged contour is set as a cutout mask region (step S405). ).
[0098]
Note that the dynamic contour processing on the mask data may not be essential, but in such a case, the processing in steps S402 and S403 may be omitted.
[0099]
Further, after the initial mask area is extracted, the initial contour may be set so as to surround the mask area by omitting the area growth process, and the above-described dynamic contour processing may be performed on the mask data. In this case, the contour shape that has been substantially converged is used as an initial contour, and contour shaping processing is further performed on the subject image to extract a detailed shape. Depending on the conditions, the dynamic contour processing may be performed on the subject image after extraction of the initial mask area.
[0100]
[Third Embodiment]
FIG. 13 is a block diagram illustrating a configuration of an imaging system according to the third embodiment. In the present embodiment, two subject images are used as an input image: a subject image showing a subject to be cut out and a background image excluding the subject.
[0101]
The imaging apparatus 1 performs an imaging optical system 1a including a lens, a diaphragm, and a lens drive control unit 1e, an image sensor 1b, a video signal processing circuit (gamma characteristic, white balance control, exposure condition control, focusing characteristic control, etc. ) 1c, the image recording unit 1d, the interface unit 1f, and the like are main components.
[0102]
The cutout processing device 2 includes an image temporary storage unit 3 including a subject image temporary storage unit 3a and a background image temporary storage unit 3b, an initial mask region extraction unit 5 that performs threshold processing of difference image data, a region growth unit 6, and a subject image output. Part 8 and interface part 11. An image display device 9 and a terminal device 10 are connected to the outside of the cutout processing device 2.
[0103]
14 and 15 are flowcharts showing the subject cutout processing procedure. In addition to the above-described configuration, the clipping processing device is one that exists on the storage device of the computer as the program shown in the flowcharts of FIGS. It can take various forms, such as what is.
[0104]
First, a subject image and a background image are input from the imaging device 1 (step S111). At this time, in order to increase the subsequent processing speed, the image data is thinned out at an appropriate scale (step S112), and the processing area is set so as to include the subject existing range on the subject image (step S113). Note that the thinning-out process in step S112 and the process area setting process in step S113 may be omitted.
[0105]
16 and 17 are photographs showing screens on which main intermediate results of the subject extraction process are displayed. In this embodiment, as in the first embodiment, (a) and (b) in FIG. 3 are set as an input subject image and a background image, respectively.
[0106]
Subsequently, an initial nucleus data extraction process for extracting the region growth nucleus is performed (steps S116, S117, and S118). This processing mainly performs initial kernel extraction by performing threshold processing on the difference absolute value of each component, for example, based on the difference in color components (RGB value or hue, saturation) for each pixel between the background image and the subject image. Here, by setting the threshold value relatively high, it is possible to eliminate the influence of fluctuations in pixel values due to noise and differences in shooting conditions, and to remove light shadows and the like.
[0107]
However, in the case where the imaging device 1 is not fixed to a tripod or the like and is taken by hand, and the background image and the subject image are each taken by exposure conditions and automatic focusing settings, the following processing is performed before threshold processing is performed. Need to do.
[0108]
That is, a conversion parameter group of geometric or RGB components between corresponding points between the subject image and the background image is extracted (step S114), and alignment between two images (shift, rotation, magnification conversion), Further, level adjustment of each color component (RGB value) (estimation of a non-linear function for correction using the least square method or the like) is performed using the extracted conversion parameter group (step S115).
[0109]
In addition, as another processing method, a correlation coefficient between blocks of a predetermined size defined around each point, or a statistic such as an average value or standard deviation value of pixel values may be extracted by threshold processing. A mask region obtained as a result of the initial nucleus extraction is shown in FIG. 16A (the subject region is shown in black). The above processing is performed by the initial mask region extraction unit 5 of the cutout processing device 2 of FIG. Instead of the initial mask area extraction unit 5, it may be executed by a program inside the computer of the terminal device 10.
[0110]
In a general combination of background and subject, the region of the subject to be cut out at this stage cannot be completely extracted. That is, if there is an area where the RGB level or local statistics (average value, standard deviation, etc.) are partially similar at the same location between the subject image and the background image, the partial area is not processed after threshold processing. It remains as an extraction area. Therefore, in the next region growth process, they are restored.
[0111]
In the area growing process, the similarity of the image feature quantity with the neighboring pixel (or area) is obtained at a point on the boundary of each core area of the initial mask, and if this is higher than a predetermined threshold, the mask area is regarded as the same subject area. To merge. The above processing is performed by the initial region growing unit 6 in the cutout processing device 2 of FIG. Instead of the initial region growing unit 6, it may be executed by a program inside the computer of the terminal device 10.
[0112]
In the present embodiment, first, before performing region growth, the edge intensity distribution extraction processing (step S119) of the subject image, the binarization processing by the threshold processing (step S120), and the initial nucleus growth range are limited. The maximum region growth range setting process (step S121) is performed based on the region data of the initial nucleus.
[0113]
Specifically, the minimum value of Y (vertical) component at each point in the horizontal (X) direction of the region where the initial nucleus exists, the maximum value (MinY (X), MaxY (X)), and the vertical (Y) direction at each point. X (horizontal) component minimum value and maximum value (MinX (Y), MaxX (Y)) are obtained, and smoothing (low-pass filter) is performed on each data.
[0114]
In the present embodiment, by applying a median filter of a predetermined size, sudden fluctuations can be suppressed, and a smooth growth range almost along the contour shape of the subject can be obtained (see FIG. 16B). .
[0115]
Next, the initial threshold values Ith and Hth of the difference from the brightness level and the vicinity of the hue are input as characteristic parameters necessary for similarity determination (step S122). These may be automatically set based on statistics such as an average value and standard deviation of difference (absolute difference value) data relating to parameters between the background image and the subject image. Further, based on the maximum growth range and the edge intensity distribution, each threshold value is variably set as follows according to the position in the image (step S123).
[0116]
The threshold is set high in the maximum growth range (MinY (X), MaxY (X)), (MinX (Y), MaxX (Y)), and is set low outside each range. Specifically, when the point (X, Y) is in any of the growth ranges, the threshold value is set to a high threshold value (for example, an initial threshold value); otherwise, the threshold value is low (for example, 10% of the initial threshold value). ).
[0117]
As the threshold distribution function, an arbitrary function that decreases as the distance from the boundary of the maximum growth range increases may be used. Other values for this function may be discrete values, and the same value may be given for the distance from the boundary of the predetermined range. Furthermore, as another method of giving a threshold value, the threshold value for region growth is set in the vertical direction and the horizontal direction (Ithy, Ithx), and Ithy is set in the vertical growth range (MinY (X), MaxY (X)). It may be set so that Ithx is high in the lateral growth range (MinX (Y), MaxX (Y)).
[0118]
In the present embodiment, the edge intensity distribution of the subject image is further binarized with a predetermined threshold, and the threshold is lowered at and near the position where the edge exists. The threshold distribution is the lowest at the edge position (for example, a value of 0 completely suppresses the growth of the region that intersects the edge), and in the vicinity thereof exhibits a slight increase with respect to the distance from the edge, or For example, an edge that has a uniformly low value in the vicinity thereof is used.
[0119]
FIG. 18 is an explanatory diagram showing a region growth threshold distribution based on a feature amount difference from neighboring pixels set based on the edge distribution and the maximum region growth range. The horizontal axis X (Y) represents the horizontal position on the cross section of the subject image at the vertical position Y, and the maximum growth ranges MinX (Y) and MaxX (Y) are also the horizontal growth ranges on the cross section. The minimum and maximum values are shown. Further, the threshold values Thh and Thl represent the maximum value and the minimum value of the threshold value, respectively. Note that the boundary line of the maximum region growth range may be superimposed on the input image and displayed on the display so that the user can set an appropriate smoothing filter size based on this.
[0120]
The above threshold setting method makes it possible to achieve robustness and stabilization of the region growth result with respect to the difference between the initial threshold values (Ith, Hth) (the change in shape along the contour shape of the subject is small). Further, even when the maximum growth range is different from the outer contour shape of the subject, the threshold values (Ith, Hth) can be set roughly so that they match as a result of the region growth.
[0121]
Next, similarity determination processing (step S124) is performed on neighboring pixels based on the feature amount. In particular, in the present embodiment, the similarity determination is made that the absolute difference values of RGB with respect to neighboring pixels (the unmasked area at that time) are all equal to or less than the threshold value, or that the absolute difference value of the hue is equal to or less than the threshold value. That is, it is used for the growth condition, and when either one is established, the neighboring pixels are merged as the same subject area (see step S125, FIG. 17A).
[0122]
The feature quantities used for similarity determination are not limited to these. For example, a low-order feature quantity such as saturation, or a partial shape of a local line segment including an edge as a high-order feature quantity (segment orientation, etc.) ), A local spatial frequency, and a feature amount obtained by statistically processing low-order feature amounts such as RGB are preferably used.
[0123]
Further, the region growth merge processing is not necessarily performed for eight neighboring pixels, and neighboring regions obtained by other methods may be used. Further, a hole filling process (step S126) for automatically filling a hole of a predetermined size or less with respect to the mask data after the region growth is performed.
[0124]
This hole filling process is performed on the binarized mask data regardless of the similarity, homogeneity, and the like of the subject image data, that is, the neighborhood of the image feature amount. As a result, the obtained grown area is set as a subject cutout mask area, the corresponding area is extracted from the subject image (step S127), and the image data is output to the display (or as an image file) (step S128). The process is completed (see FIG. 17B).
[0125]
In the subject image extraction process (step S127), the subject image corresponding to the mask may be extracted after performing the boundary line smoothing process or the boundary line correction process. The cutout image output process (step S128) is performed by the subject image output unit 8 of the cutout processing device 2 in FIG. Instead of being performed by the subject image output unit 8, it may be executed as a program on a computer.
[0126]
[Fourth Embodiment]
FIG. 19 is a block diagram illustrating a configuration of an imaging system according to the fourth embodiment. In the figure, reference numeral 51 denotes an image input device, which indicates an imaging device or an image database unit. The imaging device is not particularly limited, such as a video camera or a compound eye type camera.
[0127]
52 is a primary feature extraction unit that extracts primary feature data in an image, and 53 is a feature homogeneity evaluation unit that evaluates the homogeneity of feature data. 54 is an image storage unit, 55 is a region dividing unit that divides a region based on the homogeneity of feature data, 56 is a divided image generating unit, and 57 is a region growing unit based on secondary features. Reference numeral 58 denotes a display device. Reference numeral 59 denotes a divided area instruction device for instructing selection of divided areas, and represents a pointing device such as a mouse.
[0128]
In the present embodiment, region growth means that regions are divided and merged by evaluating the homogeneity of feature amounts in a predetermined region in an image. As features in the present embodiment, primary feature data in an image is extracted, and image regions that can be extracted are roughly extracted by performing image region division based on the distribution. As the next step, region growth based on secondary feature amounts (features having different types, attributes, etc. from the primary features) with the candidate region as the core for finer extraction is performed in the same manner as in the third embodiment. .
[0129]
Further, it is assumed that the region extracted based on the primary feature data includes image information necessary for performing region growth based on the secondary feature. In the present embodiment, the secondary feature may basically be different from the primary feature, and necessarily has a geometric structure, or a higher-order feature obtained by processing the luminance distribution and the color component. For example, color component information is preferably used. However, even if the type is the same as the primary feature (in this embodiment, a motion vector or a parallax vector), there is no problem as long as fine image division by region growth is possible.
[0130]
The purpose of primary feature data detection is to perform fast selection of the region that will be the growth core when performing region growth processing, allowing the operator to make rough selections and instructions when actually extracting a specific target image region. Or to facilitate the automatic extraction process.
[0131]
When inputting time-series image data from the image input device 51 as primary feature data, the motion vector of each point on the screen, and when inputting an image from a compound eye camera, the distribution of the disparity vectors of corresponding points between the left and right images Is used. Although the extraction accuracy of the primary feature extraction unit 52 may be slightly inferior to the accuracy (resolution) of the secondary feature extraction unit used later, it is desirable that extraction is possible at high speed. For this reason, dedicated hardware for extracting primary features and integrating homogeneous regions may be set.
[0132]
FIG. 20 is a block diagram showing a specific configuration of the imaging system of FIG. Note that the parallax vector (motion vector) detection algorithm is not the main point of the present invention, and therefore the description thereof is omitted here (see “Robot Vision” by Taniuchida, Shosodo, etc.).
[0133]
In the present embodiment, as a measure of the feature homogeneity, it is represented by a variance relating to the size and direction of the primary feature amount (motion vector, parallax vector, etc.) in a predetermined region. By appropriately setting the size of the area (block) to be evaluated based on the target in-screen size, reduction in processing time and efficiency can be achieved. That is, a divided area of about 1/10 to 1/10 is typically used as compared with the vertical and horizontal size of the target screen. This value may be appropriately set in advance on the operator side.
[0134]
When the homogeneity value (dispersion value in the area, etc.) is smaller than a predetermined threshold, the area dividing unit 55 determines that the area is homogeneous and connects the areas where the homogeneity representative value is within the predetermined allowable range. By separating out-of-range regions, a homogeneous block region (s) that can be represented by a constant primary feature value (parallax vector, motion vector, etc.) is formed. This process can also be regarded as one of so-called area growth processes, but at this stage, the conditions (constraints) for area growth similar to those in the third embodiment are not given.
[0135]
FIG. 21 shows an example of an input image captured by a compound eye camera (a), a rough region segmentation result (b) based on the magnitude of a disparity vector, and region growth based on secondary features such as color components (and It is explanatory drawing which shows the image cutting-out result (c) by division | segmentation.
[0136]
Note that, as described above, the divided region (initial nucleus) shown in FIG. 21B does not necessarily reflect the actual shape of the object as in the first embodiment. However, it is desirable that the background area does not coexist in the divided area (initial nucleus).
[0137]
For this reason, after dividing the region (after initial kernel extraction), the result of performing the mask thinning (erosion) process by reducing each region at a certain rate or by the action of the morphological operator is shown as the divided region (initial Nuclear).
[0138]
Also, if the background area is partially mixed, first divide the initial nucleus area based on color components, etc., delete the small area including the outline of the initial nucleus, and then perform area growth, etc. A more precise cutting of the shape is possible.
[0139]
In the divided image (initial nuclear image) generation unit 56, image data having different attributes (for example, different colors, different hatching patterns, etc.) is assigned to the plurality of regions divided based on the primary feature data, The image is displayed on the display device 58 so as to overlap the input image data. Specifically, labeling is performed on the divided areas, and processing such as filling the same label area with a unique pattern (or color) is performed.
[0140]
By doing so, the operator can easily distinguish candidate areas to be instructed visually, and the division area indicating device 59 can easily specify the extraction target. A mouse or the like is typically used as the divided area instruction device 59, but is not necessary when a touch panel display device is used. The instruction selection is not necessary even when there is a moving object in the image or there is only one region of the predetermined range of the parallax value. Further, all of the plurality of extracted initial nuclei may be used as region growth nuclei.
[0141]
The region growing unit 57 extracts a secondary feature (for example, hue, RGB value, etc.) from the vicinity of the initial core region extracted and selected based on the primary feature data, and is based on the secondary feature. Threshold distribution for evaluating similarity between the region growing unit 572, a region connecting unit (filling means) 573 that connects the grown regions, a maximum region growing range setting unit 574, and neighboring pixels of the secondary feature amount The setting unit 575 is configured.
[0142]
The processing contents in the setting units 574 and 575 are the same as those in the third embodiment. However, as a specific method of threshold distribution setting in the present embodiment, for example, integration by instruction selection (or automatic processing) is performed. In addition, discontinuous parts of primary feature values (parallax vectors, motion vectors, etc.) in the initial core area are treated as discontinuous parts of secondary feature values (RGB values, hues, etc.) You may set the threshold value of similarity evaluation low.
[0143]
FIG. 22 is an explanatory diagram showing setting of the threshold distribution. An example of setting a threshold (allowable value of difference in feature quantity with neighboring pixels) distribution including the above cases is shown. A parallax edge is a rate of change of a parallax vector that is greater than a predetermined threshold in the vicinity including that point. Means a place. Here, a so-called edge intensity distribution (obtained by applying a differential operator such as the SOBEL operator to the intensity distribution or the intensity distribution of each color component) and the parallax edge in the image are observed at the same point. If the threshold is the lowest (Thl) and only one of them is observed, the medium threshold (Thm), no edge is observed, and the highest threshold for an area that does not belong to any edge (Thh) is given.
[0144]
【The invention's effect】
  The present inventionPaintingAccording to the image extraction method, an initial mask of an area to be extracted is generated based on difference data between a first image in which a subject to be extracted is recorded and a second image in which a background excluding the subject is recorded. A first step, a second step of growing the generated initial mask region based on a feature amount similarity with a neighboring region, and extracting the subject from the first image based on the mask region after the region growth Therefore, it is possible to perform subject extraction by eliminating the influence of noise and fluctuations in photographing conditions and automatically removing light shadow portions. Further, it is possible to extract a subject area including an area having image characteristics similar to the background image inside the subject.
In the second step, an edge density obtained based on differential edge data between the first and second edge intensity images extracted from the first and second images, respectively, is a predetermined threshold value or less in the growth direction. In this case, since the growth is suppressed, even if the region growth conditions are relaxed or set roughly, the region growth outside the subject is suppressed and the subject can be extracted with high accuracy. Even when the initial mask area includes an area other than the subject (such as a shaded portion), growth from such an area can be suppressed.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of an imaging system according to a first embodiment.
FIG. 2 is a flowchart showing a subject cutout process procedure;
FIG. 3 is a photograph showing a halftone image on which a subject image and a background image are displayed.
FIG. 4 is a photograph showing a halftone image on which a main halfway result of subject cutout processing is displayed.
FIG. 5 is a photograph showing a halftone image on which a main halfway result of subject cutout processing is displayed.
FIG. 6 is a photograph showing a halftone image on which a main halfway result of subject cutout processing is displayed.
FIG. 7 is a photograph showing a halftone image on which a main halfway result of subject extraction processing is displayed.
FIG. 8 is a photograph showing a halftone image on which a result of subject cutout processing is displayed.
FIG. 9 is a flowchart showing a region growth processing procedure in step S30.
FIG. 10 is a flowchart showing a contour shaping process procedure;
FIG. 11 is an explanatory diagram showing edge selection processing.
FIG. 12 is a flowchart showing a dynamic contour shaping processing procedure.
FIG. 13 is a block diagram illustrating a configuration of an imaging system according to a third embodiment.
FIG. 14 is a flowchart showing a subject cutout processing procedure;
15 is a flowchart showing a subject cutout processing procedure continued from FIG. 14;
FIG. 16 is a photograph showing a halftone image on which a main halfway result of subject cutout processing is displayed.
FIG. 17 is a photograph showing a halftone image on which a main halfway result of subject cutout processing is displayed.
FIG. 18 is an explanatory diagram showing a region growth threshold distribution based on a feature amount difference from neighboring pixels set based on an edge distribution and a maximum region growth range;
FIG. 19 is a block diagram illustrating a configuration of an imaging system according to a fourth embodiment.
20 is a block diagram illustrating a specific configuration of the imaging system in FIG. 19;
FIG. 21 shows an example of an input image captured by a compound eye camera (a), a rough region segmentation result based on the magnitude of a disparity vector (b), and region growth based on secondary features such as color components ( FIG. 10 is an explanatory diagram showing an image cutout result (c) by (and division).
FIG. 22 is an explanatory diagram showing setting of a threshold distribution.
[Explanation of symbols]
1 Imaging device
2 Cutout processing device
3 Image temporary storage
4 Normalized edge strength extraction unit
5 Initialization mask area extraction unit
6, 57 Region growth section
7 Contour shaping processor
51 Image input device
52 Primary feature extraction unit
53 Feature homogeneity evaluation section
55 Region division
56 Divided image generator

Claims (22)

A first step of generating an initial mask of an area to be extracted based on difference data between a first image in which a subject to be extracted is recorded and a second image in which a background excluding the subject is recorded;
The generated initial mask region is grown based on the feature amount similarity with the neighboring region, and differential edge data between the first and second edge intensity images respectively extracted from the first and second images is used. A second step of inhibiting growth if the edge density obtained based on the growth direction is below a predetermined threshold in the growth direction ;
An image extraction method comprising: a third step of extracting the subject from the first image based on the mask region after the region growth.   The first step includes a color difference nucleus extracted based on a color component difference at each point between the first and second images, or first and second extracted from the first and second images, respectively. The image extraction method according to claim 1, wherein an initial nucleus is extracted based on an edge difference nucleus extracted based on difference edge data between edge intensity images.   The third step performs shape correction processing of the mask region after the region growth based on difference edge data between the first and second edge intensity images respectively extracted from the first and second images, 2. The image extraction method according to claim 1, wherein the subject image in the first image corresponding to the corrected mask region is extracted.   3. The image extraction method according to claim 2, wherein the first and second edge intensity images in the first step are normalized by a predetermined normalization coefficient.   The first and second edge intensity images in the first step are normalized by a predetermined normalization coefficient, and each of the normalization coefficients is a maximum value of the first edge intensity image. The image extraction method according to claim 2, wherein: A first step of generating an initial mask of an area to be extracted based on difference data between a first image in which a subject to be extracted is recorded and a second image in which a background excluding the subject is recorded;
A second step of growing the generated initial mask region based on a feature amount similarity with a neighboring region;
A third step of extracting the subject from the first image based on the mask region after the region growth,
The first step includes a color difference nucleus extracted based on a color component difference at each point between the first and second images, or first and second extracted from the first and second images, respectively. Based on the edge difference kernel extracted based on the difference edge data between the edge strength images, the initial nucleus is extracted,
A value at a predetermined point of the first and second edge intensity images in the first step is a value normalized by a predetermined normalization coefficient, and the normalization coefficient includes the first edge before normalization. images extracted how to characterized in that it is used topically edge intensity maximum in the region of a predetermined size centering on a said point of intensity images for the first and second edge intensity images. A first step of generating an initial mask of an area to be extracted based on difference data between a first image in which a subject to be extracted is recorded and a second image in which a background excluding the subject is recorded;
A second step of growing the generated initial mask region based on a feature amount similarity with a neighboring region;
A third step of extracting the subject from the first image based on the mask region after the region growth,
The third step performs shape correction processing of the mask region after the region growth based on difference edge data between the first and second edge intensity images respectively extracted from the first and second images, Extracting the subject image in the first image corresponding to the corrected mask region;
In the shape correction process, a predetermined processing area is set at each point on the contour line of the mask area after the area growth, and the difference edge data or the edge data of the first image is equal to or larger than a predetermined threshold in the processing area. Edge selection processing for extracting a predetermined evaluation function value consisting of a shape continuity evaluation term and a continuity evaluation term of an image feature amount among edge candidates having a value of An image extraction method comprising a smoothing process for smoothing a correction mask region including the correction mask area. An input means for inputting a subject image and a background image;
Image temporary storage means for storing the input subject image and background image;
Initial mask generation means for generating initial data of the extraction region based on the difference data between the stored subject image and the background image;
Region growing means for growing the region of the initial mask based on the feature amount similarity with the neighboring region;
Subject image cutout processing means for extracting a subject from the subject image based on the mask region after the region growth ,
The subject image cut-out processing means includes difference edge data extraction means for extracting difference edge data between the subject image and the background image, and contour shaping means for shaping a subject outline based on the difference edge data,
The contour shaping means includes threshold processing means for performing threshold processing of the difference edge data or the edge data of the subject image, first continuity evaluation means for evaluating shape continuity for edge candidates remaining after the threshold processing, Second continuity evaluating means for evaluating the continuity of image feature amounts with respect to edge candidates; edge selecting means for selecting edge candidates based on outputs of the first and second continuity evaluating means; and the extracted contour line or images extractor you characterized in that a smoothing means for correcting the mask region containing it. The initial mask generation means includes color difference nucleus extraction means for performing threshold processing of color component difference at each point between the subject image and the background image;
Edge difference nucleus extraction means for extracting first and second edge intensity images respectively extracted from the subject image and the background image, and performing threshold processing of difference data between the edge intensity images;
9. The image extracting apparatus according to claim 8, further comprising initial nucleus extracting means for generating an initial nucleus based on outputs of the color difference nucleus extracting means and the edge difference nucleus extracting means. 9. The image extracting apparatus according to claim 8 , wherein the region growing unit includes a growth suppression condition determining unit and a similarity determining unit. The shape correction processing includes initial contour setting processing based on the mask region after the initial mask extraction or region growth, and dynamic contour processing for deforming or moving the contour shape so as to minimize a predetermined evaluation function. The image extracting method according to claim 3, further comprising : 12. The image extraction method according to claim 11 , wherein the dynamic contour processing is performed on the subject image after being performed on the mask data after the region has been grown. A partial region extraction step of extracting a partial region that is a part of the subject to be cut out from the input image;
A region growing step of growing the partial region by processing the similarity with a neighboring region with a threshold value using the extracted partial region as a nucleus;
In an image extraction method comprising: a cutout step of extracting the subject image based on the region after the region has been grown,
The image extraction method, wherein the threshold value in the region growing step is set based on a feature amount distribution at each point of the input image. The image extraction method according to claim 13, wherein the partial region extraction step extracts the partial region based on a difference between a background image excluding the subject and a subject image including the subject. The image extraction method according to claim 13 , wherein the feature amount distribution is an edge distribution of the subject. The image extraction method according to claim 13 , wherein the feature amount distribution is a distribution of a maximum growth range set based on the partial region. 14. The image extraction method according to claim 13 , wherein the threshold value is set so as to take a lower value at a position where the edge exists than at a position where the edge does not exist. The image extraction method according to claim 13 , wherein the threshold value is set to take a high value in a region within the maximum growth range and a low value outside the growth range. The image extraction method according to claim 13 , wherein the maximum growth range is obtained as an output when the shape of the partial region is smoothed by a smoothing filter of a predetermined scale. The image extraction method according to claim 13, wherein the input image is a time-series image, and the partial region extraction step extracts based on difference data between image frames at different times of the input image. The image extraction method according to claim 13, wherein the input image is a plurality of images from a plurality of different viewpoint positions, and the partial region extraction step extracts based on a parallax distribution between the input images. Partial area extraction means for extracting a partial area that is part of the subject to be cut out from the input image;
A region growing means for growing the partial region by processing the similarity with a neighboring region as a threshold using the extracted partial region as a nucleus;
In an image extraction apparatus having a cutout unit that extracts the subject image based on the region after the region growth,
The image extraction apparatus characterized in that the threshold value in the region growing means is set based on a feature amount distribution at each point of the input image.

Images