JP2004326192A - Image management system, image management method, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To facilitate management of a photograph by combining the photographs taken with the object imaged on the photograph. <P>SOLUTION: In a system facilitating photograph retrieval, by combining the person who is photographed with the taken photograph, the person to be photographed has information to acquire the position, the system is integrated with a system for processing the camera position, lens direction, focal length, angle of field, stop down value, and the positional information of the object at taking of a photograph, recognizes who is photographed, and practically performs object recognition, by deleting and editing the recognition result by a person taking the photograph thereafter. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

【０１０４】
但し、Ａ_ｉｊはｉ行ｊ列目のセルが持つ撮影空間の重み、Ｃ_ｉはカメラ位置の重み、Ｄ_ｊはレンズ方向の重み、Ｏ_ｋｓは被写体の確からしさをそれぞれ表している。これら重みＡ_ｉｊ、Ｃ_ｉ、Ｄ_ｊ、Ｏ_ｋｓはそれぞれ値が規格化されているものとする。
【０１０５】
図２２には、撮影画像の中から認識対象インデックスが取得された様子を示している。図１２を参照しながら、写真撮影時に撮影画像とともに撮影状態が取得されることを既に説明した。参照番号５７は認識対象インデックスを示している。認識対象インデックスが取得された場合、撮影ログに加えて、認識種類として、人物５１０と、場所５２０、イベント５３０が追加される。また、参照番号５６は、個々の認識単位インデックスに対して設定されたランキング・ポイント値を示している。
【０１０６】
図２３には、認識対象インデックスを記述するデータ・フォーマットの構成例を示している。
【０１０７】
図１３を参照しながら、撮影状態を記述するためのデータ・フォーマットの構成について既に説明した。図１３に示す例では、ｘｍｌ形式で撮影状態が記述され、撮影画像とのリンクと、撮影時刻と、撮影場所と、撮影方向が含まれている。
【０１０８】
図２３では、このｘｍｌデータに対してさらに、撮影画像に含まれる認識インデックスとそのポイント値が記載されるとともに、撮影時間と認識位インデックスから取り出されたイベントとそのポイント値が記載されている。図示の例では、認識対象を認識種類毎に記述するタグ・フィールドが設けられ、認識種類「人物（ｐｅｒｓｏｎ）」のタグ・フィールド５１０には、撮影画像に含まれる認識対象としての「なっち」、「ひかり」がそれぞれのポイント値０．７２、０．３２ともにタグ情報５１１、５１２として記載されている。また、認識種類「場所（ｌｏｃａｔｉｏｎ）」のタグ・フィールド５２０には、撮影画像に含まれる認識単位としての平安神宮、神宮通り、京都がそれぞれのポイント値０．６３、０．２８、０．１９とともにタグ情報５２１、５２２、５２３として記載されている。また、認識種類「イベント（ｅｖｅｎｔ）」のタグ・フィールド５３０には、認識単位「平安神宮」と撮影時間に基づいて取り出されたイベント「時代祭」と、認識単位「京都」と撮影時間に基づいて取り出されたイベント「紅葉」がそれぞれのポイント値０．６３、０．１９とともにタグ情報として記載されている。
【０１０９】
本実施形態に係る画像管理システムによれば、撮影時刻、撮影状態、撮影時の位置測定や方向取得に包含される誤差情報などに基づいて、撮影画像に含まれる被写体の認識を行なうとともに、各被写体に対するランキング・ポイントの付与、被写体に関連するイベントの取得並びのランキング・ポイントの付与が行なわれる。そして、ユーザ側では、ランキング・ポイントに基づいた優先順位で被写体のリストが提示されるので、これに基づいて写真の管理を好適に行なうことができる。
【０１１０】
図２４には、画像管理用ユーザ・インターフェースの画面構成例を示している。参照番号２７０４で示される領域には撮影した写真（画像）が表示される。また、参照番号２７５１で示される領域には、撮影時間が、参照番号２７５４で示される領域には、認識種類が優先順位に従って表示され、その右側には各項目の値が表示出力される。
【０１１１】
参照番号２７６１〜２７６３は、コマンド・ボタン群であり、いずれかのボタンをマウスでクリックするなどの選択操作を印加すると、表示中の写真に対して該当するコマンド処理が適用される。
【０１１２】
参照番号２７６４で示される領域には、サムネイル化された写真が、画像ポイントが高い順にリストアップされている。このサムネイル・リスト２７６４上で選択された写真が、表示領域２７０４に表示出力される。ジョグダイヤルやカーソル・キー、マウス・ポインタなどを使って、サムネイル・リストから所望の写真を選択することができる。
【０１１３】
画像ポイントを算出する計算式は、例えば以下のようなものである。すなわち、画像内で認識された各被写体のランキング・ポイント値と認識種類に対する優先順位を乗算したものの総和として表現される。
【０１１４】
【数２】

【０１１５】
図２５には、上下ボタンの操作により認識対象インデックスを変更する様子を示している。
【０１１６】
参照番号５１０１は、ユーザ指定の認識対象インデックスを書き込むフィールドである。また、参照番号５１０５は該当する認識単位のリスト順位を１つずつ上げるボタン、参照番号５１０６は該当する認識単位のリスト順位を１つずつ下げるボタンを、それぞれ示している。
【０１１７】
図示の例では、現在の認識単位のインデックス・リスト５１００は、なっち５１０２、ひかり５１０３からなる。これに対し、参照番号５１０７で示すように、認識対象「ひかり」を削除し、認識対象「なおみ」を追加し、認識対象「なおみ」の順位を下げる操作を行なうと、リスト順位が変更されて、なっち５１１２、なおみ５２１８というリスト順位になる。
【０１１８】
図２６には、認識対象の変更によって変化したデータの様子を示している。
【０１１９】
図２３に示した認識単位インデックスを記述するデータ・フォーマット例では、認識単位を認識種類毎に記述するタグ・フィールドが設けられ、認識種類「人物（ｐｅｒｓｏｎ）」のタグ・フィールド５１０には、撮影画像に含まれる認識対象としての「なっち」、「ひかり」がそれぞれのポイント値０．７２、０．３２ともにタグ情報５１１、５１２として記載されている。
【０１２０】
これに対し、図２５に示したような認識単位インデックスの変更を行なった結果、認識種類「人物」のリスト５１１０内が、なっち５１１２、おなみ５２１８に変わっている。
【０１２１】
上述した実施形態では、撮影側の機器３１は、図４及び図５に示したように通信機能を備えていることを前提としている。しかしながら、本発明を実現するためには、通信機能自体は必須ではない。
【０１２２】
以下では、一般的なデジタル・カメラのように通信機能を持たない撮影機器により写真撮影された場合に被写体認識を行なうという変形例について説明する。
【０１２３】
この場合の撮影機器は、例えば、通信機能を持たない一般的なデジタル・カメラである。また、このユーザに対して移動ログ記録装置（後述）がデジタル・カメラとともに貸し出され、撮影状態の保存のために利用してもらう。そして、貸し出した移動ログ記録装置を回収し、撮影ログと移動ログを取り出して、ログ解析を行なうことにより、撮影空間の算出、被写体認識という処理を行なう。最後に、この結果をユーザに配布する。
【０１２４】
図２８に示すように、デジタル・カメラは、機器前面２８１１に撮影光学系を構成するレンズ２８１１−１を持つ。またその背面２８１２には、撮影画像や入力画面を表示する表示部２８１２−２や、撮影条件の設定、写真のビュー、削除などの機器操作を行なう入力部２８１２−４が配設されている。また、機器上面には、位置測定用のＧＰＳ信号を受信するためのアンテナ１２１と、画像の捕捉を指示するシャッター・ボタン２８１２−４１が配設されている。
【０１２５】
図２９には、この撮影機器の内部構成を示している。撮影機器の動作は、ＣＰＵ４１５がオペレーティング・システムの制御下で、カメラ機能を実現するための各プログラムを実行することによって、コントロールされる。ＣＰＵ４１５は、バス４１７を介して各部に相互接続されている。
【０１２６】
ＲＡＭ４１３は、ＣＰＵ４１５の実行プログラム・コードをロードしたり、カメラ機能の起動時における作業データを一時的に保存したりするために使用される。また、ＲＯＭ４１４は、ＣＰＵ４１５の実行プログラム・コードや製造情報など工場出荷時に書き込まれる情報を恒久的に保存している。
【０１２７】
入力部４０８は、ユーザ操作可能なボタンなどからなり、データ入力のために使用される。また、操作ボタンの１つはシャッター４０９に割り当てられている。
【０１２８】
位置測定部４０３は、アンテナ１２１によって受信されるＧＰＳ信号に基づいて当該機器の現在位置を測定する。また、方向取得部４０４は、デジタル磁気コンパスなどからなり、当該機器の姿勢、若しくはカメラ・レンズの方向を取得する。位置測定にはＧＰＳ信号の信号強度とＧＰＳ衛星の空間的な広がりに基づく位置誤差が含まれるが、位置測定部４０３は位置誤差を推定し、これを出力するようになっている。また、方向測定部４０４は、固定値である方向誤差を出力する。
【０１２９】
撮像部４０５は、カメラ・レンズ２８１１−１とその結像面において画像を捕捉する撮像素子と、画像信号を処理する信号処理モジュールなどで構成される。本実施形態では、撮像部４０５は、カメラ位置、レンズ方向、焦点距離、画角、絞り値などの撮影状態を出力する。
【０１３０】
表示部４０６は、ＣＰＵ４１５による処理結果を画面出力する。例えば、カメラ・レンズ２８１１−１を介して得られるファインダ画面や撮影後の画像が画面表示される。
【０１３１】
出力部４０７は、スピーカによる音声出力や振動、その他ユーザにフィードバックを与える装置からなる。
【０１３２】
時計４１６は、実時間を計時するとともに、システムに対しタイマ信号を供給する。本実施形態では、時計４１６は、撮像部４０５による撮像時刻や、位置測定部４０３による位置測定時刻を出力するようになっている。
【０１３３】
写真保存部４３１は、撮像部４０５による撮影画像を保存する。また、撮影ログ保存部４３２は、各撮影画像についての撮影時刻、撮影状態、撮影時の位置測定や方向取得に包含される誤差情報などからなる撮影ログを保存する。
【０１３４】
ＩＤ保持部４０２は、機器同定のための機器識別情報を格納している。
【０１３５】
図２９に示した機器を用いて写真を撮影する場合、入力部４０８にあるシャッター４０９からの入力に連動して撮影部４０５が動作して写真を撮影し、この撮影画像を画像保存部４３１に保存する。また、写真の撮影並びに画像保存に伴って、時計４１６により撮影時間と、位置測定部４０３より得られるカメラ位置とその誤差範囲、方向取得部４０４より得られるレンズ方向とその誤差範囲を取得し、撮影ログ保存部４３２に保存する。
【０１３６】
図３０には、各ユーザの位置情報ログを記録する移動ログ記録装置の内部構成を示している。
【０１３７】
この移動ログ記録装置の動作は、ＣＰＵ３０１５がオペレーティング・システムの制御下で、携帯電話機能並びにカメラ機能を実現するための各プログラムを実行することによって、コントロールされる。ＣＰＵ３０１５は、バス３０１７を介して各部に相互接続されている。
【０１３８】
ＲＡＭ３０１３は、ＣＰＵ３０１５の実行プログラム・コードをロードしたり、作業データを一時的に保存したりするために使用される。また、ＲＯＭ３０１４は、ＣＰＵ３０１５の実行プログラム・コードや製造情報など工場出荷時に書き込まれる情報を恒久的に保存している。
【０１３９】
時計３０１６は、実時間を計時するとともに、システムに対しタイマ信号を供給する。本実施形態では、時計４１６は、撮像部４０５による撮像時刻や、位置測定部４０３による位置測定時刻を出力するようになっている。
【０１４０】
位置測定部３００３は、アンテナ（図示しない）によって受信されるＧＰＳ信号に基づいて当該機器の現在位置を測定する。位置測定にはＧＰＳ信号の信号強度とＧＰＳ衛星の空間的な広がりに基づく位置誤差が含まれるが、本実施形態では、位置測定部３００３は位置誤差を推定し、これを出力する。位置測定部３００３の一定時間毎の測定結果は、当該機器を携帯するユーザの移動ログとして移動ログ蓄積部３０３４に保存される。
【０１４１】
ＩＤ保持部３００２は、機器同定のための機器識別情報を格納している。
【０１４２】
図２７には、撮影側の機器３１が通信機能を持たないという変形例における被写体認識の処理手順をフローチャートの形式で示している。
【０１４３】
まず、図２８並びに図２９に示したデジタル・カメラなどの撮影機器、並びに図３０に示した移動ログ記録装置をユーザに貸し出し（ステップＳ１）、撮影に利用してもらう（ステップＳ２）。
【０１４４】
その後、貸し出した機器をユーザから回収し、撮影ログや移動ログを取り出す（ステップＳ３）。そして、ログ解析を行なうことにより、撮影空間の算出、被写体認識という処理を行なう（ステップＳ４）。最後に、この結果をユーザに配布する（ステップＳ５）。
【０１４５】
図３１には、被写体とカメラとの位置関係を移動ログから取得する様子を示している。
【０１４６】
参照番号３１０１は撮影側の機器の移動ログを示している。また、参照番号３１１１は、この移動ログ３１０１上で写真撮影が行なわれた地点であり、そのときの時刻３１１３は１２時３５分である。
【０１４７】
一方、被写体位置は被写体が携帯していた移動ログ記録装置の移動ログ３０３４から取り出される。参照番号３１２１は、撮影位置３１１１にて撮影側装置で写真撮影が行なわれたときの被写体位置を示している。なお、被写体の移動ログは、移動ログ記録装置において一定時間毎にその位置を記録していることから、撮影時間３１１３に対応するように、サンプリング値から逆算して求める。
【０１４８】
撮影位置３１１１と撮影側の機器から取り出された撮影ログに基づいて、写真撮影時の撮影空間３１１２が求められる。また、移動ログ記録装置から取り出された移動ログに基づいて、写真撮影時の被写体位置３１２１が求められる。そして、撮影空間３１１２と被写体位置３１２１とを照合することによって、撮影された写真に被写体が入っているかどうかを被写体認識することができる。そして、図２１を参照しながら説明した手順に従って、被写体についてのランキング・ポイント値を計算することができる。
【０１４９】
図３２には、本実施形態における被写体認識の処理手順をフローチャートの形式で示している。
【０１５０】
まず、撮影データをキューに入れる（ステップＳ１１）。そして、このキューから１つずつ撮影データを取り出す（ステップＳ１２）。このとき、未処理データがなくなれば（ステップＳ１３）、本処理ルーチン全体を終了する。
【０１５１】
次いで、登録メンバー表から１人分の移動ログを取り出す（ステップＳ１４）。ここで、未処理メンバーがいなくなった時点で（ステップＳ１５）、ステップＳ１２に戻り、次のキューを取り出す。
【０１５２】
そして、取り出した移動ログから、撮影時間における位置を取得し、撮影空間に入っているかどうかをチェックする（ステップＳ１７）。そして、撮影空間に入っている移動ログが発見されたならば、該当する被写体に対するランキング・ポイント値を計算し（ステップＳ１８）、そのメンバーＩＤとランキング・ポイント値を保存する（ステップＳ１９）。その後、ステップＳ１４に戻り、次の登録メンバーについて、被写体認識並びにランキング・ポイント値の計算処理を繰り返し行なう。
【０１５３】
また、デジタル・カメラなどの最近の撮影機器の中には、機器本体に対してレンズ方向が回転可能に取り付けられていることがある。この場合、撮影機器のユーザは、機器本体を握ったままレンズを自分に向けて自分自身を撮影することができる。この場合、図１６に示したような被写体リストの中からではなく、自分自身を認識対象としなければならない。
【０１５４】
図３３には、レンズ部が回転する撮影装置の外観構成を示している。機器の背面２８１２には、撮影画像や入力画面を表示する表示部２８１２−２や、撮影条件の設定、写真のビュー、削除などの機器操作を行なう入力部２８１２−４が配設されている。また、機器上面には、位置測定用のＧＰＳ信号を受信するためのアンテナ１２１と、画像の捕捉を支持するシャッター・ボタン２８１２−４１が配設されている。また、レンズ２８１１−１を搭載するレンズ部２８１３は、機器本体に対して図中矢印方向に回転可能に軸支されているので、ユーザは機器本体を握ったまま、前方、後方（自分の方向）を含む任意の方向を撮影することができる。
【０１５５】
図３４には、図３３に示したレンズ部が回転する撮影装置の内部構成を示している。
【０１５６】
撮影機器の動作は、ＣＰＵ４１５がオペレーティング・システムの制御下で、携帯電話機能並びにカメラ機能を実現するための各プログラムを実行することによって、コントロールされる。ＣＰＵ４１５は、バス４１７を介して各部に相互接続されている。
【０１５７】
ＲＡＭ４１３は、ＣＰＵ４１５の実行プログラム・コードをロードしたり、携帯電話機能やカメラ機能の起動時における作業データを一時的に保存したりするために使用される。また、ＲＯＭ４１４は、ＣＰＵ４１５の実行プログラム・コードや製造情報など工場出荷時に書き込まれる情報を恒久的に保存している。
【０１５８】
入力部４０８は、ユーザ操作可能なボタンなどからなり、電話番号入力その他のデータ入力のために使用される。また、操作ボタンの１つはカメラ機能起動時におけるシャッター４０９に割り当てられている。
【０１５９】
通信部４０１は、携帯電話網上の基地局との通信処理を行ない、さらにサーバと通信を行なう。
【０１６０】
位置測定部４０３は、アンテナ１２１によって受信されるＧＰＳ信号に基づいて当該機器の現在位置を測定する。また、方向取得部４０４は、デジタル磁気コンパスなどからなり、当該機器の姿勢、若しくはカメラ・レンズの方向を取得する。
【０１６１】
撮像部４０５は、カメラ・レンズとその結像面において画像を捕捉する撮像素子と、画像信号を処理する信号処理モジュールなどで構成される。本実施形態では、撮像部４０５は、カメラ位置、レンズ方向、焦点距離、画角、絞り値などの撮影状態を出力する。上述したように、カメラ・レンズは機器本体に対し回転可能に軸支されている。そして、回転角度測定部４１８は、レンズ部２８１３の回転位置を測定する。
【０１６２】
表示部４０６は、ＣＰＵ４１５による処理結果を画面出力する。例えば、カメラ・レンズを介して得られるファインダ画面や撮影後の画像が画面表示される。
【０１６３】
出力部４０７は、画像信号を外部出力したり、スピーカによる音声出力や振動、その他ユーザにフィードバックを与える装置からなる。
【０１６４】
時計４１６は、実時間を計時するとともに、システムに対しタイマ信号を供給する。本実施形態では、時計４１６は、撮像部４０５による撮像時刻や、位置測定部４０３による位置測定時刻を出力するようになっている。
【０１６５】
写真保存部４３１は、撮像部４０５による撮影画像を保存する。また、撮影ログ保存部４３２は、各撮影画像についての撮影時刻、撮影状態、撮影時の位置測定や方向取得に包含される誤差情報などからなる撮影ログを保存する。
【０１６６】
また、機器は機器同定のための機器識別情報を格納したＩＤ保持部４０２を備えており、位置測定部４０３で取得された位置情報とともに通信部４０１からサーバ３３へ送信される。また、自らの機器位置の公開を許可している他の機器の機器ＩＤをＩＤ名簿４３３に保持している。
【０１６７】
図５に示した携帯電話機上で写真を撮影する場合、入力部４０８にあるシャッター４０９からの入力に連動して撮影部４０５が動作して写真を撮影し、この撮影画像を画像保存部４３１に保存する。また、写真の撮影並びに画像保存に伴って、時計４１６により撮影時間と、位置測定部４０３より得られるカメラ位置とその誤差範囲、方向取得部４０４より得られるレンズ方向とその誤差範囲を取得し、撮影ログ保存部４３２に保存する。
【０１６８】
また、回転角度測定部４１８は、撮影時におけるレンズ部２８１３の回転角度を測定する。ここで、レンズ部２８１３が正面を向いているときには、ＩＤ保持部４０２に格納されているＩＤを被写体に含めないが、レンズ部２８１３が後方を向いているときは、ＩＤ保持部４０２に格納されているＩＤを被写体に含めることによって、自分自身を被写体に含める。
【０１６９】
上述したように、本実施形態では、カメラの位置情報とレンズ方向、並びに焦点距離、画角、絞り値などからなる撮影状態を取得し、カメラ位置を中心とし、ピント面と被写界深度に基づいて定まる半径範囲の領域で、レンズ方向で画角に相当する部分が撮影空間として算出される。そして、撮影空間と各被写体の位置情報を照合し、撮影空間内の人物を被写体として認識する。さらに、撮影空間内の認識対象についてのランキング・ポイント値を算出するに際し、撮影空間の重みすなわち中心角とピント面からの距離に応じた重み付けがなされるとともに、カメラ位置誤差によりカメラ位置の確からしさに応じた重み付けがなされる。
【０１７０】
認識された各被写体に対するランキング・ポイント値の計算方法については、図２１を参照しながら概略的に説明したが、この詳細な処理について以下に説明する。
【０１７１】
図３５には撮影領域を示している。 個々の写真データにおいて、カメラ１の撮影位置、レンズ方向４００１、 画角４００３、ピント距離４０２２、焦点距離情報、絞り値があり、これら撮影状態のパラメータ値を用いて撮影領域４００２を計算する。ここで、参照番号４０１２、４０２２、４０３２で表される各点を通る弧はピント面を表している。また、参照番号４０１３、４０２３、４０３３で表される各点を通る弧は前方被写界深度を表している。また、参照番号４０１１、４０２１、４０３１で表される各点を通る弧は後方被写界深度を表している。
【０１７２】
上述したように、撮影空間には、中心角とピント面からの距離に応じた重み付けがなされている。図３６には、撮影空間における重み傾斜の様子を示している。 参照番号４１０１で示されるグラフはカメラ方向における重み傾斜を示し、また、参照番号４１０２で示されるグラフはピント面の左右方向における重み傾斜を示している。本実施形態では、同図に示すように、点４０２２を中心として、上下方向（点４０２３並びに点４０２１）と左右方向（点４０１２並びに点４０３２）へ向けて、重みを減少させている。
【０１７３】
図３７には、被写体のランキング・ポイント値を計算するための処理手順をフローチャートの形式で示している。
【０１７４】
まず、カメラ位置とレンズ方向を入力する（ステップＳ２１）。次いで、被写体としてのユーザの位置を入力する（ステップＳ２２）。そして、カリングを行なった結果（ステップＳ２３）、カリングされた場合には０を返し（ステップＳ２５）、そうでない場合にはランキング・ポイント値が計算される（ステップＳ２６）。
【０１７５】
図３８には、図３７に示したフローチャート中のステップＳ２３に相当するカリング処理の詳細な手順をフローチャートの形式で示している。
【０１７６】
まず、カメラ位置とレンズ方向を入力する（ステップＳ３１）。次いで、被写体であるユーザの位置を入力し（ステップＳ３２）、対象物を含む最小半径の円を境界円として作成する（ステップＳ３３）。次いで、図３９で述べる距離条件を満たし（ステップＳ３４）、図４０で述べる角度条件１を満たし（ステップＳ３５）、図４１で述べる角度条件２を満たす（ステップＳ３６）場合、ＴＲＵＥを返し（ステップＳ３７）、そうでない場合にはＦＡＬＳＥを返す（ステップＳ３８）。
【０１７７】
図３９には、カリングの距離条件を判定する様子を示している。参照番号４４１１はカメラ位置を、参照番号４４１２はカメラ位置の誤差半径を、参照番号４４２１は被写体境界円の中心位置を、参照番号４４２２は被写体境界円の半径を、参照番号４４３２は撮影領域を、参照番号４４３６はピント距離を、参照番号４４３４は後方被写界深度を、参照番号４４３５は前方被写界深度を、それぞれ示している。また、参照番号４４３７は、カメラ位置から被写体境界円の中心に向かうベクトルを示している。
【０１７８】
下式に従ってカリングの距離条件の判定を行なう。同式によれば、参照番号４４３７に示すベクトルの大きさが、参照番号４４３６で示されるピント距離を中心として、前方被写界深度４４３５と後方被写界深度４４３４の幅に、カメラ位置の誤差半径４４１２と被写体境界円の半径分４４２２の余裕を持って入っていることが条件となる。
【０１７９】
【数３】

【０１８０】
図４０には、カリングの角度条件１を判定する様子を示している。同図において、参照番号４４１１はカメラ位置を、参照番号４４１２にカメラ位置の誤差半径を、参照番号４４２１は被写体境界円の中心位置を、参照番号４４２２は被写体境界円の半径を、参照番号４４３１はレンズ方向を、参照番号４４３３は画角を、参照番号４４３２は撮影空間を、それぞれ示している。また、参照番号４４３８はレンズ方向に向かって右にある画角限界ベクトルを示しており、 これに直交するベクトル４４３９と、カメラ位置４４１１から被写体境界円の中心位置４４２２へ向かうベクトル４４３７の内積を計算する。この内積の値は、ベクトル４４３８から被写体境界円の中心位置４４２２までの符号付距離を表すことになる。
【０１８１】
下式には、カリングの角度条件１の判定を行なう式である。 図４５において求めた符号付距離が、ピント距離４４３６を中心として前方被写界深度４４３４と後方被写界深度４４３５の幅に、カメラ位置の誤差半径４４１２と被写体境界円半径分４４２２の余裕を持って入っていることが条件となる。
【０１８２】
【数４】

【０１８３】
図４１には、カリングの角度条件２を判定する様子を示している。同図において、参照番号４４１１はカメラ位置を、参照番号４４１２はカメラ位置の誤差半径を、参照番号４４２１は被写体境界円の中心位置を、参照番号４４２２は被写体境界円の半径を、参照番号４４３１はレンズ方向を、参照番号４４３３は画角を、参照番号４４３２は撮影空間を、それぞれ示している。参照番号４４３８は、レンズ方向に向かって左にある画角限界ベクトルを示しており、 これに直行するベクトル４４３９と、カメラ位置４４１１から被写体境界円の中心位置４４２２へ向かうベクトル４４３７の内積を計算する。この内積の値は、ベクトル４４３８から被写体境界円の中心位置４４２２までの符号付距離を表すことになる。
【０１８４】
下式には、カリングの角度条件２の判定を行なう式を示している。図４６において求めた符号付距離が、ピント距離４４３６を中心として前方被写界深度４４３４と後方被写界深度４４３５の幅に、カメラ位置の誤差半径４４１２と被写体境界円の半径分４４２２の余裕を持って入っていることが条件となる。
【０１８５】
【数５】

【０１８６】
図４２には、被写体人物に対するランク値の計算を行なうための処理手順をフローチャートの形式で示している。ここでは、被写体人物の誤差円部分の積分を行なう。
【０１８７】
まず、被写体人物の位置Ｏを入力する（ステップＳ４１）。そして、ランク値合計Ｓｕｍを０に初期化するとともに（ステップＳ４２）、半径変数ｒを０に初期化する（ステップＳ４３）。
【０１８８】
次いで、カメラ位置の距離が大きくなるにつれて減少する重みパラメータｗを計算し（ステップＳ４４）、角度変数θを０に初期化する（ステップＳ４５）。
【０１８９】
次いで、被写体人物位置の誤差円内の点座標Ｐを求め（ステップＳ４６）、カメラが位置Ｐにあると仮定したときのランキング・ポイント値を計算して、ｓｕｍに加える（ステップＳ４７）。
【０１９０】
次いで、θに角度刻み幅ｄθを加えて（ステップＳ４８）、θが２πを越えなければ（ステップＳ４９）、ステップＳ４６へ移動する。
【０１９１】
次いで、ｒに距離刻み幅ｄｒを加えて（ステップＳ５０）、ｒが誤差半径Ｃｒを越えなければ（ステップＳ５１）、ステップＳ４４へ移動する。
【０１９２】
そして、誤差半径面積Ｓを計算し（ステップＳ５２）、ｓｕｍをＳで規格化して出力し（ステップＳ５３）、本処理ルーチン全体を終了する。
【０１９３】
また、図４３には、被写体人物に対するランク値の計算を行なうための処理手順をフローチャートの形式で示している。ここでは、カメラ位置の誤差円部分の積分を行なう。
【０１９４】
まず、被写体人物の位置Ｏを入力する（ステップＳ６１）。そして、ランク値合計Ｓｕｍを０に初期化するとともに（ステップＳ６２）、半径変数ｒを０に初期化する（ステップＳ４３）。
【０１９５】
次いで、カメラ位置の距離が大きくなるにつれて減少する重みパラメータｗを計算し（ステップＳ６４）、角度変数θを０に初期化する（ステップＳ６５）。
【０１９６】
次いで、カメラ位置の誤差円内の点座標Ｐを求め（ステップＳ６６）、カメラが位置Ｐにあると仮定したときのランキング・ポイント値を計算して、ｓｕｍに加える（ステップＳ６７）。
【０１９７】
次いで、θに角度刻み幅ｄθを加えて（ステップＳ６８）、θが２πを越えなければ（ステップＳ６９）、ステップＳ６６へ移動する。
【０１９８】
次いで、ｒに距離刻み幅ｄｒを加えて（ステップＳ７０）、ｒが誤差半径Ｃｒを越えなければ（ステップＳ７１）、ステップＳ６４へ移動する。
【０１９９】
そして、誤差半径面積Ｓを計算し（ステップＳ７２）、ｓｕｍをＳで規格化して出力し（ステップＳ７３）、本処理ルーチン全体を終了する。
【０２００】
図４４には、距離条件を判定する様子を示している。同図において、参照番号４４３６はピント距離を、参照番号４４３４は後方被写界深度を、参照番号４４３５は前方被写界深度を、参照番号４４３２は撮影空間を、それぞれ示している。また、参照番号４４３７は、カメラ位置４４１１から被写体位置４４２１に向かうベクトルを示している。
【０２０１】
下式には、距離条件の判定を行なう式を示している。ベクトル４４３７の長さが、ピント距離４４３６を中心として、前方被写界深度４４３５と後方被写界深度４４３４の範囲内に含まれていることを条件とする。
【０２０２】
【数６】

【０２０３】
図４５には、角度条件を判定する様子を示している。 同図において、参照番号４４３１は北からのレンズ方向までの角度を、参照番号４４３２は画角を、参照番号４４３８はレンズ方向ベクトルを、それぞれ示している。また、参照番号４４３７は、カメラ位置４４１１から被写体位置４４２１に向かうベクトルを示している。また、参照番号４４３９は、このベクトル４４３７とレンズ方向ベクトル４４３８のなす角度を示している。
【０２０４】
下式には、角度条件の判定を行なう式を示している。図５１で求めた角度４４３９が画角４４３３未満となることを条件とする。
【０２０５】
【数７】

【０２０６】
図４６には、前後方被写界を分ける様子を示している。同図において、参照番号４４３６はピント距離を、参照番号４４３４は後方被写界深度を、参照番号４４３５は前方被写界深度を、参照番号４４３２−１は前方被写界深度内撮影空間を、 参照番号４４３２−２は後方被写界深度内撮影空間を、それぞれ示している。 また、参照番号４４３７は、カメラ位置４４１１から被写体位置４４２１に向かうベクトルを示している。
【０２０７】
下式により前後方被写界におけるランク値を計算する。 図５２に示すように、被写体が前方被写界深度内撮影空間内４４３２−１にあるときと、後方被写界深度内撮影空間内４４３２−２にあるときでその計算式が異なっている。
【０２０８】
【数８】

【０２０９】
［追補］
以上、特定の実施形態を参照しながら、本発明について詳解してきた。しかしながら、本発明の要旨を逸脱しない範囲で当業者が該実施形態の修正や代用を成し得ることは自明である。すなわち、例示という形態で本発明を開示してきたのであり、本明細書の記載内容を限定的に解釈するべきではない。本発明の要旨を判断するためには、冒頭に記載した特許請求の範囲の欄を参酌すべきである。
【０２１０】
【発明の効果】
以上詳記したように、本発明によれば、１以上の移動体が被写体として含まれる写真画像を好適に管理することができる、優れた画像管理システム及び画像管理方法、並びにコンピュータ・プログラムを提供することができる。
【０２１１】
また、本発明によれば、撮影した写真に写っている人物などの被写体を認識し、写真と被写体とを結合させることによって写真の管理を容易にすることができる、優れた画像管理システム及び画像管理方法、並びにコンピュータ・プログラムを提供することができる。
【０２１２】
また、本発明によれば、複数存在する撮影対象間の優先順位付けを行ない実用的な被写体認識を行なうことができる、優れた画像管理システム及び画像管理方法、並びにコンピュータ・プログラムを提供することができる。
【０２１３】
現在位置を測定するＧＰＳ機能付き携帯機器が普及していることから、個人の位置情報を利用した被写体位置に基づく被写体認識を行なうことができる。また、位置情報の精度がよくない状況に対応するために、位置計測誤差を対象が存在する確からしさとして捉える。本発明によれば、複数の撮影されているかもしれない対象に対してランキングによる重み付けを行ない、リスト順位としてユーザに提示することができる。ユーザは、誰が写っているのかを記述する手間を大幅に削減することができる。また、各自に位置計測装置付きのレシーバを撮影対象に携帯させることによって、地図データ上に記載することが不可能であった人物の認識が可能になる。本発明によれば、計測精度が十分に高くない状況において、計測誤差を用いて、被写体と思われる対象集合の優先度を計算し、ユーザに提示することができる。
【図面の簡単な説明】
【図１】カメラ位置とレンズ方向と地図情報を用いて被写体を認識する様子を示した図である。
【図２】本発明の実施形態に係る画像管理システムのシステム構成を模式的に示した図である。
【図３】被写体の位置情報と撮影状態に基づいて被写体の認識処理が行なわれる仕組みを説明するための図である。
【図４】各自が携帯する機器の外観構成を示した図である。
【図５】図４に示した機器の内部構成を示した図である。
【図６】各自が携帯する機器（但し、カメラ機能なし）の外観構成を示した図である。
【図７】図６に示した機器の内部構成を示した図である。
【図８】図４や図６に示した各機器との通信を行なうサーバの構成を模式的に示した図である。
【図９】被写体への位置情報の利用許可申請を行なう処理手続を示した動作シーケンス図である。
【図１０】被写体への位置情報の利用許可申請を行なった際に、申請が拒否される場合の処理手順を示したシーケンス図である。
【図１１】撮影側機器３１上で撮影された画像に含まれる被写体をセンター・サーバ３３で認識して各被写体にランキング・ポイントを付与して機器３１に提供し、撮影側機器３１上でランキング・ポイントに基づいたユーザの編集操作を行なうための処理手順を示した動作シーケンス図である。
【図１２】撮影側機器３１上において写真撮影時に取得する情報を示した図である。
【図１３】写真撮影時に取得される撮影状態を記録するためのデータ・フォーマットの構成例を示した図である。
【図１４】カメラのレンズ方向の表現方法についての一例を示した図である。
【図１５】セル分割された地図上におけるカメラ位置とレンズ方向、被写体の関係を示した図である。
【図１６】機器が認証を受けている被写体リストの構成例を示した図である。
【図１７】セル内に存在する認識対象を登録している様子を示した図である。
【図１８】撮影空間を含むセルを選択する様子を示した図である。
【図１９】各機器についての被写体リストから認識対象を取得する様子を示した図である。
【図２０】セル内認識対象早見表を利用して、選択されたセルから認識対象を取得する様子を示した図である。
【図２１】撮影空間内の認識単位に対するランキング・ポイントを計算する様子を示した図である。
【図２２】撮影画像の中から認識対象インデックスが取得された様子を示した図である。
【図２３】認識対象インデックスを記述するデータ・フォーマットの構成例を示した図である。
【図２４】ランキング・ポイントに基づいた画像管理用ユーザ・インターフェースの画面構成例を示した図である。
【図２５】上下ボタンの操作により認識対象インデックスを変更する様子を示した図である。
【図２６】認識対象の変更によって変化したデータの様子を示した図である。
【図２７】撮影側の機器３１が通信機能を持たない場合の被写体認識の処理手順を示したフローチャートである。
【図２８】通信機能を持たない撮影機器の外観構成例を示した図である。
【図２９】通信機能を持たない撮影機器の内部構成を示した図である。
【図３０】移動ログ記録装置の内部構成を示した図である。
【図３１】被写体とカメラとの位置関係を移動ログから取得する処理を説明するための図である。
【図３２】機器が通信機能を持たない場合における被写体認識の処理手順を示したフローチャートである。
【図３３】レンズ部が回転する撮影装置の外観構成を示した図である。
【図３４】レンズ部が回転する撮影装置の内部構成を示した図である。
【図３５】被写体に対するランキング・ポイント値の計算方法を説明するための図である。
【図３６】被写体に対するランキング・ポイント値の計算方法を説明するための図である。
【図３７】被写体に対するランキング・ポイント値の計算方法を説明するための図である。
【図３８】被写体に対するランキング・ポイント値の計算方法を説明するための図である。
【図３９】被写体に対するランキング・ポイント値の計算方法を説明するための図である。
【図４０】被写体に対するランキング・ポイント値の計算方法を説明するための図である。
【図４１】被写体に対するランキング・ポイント値の計算方法を説明するための図である。
【図４２】被写体に対するランキング・ポイント値の計算方法を説明するための図である。
【図４３】被写体に対するランキング・ポイント値の計算方法を説明するための図である。
【図４４】被写体に対するランキング・ポイント値の計算方法を説明するための図である。
【図４５】被写体に対するランキング・ポイント値の計算方法を説明するための図である。
【図４６】被写体に対するランキング・ポイント値の計算方法を説明するための図である。
【符号の説明】
１０１…撮像装置
１０２…撮影状態取得部
１０３…被写体認識部
１０４…ランキング・ポイント付与部
１０５…画像保存部
１０６…画像検索／編集部
４０１…通信部
４０２…ＩＤ保持部
４０３…位置測定部
４０４…方向取得部
４０５…撮像部
４０６…表示部
４０７…出力部
４０８…入力部
４０９…シャッター
４１３…ＲＡＭ
４１４…ＲＯＭ
４１５…ＣＰＵ
４１６…時計
４１７…バス
４３１…写真保存部
４３２…撮影ログ保存部
４３３…ＩＤ名簿
５０１…通信部
５１０…撮影対象範囲計算部
５１１…被写体リスト取得部
５１２…ランキング・ポイント計算部
５１３…ＲＡＭ
５１４…ＲＯＭ
５１５…ＣＰＵ
５２１…端末位置情報蓄積部
５２２…ＩＤ公開情報蓄積部
５２４…地図情報蓄積部
５２５…催し物カレンダー蓄積部[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image management system and an image management method for managing a large number of photographic images, and a computer program, and more particularly to an image management system and an image management method for managing a photographic image including one or more moving objects as subjects. , As well as computer programs.
[0002]
More specifically, the present invention relates to an image management system, an image management method, and a computer program for recognizing a subject in a photographed photograph and for facilitating the management of the photograph by combining the photograph and the subject. More particularly, the present invention relates to an image management system and an image management method for prioritizing a plurality of photographing targets and performing practical object recognition, and a computer program.
[0003]
[Prior art]
In recent years, devices for outputting and reproducing captured images as digital contents, such as digital cameras, have become widespread. This type of photograph is stored on a magnetic tape, a magnetic disk, a semiconductor memory, or the like. The simple operation of the device and the easy output of the photos, combined with the simplicity of photographing, increase the number of photos. In such a case, from the viewpoint of effective use of the content, a suitable management method of the photograph is important.
[0004]
For example, a technique has been adopted in which predetermined meta information is added to an image, and the image is managed and searched based on the meta information. In this case, events and other situations at the time of photographing the photograph, episodes related to the photographing, information and impressions on the subject, and these keywords are managed together with the image as meta information. However, depending on the user's manual input of the meta information, the work load is excessive and troublesome.
[0005]
In addition, there have been proposed some methods of automatically adding a shooting time, a shooting location detected using a GPS (Global Positioning System) or the like as meta information to an image body, and the like.
[0006]
Here, the present inventors think that it is possible to facilitate a photo search by combining what is in the photograph (object) with the photographed photograph.
[0007]
For example, a method using a camera position and map information, a method of attaching a marker (visibility identification information) having a unique recognition ID to each subject, and specifying the subject based on a marker in a photographic image are considered. Can be A method is also conceivable in which the automatic recognition is not performed, and the photographer or other user later describes who is in the photograph.
[0008]
However, in the method using the camera position and the map information, it is not possible to perform subject recognition on a moving object such as a person or a car.
[0009]
In addition, since the method of performing image recognition of a photograph cannot follow changes in the direction of a face or a facial expression, practical recognition of a person has not been achieved, and the situation is far from commercialization.
[0010]
Also, in the method of attaching a marker having a unique recognition ID to a subject, it is necessary to take the marker together at the time of shooting, so it is meaningful if the marker becomes a shadow of another person or is hidden by the subject's own pose. Absent.
[0011]
Further, a method in which the photographer later attaches the recognition information to the subject is not realistic because the work load is excessive.
[0012]
[Problems to be solved by the invention]
An object of the present invention is to provide an excellent image management system, an excellent image management method, and a computer program that can appropriately manage a photographic image including one or more moving objects as subjects.
[0013]
It is a further object of the present invention to provide an excellent image management system capable of recognizing a moving object such as a person in a photographed photograph and facilitating the management of the photograph by combining the photograph and the object. It is to provide a system, an image management method, and a computer program.
[0014]
It is a further object of the present invention to provide an excellent image management system, an excellent image management method, and a computer program, which can prioritize a plurality of photographing targets and perform practical object recognition. .
[0015]
Means and Action for Solving the Problems
The present invention has been made in view of the above problems, and a first aspect thereof is an image management system that manages a photographic image including one or more moving objects as a subject by combining the photographic image with the subject,
Shooting state obtaining means for obtaining a shooting state at the time of image shooting,
Subject position obtaining means for obtaining the position of each moving body at the time of image capturing,
A photographing space estimating means for calculating a photographing space to be photographed in a photographed image based on the photographing state;
A subject recognition unit that compares a shooting space calculated by the shooting space estimation unit with a position of each moving body obtained by the subject position obtaining unit, and recognizes a moving body in the shooting space as a subject;
Subject evaluation value calculation means for calculating an evaluation value according to a situation in the image of the recognized subject,
An image management system comprising:
[0016]
However, the term “system” as used herein refers to a logical collection of a plurality of devices (or functional modules that realize specific functions), and each device or functional module is in a single housing. It does not matter in particular.
[0017]
Here, the image management system according to the present invention may manage the subjects included in each image according to the priority order based on the evaluation value. In such a case, an image including a desired subject can be searched according to the priority order.
[0018]
Further, the photographing state acquiring unit acquires a camera position, a lens direction, a focal length, an angle of view, and an aperture value at the time of photographing as a photographing state, and the photographing space estimating unit acquires the photographing state based on the indicated values of these photographing states. A shooting space including a focus plane and a depth of field may be calculated. The subject evaluation value calculation means may calculate the evaluation value based on the likelihood that the subject exists in the shooting space.
[0019]
According to the present invention, in the object recognition, the distance from the focus plane of the photographing space, the distance from the central axis, the measured value and error radius of the camera position, and the photographed space weighted according to the direction measured value and the error width. By using the likelihood that a subject to be recognized exists, it is possible to assign a priority to each of the plurality of recognition indexes.
[0020]
As a result, the list order of the recognition index set can be determined and used for photo search and other management / editing of photos. In particular, according to the present invention, even in a photographic image in which a moving object such as a person is included as a subject, the subject can be recognized by collating positional information acquired from the subject with the shooting space. The management of the photograph can be performed by combining the above.
[0021]
The subject evaluation value calculating means calculates an evaluation value by giving a weight based on a shooting position error, a visual line direction error, and a position measurement error of the subject to the likelihood that the subject exists in the shooting space.
[0022]
For example, in a situation where the measurement accuracy is not sufficiently high, a large number of recognition candidates are obtained, and presented to the user in a form in which they are ranked according to the certainty of the information, thereby changing the list rank or deleting items. At the time of editing, etc., the user is less burdened than the effort of adding items by manual input.
[0023]
Further, the image management system according to the present invention may further include a position information use permission unit that obtains in advance use permission of position information from a moving body that can be a subject. In such a case, the subject position obtaining means obtains the position information of the moving body for which use permission of the position information is obtained, or the subject recognizing means performs the subject recognition processing only for the moving body for which use permission of the position information is obtained. Do it.
[0024]
The use of position information is deeply related to the privacy of a device user as a subject. Therefore, in the present invention, in order to estimate a subject, it is assumed that each device user has permitted use of its own position information in advance. In addition, devices that refuse to use the position information are excluded from the subject estimation processing.
[0025]
Further, a second aspect of the present invention is described in a computer-readable format so that a process for managing a photographic image including one or more moving objects as a subject in combination with the subject is executed on a computer system. Computer program,
A photographing state acquiring step for acquiring a photographing state at the time of photographing an image,
Subject position obtaining step of obtaining the position of each moving body at the time of image capturing,
A photographing space estimation step of calculating a photographing space to be photographed in a photographed image based on the photographing state;
A subject recognition step of collating a shooting space obtained in the shooting space estimation step with a position of each moving body obtained in the subject position obtaining step, and recognizing a moving body in the shooting space as a subject;
A subject evaluation value calculating step of calculating an evaluation value according to a situation in the image of the recognized subject;
A computer program characterized by comprising:
[0026]
The computer program according to the second aspect of the present invention defines a computer program described in a computer-readable format so as to realize a predetermined process on a computer system. In other words, by installing the computer program according to the second aspect of the present invention in a computer system, a cooperative action is exerted on the computer system, and the image management according to the first aspect of the present invention is performed. The same operation and effect as those of the system can be obtained.
[0027]
Further objects, features, and advantages of the present invention will become apparent from more detailed descriptions based on embodiments of the present invention described below and the accompanying drawings.
[0028]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0029]
FIG. 1 shows how a subject is recognized using a camera position, a lens direction, and map information. In the figure, reference numeral 1 denotes a camera used for photographing. In the illustrated example, two persons 25 and 26 are photographed as moving objects. Reference numeral 3 indicates a state in which the camera position and the subjects 25 and 26 are mapped on a map. Reference numeral 4 indicates a photograph of a scene including the subjects 25 and 26 at the illustrated camera position and lens direction.
[0030]
FIG. 2 schematically shows a system configuration of the image management system according to the embodiment of the present invention. The illustrated image management system manages a photograph by combining a photographed photograph with a subject shown in the photograph.
[0031]
First, photographing is performed by the photographing device 101 such as a digital camera. Further, the photographing state acquiring unit 102 simultaneously acquires the photographing state at this time. The shooting state here includes a camera position, a lens direction, a focal length, an angle of view, an aperture value, and the like at the time of shooting.
[0032]
The subject recognizing unit 103 recognizes a subject in a captured image using a shooting state. More specifically, a shooting space including a focus plane and a depth of field is calculated based on each indication value of a shooting state, and the shooting space is compared with position information of each moving object, and a moving object in the shooting space is calculated. Is recognized as a subject.
[0033]
The ranking point giving unit 104 calculates an evaluation value, that is, a ranking point, according to the estimated situation of the subject in the captured image. The ranking point mentioned here is calculated based on, for example, the likelihood that the subject exists in the photographing space, and can be weighted based on the photographing position error and the gaze direction error (described later).
[0034]
The image storage unit 105 stores the captured image and the index of the subject included in the captured image in association with each other. Then, the image search / edit unit 106 determines the order of the list of the recognition index set, provides a predetermined user interface (described later), and supports image search and editing by user operation.
[0035]
FIG. 3 illustrates a mechanism in which the subject management processing is performed based on the subject position information and the shooting state in the image management system shown in FIG.
[0036]
The photographing device 31 transfers photographing information acquired in the device 31 at the time of photographing to the center 33. The person as the subject 32 has a portable terminal with a position measurement function such as GPS, and transfers his / her own position information to the center 33. Thereafter, in the center 33, a subject recognition process is performed. More specifically, a shooting space consisting of a focus plane and a depth of field is calculated based on each indication value of a shooting state, and the shooting space is compared with position information of each moving object on predetermined map information, and shooting is performed. Recognize a person in space as a subject.
[0037]
FIG. 4 shows an external configuration of a device carried by each user in the image management system described above. The illustrated device is, for example, a mobile phone with a camera function. The device functions as the image capturing device 101, and can acquire position information and transfer the acquired position information to the center 33 in the case of a subject.
[0038]
The illustrated device includes a main body including a user operation unit such as a button, and a lid rotatably supported at a substantially rear edge of the main body. An antenna 112 for mobile phone communication and an antenna 121 for receiving a GPS signal are arranged at the tip of the lid, and a display device made up of a liquid crystal panel is incorporated on the front side 12 thereof. A camera lens 111 appears on the back surface 11 of the lid, and an image capturing process is activated by pressing a button 1241 assigned to a shutter function on the upper surface of the main body, and an object through the lens 111 is photographed. You.
[0039]
FIG. 5 shows the internal configuration of the device shown in FIG.
[0040]
A CPU (Central Processing Unit) 415 executes each program for realizing the mobile phone function and the camera function under the control of the operating system, so that the operation of the photographing apparatus 101 is controlled overall. The CPU 415 is interconnected to each unit via a bus 417.
[0041]
A RAM (Random Access Memory) 413 is configured by a readable / writable semiconductor memory, and loads an execution program code of the CPU 415 and temporarily stores work data at the time of activation of a mobile phone function or a camera function. Used for A ROM (Read Only Memory) 414 is configured by a read-only semiconductor memory, and permanently stores information written at the time of factory shipment, such as an execution program code of the CPU 415 and manufacturing information.
[0042]
The input unit 408 includes buttons that can be operated by the user, and is used for inputting a telephone number and other data. One of the operation buttons is assigned to the shutter 409 when the camera function is activated.
[0043]
The communication unit 401 performs a communication process with a base station on the mobile phone network, and further communicates with a server (described later).
[0044]
The position measuring unit 403 measures the current position of the device based on the GPS signal received by the antenna 121. The direction obtaining unit 404 includes a digital magnetic compass or the like, and obtains the orientation of the device or the direction of the camera / lens. The position measurement includes a position error based on the signal strength of the GPS signal and the spatial spread of the GPS satellite. In the present embodiment, the position measurement unit 403 estimates the position error and outputs the position error. I have. The direction measuring unit 404 outputs a direction error that is a fixed value.
[0045]
The imaging unit 405 includes a camera / lens, an imaging element that captures an image on an image plane thereof, a signal processing module that processes an image signal, and the like. In the present embodiment, the imaging unit 405 outputs a shooting state such as a camera position, a lens direction, a focal length, an angle of view, and an aperture value together with information on a shot image.
[0046]
The display unit 406 outputs the processing result of the CPU 415 on the screen. For example, when the mobile phone function is activated, the entered telephone number, the status of other devices during a call, etc. are displayed, and when the camera function is activated, the viewfinder screen obtained through the camera and lens and the captured image are displayed on the screen. Is done.
[0047]
The output unit 407 is a device that externally outputs an image signal, outputs sound from a speaker, vibrates, and provides feedback to a user.
[0048]
The clock 416 measures real time and supplies a timer signal to the system. In the present embodiment, the clock 416 outputs the imaging time by the imaging unit 405 and the position measurement time by the position measurement unit 403.
[0049]
The photo storage unit 431 stores an image captured by the imaging unit 405. The photographing log storage unit 432 stores a photographing log including a photographing time, a photographing state, position measurement at the time of photographing, and error information included in direction acquisition for each photographed image.
[0050]
The device includes an ID holding unit 402 storing device identification information for device identification, and is transmitted from the communication unit 401 to a server (described later) together with the position information acquired by the position measuring unit 403. Further, the device IDs of other devices permitted to disclose their device positions are stored in the ID list 433.
[0051]
When taking a photograph on the mobile phone shown in FIG. 5, the photographing unit 405 operates in synchronization with the input from the shutter 409 of the input unit 408 to take a photograph, and the photographed image is stored in the image storage unit 431. save. Along with taking a picture and storing an image, a clock 416 acquires a photographing time, a camera position obtained by the position measuring unit 403 and its error range, and a lens direction obtained by the direction obtaining unit 404 and its error range, The shooting state is stored in the shooting log storage unit 432 as the shooting state. In addition, even when photographing is not performed, the position of the device is grasped at regular intervals and recorded as a log together with the time measured by the clock 416.
[0052]
Of course, a person who does not take a picture himself may have a portable device having a position measuring function without a camera function. FIG. 6 shows the external configuration of the device in this case, and FIG. 7 shows the internal configuration thereof.
[0053]
As shown in FIG. 6, the illustrated device includes an antenna 112 for mobile phone communication and an antenna 121 for receiving a GPS signal.
[0054]
The CPU 415 executes each program for realizing the mobile phone function and the camera function under the control of the operating system, so that the operation of the photographing apparatus 101 is totally controlled. The CPU 415 is interconnected to each unit via a bus 417.
[0055]
The RAM 413 is used for loading an execution program code of the CPU 415 and temporarily storing work data when the mobile phone function is activated. The ROM 413 permanently stores information written at the time of factory shipment, such as an execution program code of the CPU 415 and manufacturing information.
[0056]
The communication unit 401 performs a communication process with a base station on the mobile phone network, and further communicates with a server (described later).
[0057]
The position measuring unit 403 measures the current position of the device based on the GPS signal received by the antenna 121. The position measurement includes a position error based on the signal strength of the GPS signal and the spatial spread of the GPS satellite. In the present embodiment, the position measurement unit 403 estimates the position error and outputs the position error. In addition, the position measurement results are arranged in time series and recorded in the movement log 434.
[0058]
The clock 416 measures real time and supplies a timer signal to the system. In the present embodiment, the clock 416 outputs the position measurement time by the position measurement unit 403.
[0059]
This device includes an ID holding unit 402 storing device identification information for device identification, and is transmitted from the communication unit 401 to the server together with the position information acquired by the position measurement unit 403. Further, the device IDs of other devices permitted to disclose their device positions are stored in the ID list 433.
[0060]
FIG. 8 schematically illustrates a configuration of a server that performs communication with each device illustrated in FIGS. 4 and 6. The server receives information on the shooting state and the shooting time from the device on the shooting side, and also receives information on the position of the subject and information on the position measurement time from the device on the subject side. A process of collating with the position information of the moving object and estimating a person in the photographing space as a subject is performed.
[0061]
The CPU 515 executes each program for realizing the mobile phone function and the camera function under the control of the operating system, so that the overall operation of the server device is controlled in a comprehensive manner. The CPU 515 is interconnected to each unit via a bus 517.
[0062]
The RAM 513 is configured by a readable and writable semiconductor memory, and is used for loading an execution program code of the CPU 515 and temporarily storing work data. The ROM 513 is configured by a read-only semiconductor memory, and permanently stores information written at the time of factory shipment, such as an execution program code of the CPU 515 and manufacturing information.
[0063]
The communication unit 501 performs communication processing with a mobile phone owned by a user via a mobile phone network or another network.
[0064]
The map information storage unit 524 stores predetermined map information. The map information includes arrangement information on buildings and other objects existing at each location. The entertainment calendar 525 manages, on a time axis, information on events and the like related to buildings and other objects arranged at each location in the map information. However, the map information storage unit 524 and the entertainment calendar 525 are not essential for subject recognition processing.
[0065]
The shooting target range calculation unit 510 obtains shooting conditions such as a camera position, a lens direction, a focal length, an angle of view, and an aperture value from a shooting log attached to a shot image, and focuses on a focus plane based on the indicated values of the shooting conditions. And the depth of field are calculated as the permissible range to be photographed by the camera (described later).
[0066]
The terminal position information storage unit 521 stores terminal position information transmitted from a device carried by each user. The ID disclosure information storage unit 522 stores the device ID of a device that is permitted to disclose its device position.
[0067]
The subject list acquisition unit 511 compares the shooting space calculated from the shooting log sent from the shooting device with the position information of the subject sent from the shooting device, and sets A set of persons is acquired as a recognition target, that is, a subject list.
[0068]
The ranking point calculation unit 512 calculates an evaluation value according to the estimated situation in the image of the subject as a ranking point. The ranking point value referred to here is calculated based on the probability that the subject exists in the shooting space. However, since the captured image includes uncertain components such as an error in the camera position and an error in the lens direction, the likelihood that the subject exists in the imaging space is determined based on the imaging position error and the gaze direction error. Weighting is given to give ranking points based on the accuracy of the information (described later).
[0069]
In the present embodiment, it is estimated whether or not the subject is a subject on a photographed image by using subject position information obtained from people carrying the devices shown in FIGS. . In order to perform the subject recognition, it is assumed that each device user permits use of subject position information that is deeply related to privacy. FIG. 9 illustrates a processing procedure for applying for a use permission of positional information to a subject.
[0070]
First, the photographing-side device 31 makes an application for a list registration to the center server 33 (T911). Next, the center server 33 confirms the name list registration with the device 32 on the subject side (T921).
[0071]
When the permission is returned from the device 32 on the subject side (T931), the center server 33 updates the ID disclosure information and sends a list registration change notification to the device 31 on the photographing side (T914).
[0072]
FIG. 10 shows a processing procedure in the case where an application for permission to use position information to a subject is made and the application is rejected.
[0073]
The device 31 on the photographing side makes an application for registering a list to the center server 33 (T911), and the center server 33 confirms the registering of the directory on the device 32 on the subject side (T921).
[0074]
On the other hand, when the rejection is returned from the device 32 on the subject side (T932), the center server 33 sends a name list registration rejection notification to the device 31 on the photographing side (T913). Since the use of position information is closely related to the privacy of a device user as a subject, devices that have refused to register a list are not subject to subject estimation processing thereafter.
[0075]
FIG. 11 shows that the center server 33 recognizes a subject included in an image photographed on the photographing device 31, assigns a ranking point to each subject, provides the ranking point to the subject, and provides the device 31 with the ranking point. 9 shows a processing procedure for performing a user's editing operation based on a ranking point.
[0076]
First, after photographing is performed by the photographing side device 31 (T1111), the device ID and the photographing log such as the focal length, the angle of view, and the aperture are transmitted to the center server 33 side (T1112).
[0077]
On the side of the center server 33, a list of target persons that can be a subject is acquired from the ID disclosure information 522 (T1121). At this time, devices that have not obtained permission to register the name list in the ID disclosure information 522 in advance are excluded from the subject list from the viewpoint of privacy protection and the like. Then, the center server 33 acquires the photographing state such as the camera position, the lens direction, the focal length, the angle of view, and the aperture value from the photographing log, and based on the instruction values of the photographing state, the focus plane and the depth of field. Is calculated as a range in which the captured image is to be captured (described later) (T1122).
[0078]
When there is a subject-side device registered in the list (that is, position acquisition permitted) in the shooting range, the center server 33 confirms the position of each of the devices 32 (T1123). Then, a position report is received from each device 32 (T1131), objects in the shooting space are extracted, and a subject list is created (T1124).
[0079]
Thereafter, the center server 33 calculates an evaluation value according to the situation in the image for each subject as a ranking point (T1125). The ranking point value referred to here is calculated based on the probability that the subject exists in the shooting space. However, since the captured image includes uncertain components such as an error in the camera position and an error in the lens direction, the likelihood that the subject exists in the imaging space is determined based on the imaging position error and the gaze direction error. Weighting is given to give ranking points based on the accuracy of the information (described later).
[0080]
Then, the center server 33 returns the created subject list and the list order to the photographing device 31 (T1126).
[0081]
The photographing-side device 31 uses the received subject list and list order to appropriately add or correct the subjects included in the photograph and their order (T1113).
[0082]
FIG. 12 shows information acquired on the photographing device 31 at the time of photographing. For example, when a photograph as indicated by reference numeral 704 is photographed, the photographing time 751 output by the clock 416, the photographing location 752 measured by the position measuring unit 403, and the photographing time acquired by the direction acquiring unit 404 are simultaneously obtained with the photographing. The lens direction 753 is acquired and stored in the shooting log storage unit 423 as a shooting state 705 in association with the shot image.
[0083]
FIG. 13 shows a configuration example of a data format for recording a photographing state acquired at the time of photographing as shown in FIG. In the illustrated example, the shooting state is described in an xml (extended markup language) format, and includes a shooting time 851, a shooting location 852, and a shooting direction 853. Further, this data format includes a link 804 to a captured image.
[0084]
FIG. 14 illustrates an example of a method for expressing the imaging direction, that is, the lens direction of the camera acquired from the direction acquisition unit 404. In the illustrated example, the lens direction 531 is described as an angle 532 in a clockwise direction when north is set to 0 degree.
[0085]
In the present embodiment, the map information 524 has two formats. One of them is map information editing data, and the other one describes a state in which a map on which a recognition unit such as a building is placed is divided into cells. The latter is used for actual processing such as assigning a ranking point to each subject in a captured image.
[0086]
FIG. 15 shows a relationship between a camera position, a lens direction, and a subject on a cell-divided map. In the example shown in the figure, the map is divided into six in the vertical direction and eight in the horizontal direction. In practice, some measures such as layering the cell division are performed, but are omitted in this specification for simplification of the description.
[0087]
People 21 to 26 as subjects are scattered on the illustrated map. Each subject carries a device having the configuration shown in FIG. 4 or FIG. 6, and each subject user obtains positional information from the center server 33 when registering a name list, that is, using information in advance is permitted. can do. FIG. 1 shows a state in which the photographing device 1 has photographed two subjects 25 and 26.
[0088]
FIG. 16 shows a configuration example of a subject list for which the device has been authenticated. In the example shown in the figure, the device user 1 has been authenticated as a subject by the device users 22, 25, and 26. The center server 33 responds to the subject list request from the device user 1 by requesting the position information of these subjects. Is obtained, the image is compared with the shooting space, and subject authentication is performed on the shot image. Similarly, the device user 25 has obtained authentication as a subject from the device users 1 and 26, and the device user 26 has obtained authentication as a subject from the device users 1 and 24.
[0089]
FIG. 17 shows a state where a recognition target existing in a cell is registered. For example, reference numeral 50 describes information in a cell located at the fifth cell in the horizontal direction and the 0th cell in the vertical direction on a cell-divided map as shown in FIG. It can be seen that the target device 24 is included in the cell. By adopting the registration unit registration method as shown in FIG. 19, the recognition unit can be quickly viewed.
[0090]
In the present embodiment, a camera position, a lens direction, a focal length, an angle of view, an imaging value such as an aperture value are acquired, and an imaging space including a focus plane and a depth of field based on an instruction value of the imaging state is obtained. It is calculated as an allowable range for the camera to shoot. Then, the photographing space of the camera used for photographing is collated with the position information of the subject who has been authenticated, and a subject in the photographing space is extracted as a recognition target, and a subject list is created. For the sake of computational convenience when searching for a recognition target in the shooting space, an intra-cell recognition target quick reference table as shown in FIG. 17 is used.
[0091]
FIG. 18 shows a state where a cell including an imaging space is selected. As shown in the figure, first, a shooting state including position information of a camera, a lens direction, a focal length, an angle of view, an aperture value, and the like is acquired, and a shooting space 11 is created. Then, a cell mass 41 overlapping this area is selected.
[0092]
Next, a recognition target included in the selected cell is obtained. FIG. 19 illustrates a state where the recognition target is acquired from the subject list for each device illustrated in FIG. 16. FIG. 20 shows a state in which the recognition target is acquired from the selected cell using the in-cell recognition target quick reference table shown in FIG.
[0093]
First, as shown in FIG. 19, it is detected from the subject list that the subjects authenticating the device 1 are the devices 22, 25, and 26, and the position information of these devices is acquired.
[0094]
Next, as shown in FIG. 20, three cells in the fifth direction in the horizontal direction and the second to fourth cells in the vertical direction, and two cells in the sixth direction in the horizontal direction and third to fourth in the vertical direction are included in the imaging space. Is selected as a cell overlapping with. Then, the imaging space is collated with the position information sent from each device, and the device 1, the device 25, and the device 26 are acquired as being included in the selected cell.
[0095]
Next, a ranking point is calculated as an evaluation value for each recognized recognition target. In the present embodiment, the evaluation value is calculated based on the likelihood that a person (or another moving object) as a subject exists in the imaging space. Further, the calculation is performed by giving a weight based on the imaging position error and the line-of-sight direction error to the likelihood that the subject exists in the imaging space. That is, the likelihood of the subject position is used for a region weighted according to the distance from the focus plane of the shooting space, the distance from the central axis, the measured value and error radius of the camera position, the direction measured value and the error width. Then, a ranking point indicating a priority is attached to each of the recognition targets.
[0096]
For example, in a situation where the measurement accuracy is not sufficiently high, a large number of recognition candidates are acquired, and presented to the user in a ranked form, so that the user can change the list rank or delete items when editing. Is less burdensome than the effort of adding items by manual input.
[0097]
FIG. 21 shows how a ranking point is calculated for a recognition unit in the shooting space.
[0098]
As described above, the camera-based photographing apparatus 1 has a camera position error and a lens direction error. The position error is output from the position measurement unit 403 due to the signal strength of the GPS signal and the spatial spread of the GPS satellite at the time of position measurement. The error in the lens direction is output from the direction measurement unit 404 due to device characteristics such as a digital magnetic compass. In the example shown in FIG. 21, the position error corresponds to an error circle indicated by reference numeral 2211. The lens direction error is indicated by reference numeral 2217. These position errors and direction errors are components of the shooting state and can be obtained from the shooting log. Further, the recognition target indicated by reference numeral 2 also has a position error generated at the time of position measurement.
[0099]
Here, the certainty when the camera 1 is at the cell position indicated by reference numeral 2212 in FIG. 21 is set according to the distance 2121 from the actual position measurement result. In the present embodiment, this value is set so as to decrease as going from the center to the periphery. In addition, standardization is performed so that the sum of the certainty of each cell 2212 corresponding to the camera position is 1.
[0100]
FIG. 21 shows a lens direction 2213, an angle of view 2216, a focus plane 2215, and a shooting space 2214 when the camera 1 is at a cell position indicated by reference numeral 2212, respectively.
[0101]
The recognition target in the photographing space 2214 has an error range 2221 centered on the subject position 2202 obtained by the position measurement, and this is divided into cells as indicated by reference numeral 2222 to divide it into cells. Perform calculations. Each cell 2222 is weighted according to the distance 2222-3 from the measured value 2202, and further weighted according to the central angle 2222-1 and the distance 2222-4 from the focus plane.
[0102]
Ranking point r for recognition unit p _p Is shown below.
[0103]
(Equation 1)

[0104]
Where A _ij Is the weight of the imaging space of the i-th row and j-th column cells, C _i Is the weight of the camera position, D _j Is the weight in the lens direction, O _ks Represents the certainty of the subject. These weights A _ij , C _i , D _j , O _ks Are assumed to have their values normalized.
[0105]
FIG. 22 illustrates a state where the recognition target index is obtained from the captured image. With reference to FIG. 12, it has been described that the shooting state is acquired together with the shot image at the time of shooting a photo. Reference numeral 57 indicates a recognition target index. When the recognition target index is acquired, a person 510, a place 520, and an event 530 are added as a recognition type in addition to the shooting log. Reference numeral 56 indicates a ranking point value set for each recognition unit index.
[0106]
FIG. 23 shows a configuration example of a data format describing the recognition target index.
[0107]
The configuration of the data format for describing the shooting state has already been described with reference to FIG. In the example illustrated in FIG. 13, the shooting state is described in the xml format, and includes a link to a shooting image, a shooting time, a shooting location, and a shooting direction.
[0108]
In FIG. 23, the xml data further describes a recognition index included in the captured image and its point value, and also describes an imaging time and an event extracted from the recognition position index and its point value. In the illustrated example, a tag field that describes a recognition target for each recognition type is provided, and a tag field 510 of a recognition type “person” includes “Nachi” as a recognition target included in a captured image, “Hikari” is described as tag information 511 and 512 together with respective point values 0.72 and 0.32. In the tag field 520 of the recognition type “location”, point values of Heian Jingu, Jingu-dori and Kyoto as recognition units included in the captured image are 0.63, 0.28, and 0.19, respectively. Along with tag information 521, 522, and 523. In addition, the tag field 530 of the recognition type “event (event)” includes an event “era festival” extracted based on the recognition unit “Heian Jingu” and the shooting time, and a recognition unit “Kyoto” and the shooting time. The extracted event “autumn leaves” is described as tag information together with the respective point values 0.63 and 0.19.
[0109]
According to the image management system according to the present embodiment, based on the photographing time, photographing state, error information included in position measurement and direction acquisition at the time of photographing, the subject included in the photographed image is recognized, and Assignment of a ranking point to a subject and provision of a ranking point in an acquisition sequence of events related to the subject are performed. Then, on the user side, a list of subjects is presented in a priority order based on the ranking points, so that it is possible to suitably manage the photographs based on the list.
[0110]
FIG. 24 illustrates a screen configuration example of the image management user interface. A photographed image (image) is displayed in an area indicated by reference numeral 2704. In the area indicated by reference numeral 2751, the shooting time is displayed, and in the area indicated by reference numeral 2754, the recognition type is displayed according to the priority order, and the value of each item is displayed and output to the right thereof.
[0111]
Reference numerals 2761 to 2763 denote a group of command buttons. When a selection operation such as clicking one of the buttons with a mouse is applied, the corresponding command processing is applied to the currently displayed photograph.
[0112]
In an area indicated by reference numeral 2764, thumbnailed photographs are listed in descending order of image points. The photo selected on the thumbnail list 2764 is displayed and output on the display area 2704. Using a jog dial, cursor keys, a mouse pointer, etc., a desired picture can be selected from the thumbnail list.
[0113]
The calculation formula for calculating the image points is, for example, as follows. That is, it is expressed as a total sum of products obtained by multiplying the ranking point value of each subject recognized in the image by the priority for the type of recognition.
[0114]
(Equation 2)

[0115]
FIG. 25 shows how the recognition target index is changed by operating the up and down buttons.
[0116]
Reference numeral 5101 denotes a field in which a user-designated recognition target index is written. Reference numeral 5105 indicates a button for increasing the list order of the corresponding recognition unit by one, and reference numeral 5106 indicates a button for decreasing the list order of the corresponding recognition unit by one.
[0117]
In the illustrated example, the index list 5100 of the current recognition unit is composed of 5102 and Hikari 5103. On the other hand, as indicated by reference numeral 5107, when the recognition target “Hikari” is deleted, the recognition target “Naomi” is added, and the rank of the recognition target “Naomi” is lowered, the list rank is changed. , 5112, and Naomi 5218.
[0118]
FIG. 26 shows the state of the data changed by the change of the recognition target.
[0119]
In the example of the data format describing the recognition unit index shown in FIG. 23, a tag field describing the recognition unit for each recognition type is provided, and the tag field 510 of the recognition type “person” is used for photographing. “Nachi” and “Hikari” as recognition targets included in the image are described as tag information 511 and 512 with both point values 0.72 and 0.32.
[0120]
On the other hand, as a result of changing the recognition unit index as shown in FIG. 25, the list 5110 of the recognition type “person” is changed to “5112” and “name 5218”.
[0121]
In the above-described embodiment, it is assumed that the photographing-side device 31 has a communication function as shown in FIGS. However, the communication function itself is not essential for realizing the present invention.
[0122]
Hereinafter, a modified example will be described in which a subject is recognized when a photograph is taken by a photographing device having no communication function such as a general digital camera.
[0123]
The photographing device in this case is, for example, a general digital camera having no communication function. In addition, a movement log recording device (to be described later) is lent to the user together with the digital camera, and is used for storing a shooting state. Then, the rented moving log recording device is collected, the photographing log and the moving log are taken out, and log analysis is performed, thereby performing a process of calculating a photographing space and recognizing a subject. Finally, distribute this result to users.
[0124]
As shown in FIG. 28, the digital camera has a lens 2811-1 that forms a photographic optical system on the front surface 2811 of the device. On the back surface 2812, a display unit 2812-2 for displaying a captured image and an input screen and an input unit 2812-4 for performing device operations such as setting of photographing conditions, view and deletion of a photograph, and the like are provided. An antenna 121 for receiving a GPS signal for position measurement and a shutter button 2812-41 for instructing capture of an image are provided on the upper surface of the device.
[0125]
FIG. 29 shows the internal configuration of this photographing device. The operation of the photographing device is controlled by the CPU 415 executing each program for realizing the camera function under the control of the operating system. The CPU 415 is interconnected to each unit via a bus 417.
[0126]
The RAM 413 is used to load an execution program code of the CPU 415 and temporarily store work data when the camera function is activated. The ROM 414 permanently stores information written at the time of factory shipment, such as an execution program code of the CPU 415 and manufacturing information.
[0127]
The input unit 408 includes buttons that can be operated by a user, and is used for data input. One of the operation buttons is assigned to the shutter 409.
[0128]
The position measuring unit 403 measures the current position of the device based on the GPS signal received by the antenna 121. The direction obtaining unit 404 includes a digital magnetic compass or the like, and obtains the orientation of the device or the direction of the camera / lens. The position measurement includes a position error based on the signal strength of the GPS signal and the spatial spread of the GPS satellite. The position measurement unit 403 estimates the position error and outputs the position error. The direction measuring unit 404 outputs a direction error that is a fixed value.
[0129]
The imaging unit 405 includes a camera lens 2811-1, an imaging element that captures an image on an image plane thereof, a signal processing module that processes an image signal, and the like. In the present embodiment, the imaging unit 405 outputs a shooting state such as a camera position, a lens direction, a focal length, an angle of view, and an aperture value.
[0130]
The display unit 406 outputs the processing result of the CPU 415 on the screen. For example, a finder screen obtained via the camera lens 2811-1 or an image after shooting is displayed on the screen.
[0131]
The output unit 407 is configured by a device that provides audio output and vibration from a speaker and feedback to the user.
[0132]
The clock 416 measures real time and supplies a timer signal to the system. In the present embodiment, the clock 416 outputs the imaging time by the imaging unit 405 and the position measurement time by the position measurement unit 403.
[0133]
The photo storage unit 431 stores an image captured by the imaging unit 405. The photographing log storage unit 432 stores a photographing log including a photographing time, a photographing state, position measurement at the time of photographing, and error information included in direction acquisition for each photographed image.
[0134]
The ID holding unit 402 stores device identification information for device identification.
[0135]
When taking a photograph using the device shown in FIG. 29, the photographing unit 405 operates in synchronization with the input from the shutter 409 of the input unit 408 to take a photograph, and this photographed image is stored in the image storage unit 431. save. Along with taking a picture and storing an image, a clock 416 acquires a photographing time, a camera position obtained by the position measuring unit 403 and its error range, and a lens direction obtained by the direction obtaining unit 404 and its error range, The data is stored in the shooting log storage unit 432.
[0136]
FIG. 30 shows an internal configuration of a mobile log recording device that records a position information log of each user.
[0137]
The operation of the movement log recording apparatus is controlled by the CPU 3015 executing programs for realizing the mobile phone function and the camera function under the control of the operating system. The CPU 3015 is interconnected to each unit via a bus 3017.
[0138]
The RAM 3013 is used for loading an execution program code of the CPU 3015 and temporarily storing work data. The ROM 3014 permanently stores information written at the time of factory shipment, such as an execution program code of the CPU 3015 and manufacturing information.
[0139]
The clock 3016 measures the actual time and supplies a timer signal to the system. In the present embodiment, the clock 416 outputs the imaging time by the imaging unit 405 and the position measurement time by the position measurement unit 403.
[0140]
The position measuring unit 3003 measures the current position of the device based on a GPS signal received by an antenna (not shown). The position measurement includes a position error based on the signal strength of the GPS signal and the spatial spread of the GPS satellite. In the present embodiment, the position measurement unit 3003 estimates the position error and outputs the position error. The measurement result of the position measurement unit 3003 at regular time intervals is stored in the movement log storage unit 3034 as a movement log of the user who carries the device.
[0141]
The ID holding unit 3002 stores device identification information for device identification.
[0142]
FIG. 27 shows, in the form of a flowchart, a subject recognition processing procedure in a modification in which the photographing-side device 31 does not have a communication function.
[0143]
First, a photographing device such as a digital camera shown in FIGS. 28 and 29 and a movement log recording device shown in FIG. 30 are lent to a user (step S1) and used for photographing (step S2).
[0144]
After that, the lent device is collected from the user, and the shooting log and the movement log are taken out (step S3). Then, by performing log analysis, processing of calculating a photographing space and recognizing a subject is performed (step S4). Finally, the result is distributed to the user (step S5).
[0145]
FIG. 31 shows how the positional relationship between the subject and the camera is obtained from the movement log.
[0146]
Reference numeral 3101 denotes a movement log of the device on the photographing side. Reference numeral 3111 is a point on the movement log 3101 where a photograph was taken, and the time 3113 at that time is 12:35.
[0147]
On the other hand, the subject position is extracted from the movement log 3034 of the movement log recording device carried by the subject. Reference numeral 3121 indicates a subject position when a photograph is taken by the photographing side device at the photographing position 3111. Since the position of the moving log of the subject is recorded at regular time intervals in the moving log recording device, the moving log is obtained from the sampling value so as to correspond to the photographing time 3113.
[0148]
The photographing space 3112 at the time of photographing is obtained based on the photographing position 3111 and the photographing log taken out from the photographing device. Further, the subject position 3121 at the time of photographing is obtained based on the moving log extracted from the moving log recording device. Then, by comparing the photographing space 3112 with the subject position 3121, it is possible to recognize whether or not the subject is included in the photographed photograph. The ranking point value of the subject can be calculated according to the procedure described with reference to FIG.
[0149]
FIG. 32 shows a processing procedure of subject recognition in the present embodiment in the form of a flowchart.
[0150]
First, photographing data is put in a queue (step S11). Then, the photographing data is taken out one by one from this queue (step S12). At this time, if there is no unprocessed data (step S13), the entire processing routine ends.
[0151]
Next, the movement log for one person is extracted from the registered member table (step S14). Here, when there are no more unprocessed members (step S15), the process returns to step S12 to take out the next queue.
[0152]
Then, the position at the photographing time is acquired from the extracted movement log, and it is checked whether the position is in the photographing space (step S17). Then, when a movement log in the shooting space is found, a ranking point value for the subject is calculated (step S18), and the member ID and the ranking point value are stored (step S19). Thereafter, the process returns to step S14, and the process of recognizing the subject and calculating the ranking point value is repeatedly performed for the next registered member.
[0153]
Further, some recent photographing devices such as digital cameras are mounted so that the lens direction can be rotated with respect to the device body. In this case, the user of the photographing device can shoot himself by pointing the lens at himself while holding the device body. In this case, the user must recognize himself / herself, not from the subject list as shown in FIG.
[0154]
FIG. 33 shows an external configuration of an imaging device in which a lens unit rotates. On a rear surface 2812 of the device, there are provided a display unit 2812-2 for displaying a captured image and an input screen, and an input unit 2812-4 for performing device operations such as setting of photographing conditions, view and deletion of a photograph. An antenna 121 for receiving a GPS signal for position measurement and a shutter button 2812-41 for supporting image capture are provided on the upper surface of the device. Further, since the lens unit 2813 on which the lens 2811-1 is mounted is rotatably supported on the device main body so as to be rotatable in the direction of the arrow in the figure, the user can hold the device main body in front and rear (in his or her own direction). ) Can be taken in any direction.
[0155]
FIG. 34 shows the internal configuration of the imaging device in which the lens unit shown in FIG. 33 rotates.
[0156]
The operation of the photographing device is controlled by the CPU 415 executing programs for realizing the mobile phone function and the camera function under the control of the operating system. The CPU 415 is interconnected to each unit via a bus 417.
[0157]
The RAM 413 is used to load an execution program code of the CPU 415 and temporarily store work data when the mobile phone function or the camera function is activated. The ROM 414 permanently stores information written at the time of factory shipment, such as an execution program code of the CPU 415 and manufacturing information.
[0158]
The input unit 408 includes buttons that can be operated by the user, and is used for inputting a telephone number and other data. One of the operation buttons is assigned to the shutter 409 when the camera function is activated.
[0159]
The communication unit 401 performs a communication process with a base station on a mobile phone network, and further communicates with a server.
[0160]
The position measuring unit 403 measures the current position of the device based on the GPS signal received by the antenna 121. The direction obtaining unit 404 includes a digital magnetic compass or the like, and obtains the orientation of the device or the direction of the camera / lens.
[0161]
The imaging unit 405 includes a camera / lens, an imaging element that captures an image on an image plane thereof, a signal processing module that processes an image signal, and the like. In the present embodiment, the imaging unit 405 outputs a shooting state such as a camera position, a lens direction, a focal length, an angle of view, and an aperture value. As described above, the camera lens is rotatably supported with respect to the device body. Then, the rotation angle measurement unit 418 measures the rotation position of the lens unit 2813.
[0162]
The display unit 406 outputs the processing result of the CPU 415 on the screen. For example, a finder screen obtained through a camera lens or an image after shooting is displayed on the screen.
[0163]
The output unit 407 is a device that externally outputs an image signal, outputs sound from a speaker, vibrates, and provides feedback to a user.
[0164]
The clock 416 measures real time and supplies a timer signal to the system. In the present embodiment, the clock 416 outputs the imaging time by the imaging unit 405 and the position measurement time by the position measurement unit 403.
[0165]
The photo storage unit 431 stores an image captured by the imaging unit 405. The photographing log storage unit 432 stores a photographing log including a photographing time, a photographing state, position measurement at the time of photographing, and error information included in direction acquisition for each photographed image.
[0166]
Further, the device includes an ID holding unit 402 in which device identification information for device identification is stored, and is transmitted from the communication unit 401 to the server 33 together with the position information acquired by the position measuring unit 403. Further, the device IDs of other devices permitted to disclose their device positions are stored in the ID list 433.
[0167]
When taking a photograph on the mobile phone shown in FIG. 5, the photographing unit 405 operates in synchronization with the input from the shutter 409 of the input unit 408 to take a photograph, and the photographed image is stored in the image storage unit 431. save. Along with taking a picture and storing an image, a clock 416 acquires a photographing time, a camera position obtained by the position measuring unit 403 and its error range, and a lens direction obtained by the direction obtaining unit 404 and its error range, The data is stored in the shooting log storage unit 432.
[0168]
The rotation angle measurement unit 418 measures the rotation angle of the lens unit 2813 during shooting. Here, when the lens unit 2813 faces the front, the ID stored in the ID holding unit 402 is not included in the subject, but when the lens unit 2813 faces the rear, the ID is stored in the ID holding unit 402. By including the ID in the subject, the self is included in the subject.
[0169]
As described above, in the present embodiment, the camera position information and the lens direction, and the shooting state including the focal length, the angle of view, the aperture value, and the like are acquired, and the camera position is centered, and the focus plane and the depth of field are set. A portion corresponding to the angle of view in the lens direction in a region of a radius range determined based on the calculated value is calculated as a shooting space. Then, the position information of each subject is compared with the shooting space, and a person in the shooting space is recognized as the subject. Furthermore, when calculating the ranking point value for the recognition target in the shooting space, the weight of the shooting space, that is, the weight according to the center angle and the distance from the focus plane is weighted, and the camera position is determined by the camera position error. Is weighted in accordance with.
[0170]
The method of calculating the ranking point value for each recognized subject has been schematically described with reference to FIG. 21, but the detailed processing will be described below.
[0171]
FIG. 35 shows a photographing area. Each piece of photograph data includes a photographing position of the camera 1, a lens direction 4001, an angle of view 4003, a focus distance 4022, focal length information, and an aperture value. The photographing area 4002 is calculated using these photographing state parameter values. Here, an arc passing through each point represented by reference numerals 4012, 4022, and 4032 represents a focus plane. An arc passing through each point represented by reference numerals 4013, 4023, and 4033 represents the front depth of field. An arc passing through each point represented by reference numerals 4011, 4021, and 4031 represents a rear depth of field.
[0172]
As described above, the imaging space is weighted according to the central angle and the distance from the focus plane. FIG. 36 shows a state of the weight gradient in the photographing space. A graph indicated by reference numeral 4101 indicates a weight inclination in the camera direction, and a graph indicated by reference numeral 4102 indicates a weight inclination in the horizontal direction of the focus plane. In the present embodiment, as shown in the figure, the weight is reduced in the vertical direction (points 4023 and 4021) and in the horizontal direction (points 4012 and 4032) around the point 4022.
[0173]
FIG. 37 shows, in the form of a flowchart, a processing procedure for calculating the ranking point value of the subject.
[0174]
First, a camera position and a lens direction are input (step S21). Next, the position of the user as a subject is input (step S22). As a result of the culling (step S23), if the culling is performed, 0 is returned (step S25), otherwise, the ranking point value is calculated (step S26).
[0175]
FIG. 38 shows, in the form of a flowchart, a detailed procedure of the culling process corresponding to step S23 in the flowchart shown in FIG.
[0176]
First, a camera position and a lens direction are input (step S31). Next, the position of the user who is the subject is input (step S32), and a circle having the minimum radius including the target object is created as a boundary circle (step S33). Next, when the distance condition described in FIG. 39 is satisfied (step S34), the angle condition 1 described in FIG. 40 is satisfied (step S35), and the angle condition 2 described in FIG. 41 is satisfied (step S36), TRUE is returned (step S37). ), Otherwise returns FALSE (step S38).
[0177]
FIG. 39 shows how the culling distance condition is determined. Reference numeral 4411 indicates the camera position, reference numeral 4412 indicates the error radius of the camera position, reference numeral 4421 indicates the center position of the subject boundary circle, reference numeral 4422 indicates the radius of the subject boundary circle, reference numeral 4432 indicates the shooting area, Reference numeral 4436 indicates the focus distance, reference numeral 4434 indicates the rear depth of field, and reference numeral 4435 indicates the front depth of field. Reference numeral 4437 indicates a vector from the camera position toward the center of the subject boundary circle.
[0178]
The culling distance condition is determined according to the following equation. According to this equation, the magnitude of the vector indicated by reference numeral 4436 is different from the focus position indicated by reference numeral 4436 by the error of the camera position in the width between the front depth of field 4435 and the rear depth of field 4434. It is a condition that the camera enters with a margin of a radius 4412 and a radius 4422 of the radius of the subject boundary circle.
[0179]
[Equation 3]

[0180]
FIG. 40 shows how the culling angle condition 1 is determined. In the figure, reference numeral 4411 denotes the camera position, reference numeral 4412 denotes the error radius of the camera position, reference numeral 4421 denotes the center position of the subject boundary circle, reference numeral 4422 denotes the radius of the subject boundary circle, and reference numeral 4431: The lens direction, reference numeral 4433 indicates the angle of view, and reference numeral 4432 indicates the shooting space. Reference numeral 4438 indicates an angle-of-view limit vector on the right side in the lens direction. The inner product of a vector 4439 orthogonal to this and a vector 4437 from the camera position 4411 to the center position 4422 of the subject boundary circle is calculated. I do. The value of the inner product represents the signed distance from the vector 4438 to the center position 4422 of the subject boundary circle.
[0181]
The following equation is an equation for determining the culling angle condition 1. The signed distance obtained in FIG. 45 has a margin of the camera position error radius 4412 and the subject boundary circle radius 4422 in the width of the front depth of field 4434 and the rear depth of field 4435 around the focus distance 4436. Is required.
[0182]
(Equation 4)

[0183]
FIG. 41 shows how the culling angle condition 2 is determined. In the figure, reference numeral 4411 denotes the camera position, reference numeral 4412 denotes the error radius of the camera position, reference numeral 4421 denotes the center position of the subject boundary circle, reference numeral 4422 denotes the radius of the subject boundary circle, and reference numeral 4431 denotes The lens direction, reference numeral 4433 indicates the angle of view, and reference numeral 4432 indicates the shooting space. Reference numeral 4438 indicates an angle-of-view limit vector on the left side in the lens direction, and calculates an inner product of a vector 4439 orthogonal to this vector and a vector 4437 from the camera position 4411 to the center position 4422 of the subject boundary circle. . The value of the inner product represents the signed distance from the vector 4438 to the center position 4422 of the subject boundary circle.
[0184]
The following equation shows an equation for determining the culling angle condition 2. The signed distance obtained in FIG. 46 has a margin of the camera position error radius 4412 and the radius 4422 of the subject boundary circle in the width of the front depth of field 4434 and the rear depth of field 4435 around the focus distance 4436. It is a condition that you bring it in.
[0185]
(Equation 5)

[0186]
FIG. 42 shows, in the form of a flowchart, a processing procedure for calculating a rank value for a subject person. Here, the error circle portion of the subject person is integrated.
[0187]
First, the position O of the subject person is input (step S41). Then, the rank sum Sum is initialized to 0 (step S42), and the radius variable r is initialized to 0 (step S43).
[0188]
Next, a weight parameter w that decreases as the distance between the camera positions increases is calculated (step S44), and the angle variable θ is initialized to 0 (step S45).
[0189]
Next, point coordinates P within the error circle of the subject person position are obtained (step S46), and a ranking point value when the camera is assumed to be at the position P is calculated and added to sum (step S47).
[0190]
Next, the angle increment dθ is added to θ (step S48). If θ does not exceed 2π (step S49), the process moves to step S46.
[0191]
Next, the distance step dr is added to r (step S50). If r does not exceed the error radius Cr (step S51), the process moves to step S44.
[0192]
Then, the error radius area S is calculated (step S52), sum is normalized by S and output (step S53), and the entire processing routine ends.
[0193]
FIG. 43 shows, in the form of a flowchart, a processing procedure for calculating a rank value for a subject person. Here, the error circle portion of the camera position is integrated.
[0194]
First, the position O of the subject person is input (step S61). Then, the rank sum Sum is initialized to 0 (step S62), and the radius variable r is initialized to 0 (step S43).
[0195]
Next, a weight parameter w that decreases as the distance of the camera position increases is calculated (step S64), and the angle variable θ is initialized to 0 (step S65).
[0196]
Next, the point coordinates P within the error circle of the camera position are obtained (step S66), and the ranking point value when the camera is assumed to be at the position P is calculated and added to sum (step S67).
[0197]
Next, the angle increment dθ is added to θ (step S68), and if θ does not exceed 2π (step S69), the process moves to step S66.
[0198]
Next, the distance increment dr is added to r (step S70). If r does not exceed the error radius Cr (step S71), the process moves to step S64.
[0199]
Then, the error radius area S is calculated (step S72), the sum is normalized by S and output (step S73), and the entire processing routine ends.
[0200]
FIG. 44 shows how the distance condition is determined. In the figure, reference numeral 4436 denotes a focus distance, reference numeral 4434 denotes a rear depth of field, reference numeral 4435 denotes a front depth of field, and reference numeral 4432 denotes a shooting space. Reference numeral 4437 indicates a vector from the camera position 4411 to the subject position 4421.
[0201]
The following equation shows an equation for determining the distance condition. The condition is that the length of the vector 4437 is included in the range of the front depth of field 4435 and the rear depth of field 4434 around the focus distance 4436.
[0202]
(Equation 6)

[0203]
FIG. 45 shows how the angle condition is determined. In the figure, reference numeral 4431 indicates an angle from the north to the lens direction, reference numeral 4432 indicates an angle of view, and reference numeral 4438 indicates a lens direction vector. Reference numeral 4437 indicates a vector from the camera position 4411 to the subject position 4421. Reference numeral 4439 indicates an angle between the vector 4437 and the lens direction vector 4438.
[0204]
The following equation shows an equation for determining the angle condition. The condition is that the angle 4439 obtained in FIG. 51 is smaller than the angle of view 4433.
[0205]
(Equation 7)

[0206]
FIG. 46 shows how the front and rear object scenes are divided. In the figure, reference numeral 4436 denotes a focus distance, reference numeral 4434 denotes a rear depth of field, reference numeral 4435 denotes a front depth of field, reference numeral 4432-1 denotes a photographing space within the front depth of field, Reference numeral 4432-2 indicates an imaging space within the rear depth of field. Reference numeral 4437 indicates a vector from the camera position 4411 to the subject position 4421.
[0207]
The rank value in the front and rear scenes is calculated by the following equation. As shown in FIG. 52, the calculation formula is different when the subject is in the front-depth-of-field imaging space 4432-1 and when the subject is in the rear-depth-of-field imaging space 4432-2.
[0208]
(Equation 8)

[0209]
[Supplement]
The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the scope of the present invention. That is, the present invention has been disclosed by way of example, and the contents described in this specification should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims described at the beginning should be considered.
[0210]
【The invention's effect】
As described above in detail, according to the present invention, an excellent image management system, an excellent image management method, and a computer program capable of suitably managing a photographic image including one or more moving objects as subjects are provided. can do.
[0211]
Further, according to the present invention, there is provided an excellent image management system and an image management system capable of recognizing a subject such as a person appearing in a photographed photograph and facilitating the management of the photograph by combining the photograph and the subject. A management method and a computer program can be provided.
[0212]
Further, according to the present invention, it is possible to provide an excellent image management system, an excellent image management method, and a computer program capable of performing a prioritization among a plurality of photographing targets and performing practical object recognition. it can.
[0213]
Since mobile devices with a GPS function for measuring the current position have become widespread, it is possible to perform subject recognition based on a subject position using personal position information. Also, in order to cope with a situation where the accuracy of the position information is not good, the position measurement error is regarded as the probability that the target exists. According to the present invention, it is possible to perform weighting based on ranking for a plurality of subjects that may have been photographed, and to present them to the user as a list ranking. The user can greatly reduce the trouble of describing who is being photographed. In addition, by carrying a receiver with a position measuring device to each photographing target, it becomes possible to recognize a person who could not be described on the map data. According to the present invention, in a situation where the measurement accuracy is not sufficiently high, it is possible to calculate the priority of a target set considered to be a subject using the measurement error and present the calculated priority to the user.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a state in which a subject is recognized using a camera position, a lens direction, and map information.
FIG. 2 is a diagram schematically showing a system configuration of an image management system according to an embodiment of the present invention.
FIG. 3 is a diagram for explaining a mechanism in which a subject recognition process is performed based on subject position information and a shooting state;
FIG. 4 is a diagram showing an external configuration of a device carried by each user.
FIG. 5 is a diagram showing an internal configuration of the device shown in FIG.
FIG. 6 is a diagram showing an external configuration of a device carried by each user (without a camera function).
FIG. 7 is a diagram showing an internal configuration of the device shown in FIG.
8 is a diagram schematically illustrating a configuration of a server that communicates with each device illustrated in FIGS. 4 and 6. FIG.
FIG. 9 is an operation sequence diagram showing a processing procedure for making a use permission application of position information to a subject.
FIG. 10 is a sequence diagram showing a processing procedure when an application for rejecting use of positional information to a subject is made and the application is rejected.
FIG. 11 is a diagram illustrating an example in which a subject included in an image photographed on the photographing device 31 is recognized by the center server 33, a ranking point is assigned to each subject, and the subject is provided to the device 31; FIG. 9 is an operation sequence diagram showing a processing procedure for performing a user editing operation based on points.
FIG. 12 is a diagram showing information acquired at the time of photographing on the photographing side device 31.
FIG. 13 is a diagram illustrating a configuration example of a data format for recording a photographing state acquired at the time of photographing.
FIG. 14 is a diagram showing an example of a method of expressing a lens direction of a camera.
FIG. 15 is a diagram showing a relationship between a camera position, a lens direction, and a subject on a cell-divided map.
FIG. 16 is a diagram showing a configuration example of a subject list for which the device has been authenticated.
FIG. 17 is a diagram showing a state where a recognition target existing in a cell is registered.
FIG. 18 is a diagram illustrating a state in which a cell including a shooting space is selected.
FIG. 19 is a diagram illustrating a state in which a recognition target is acquired from a subject list for each device.
FIG. 20 is a diagram illustrating a state in which a recognition target is acquired from a selected cell by using an in-cell recognition target reference table.
FIG. 21 is a diagram showing how to calculate a ranking point for a recognition unit in an imaging space.
FIG. 22 is a diagram illustrating a state in which a recognition target index is obtained from a captured image.
FIG. 23 is a diagram showing a configuration example of a data format for describing a recognition target index.
FIG. 24 is a diagram showing a screen configuration example of an image management user interface based on ranking points.
FIG. 25 is a diagram illustrating a state in which a recognition target index is changed by operating an up / down button.
FIG. 26 is a diagram showing a state of data changed by a change of a recognition target.
FIG. 27 is a flowchart illustrating a processing procedure of subject recognition when the photographing-side device 31 does not have a communication function.
FIG. 28 is a diagram illustrating an example of an external configuration of a photographing device having no communication function.
FIG. 29 is a diagram illustrating an internal configuration of a photographing device having no communication function.
FIG. 30 is a diagram showing an internal configuration of a movement log recording device.
FIG. 31 is a diagram illustrating a process of acquiring a positional relationship between a subject and a camera from a movement log.
FIG. 32 is a flowchart showing a procedure of subject recognition when the device does not have a communication function.
FIG. 33 is a diagram illustrating an external configuration of a photographing apparatus in which a lens unit rotates.
FIG. 34 is a diagram illustrating an internal configuration of a photographing apparatus in which a lens unit rotates.
FIG. 35 is a diagram for explaining a method of calculating a ranking point value for a subject.
FIG. 36 is a diagram for explaining a method of calculating a ranking point value for a subject.
FIG. 37 is a diagram illustrating a method of calculating a ranking point value for a subject.
FIG. 38 is a diagram for explaining a method of calculating a ranking point value for a subject.
FIG. 39 is a diagram for explaining a method of calculating a ranking point value for a subject.
FIG. 40 is a diagram for explaining a method of calculating a ranking point value for a subject.
FIG. 41 is a diagram for explaining a method of calculating a ranking point value for a subject.
FIG. 42 is a diagram for explaining a method of calculating a ranking point value for a subject.
FIG. 43 is a diagram illustrating a method of calculating a ranking point value for a subject.
FIG. 44 is a diagram illustrating a method of calculating a ranking point value for a subject.
FIG. 45 is a diagram for explaining a method of calculating a ranking point value for a subject.
FIG. 46 is a diagram for explaining a method of calculating a ranking point value for a subject.
[Explanation of symbols]
101 ... Imaging device
102: shooting state acquisition unit
103 ... Subject recognition unit
104: Ranking / point giving unit
105 ... Image storage unit
106 image search / editing unit
401 ... communication unit
402 ... ID holding unit
403 Position measurement unit
404 ... Direction acquisition unit
405 ... Imaging unit
406 display unit
407 ... Output unit
408 input unit
409 ... Shutter
413 ... RAM
414 ... ROM
415 ... CPU
416 ... clock
417… Bus
431 ... Photo storage unit
432: shooting log storage unit
433 ... ID list
501: Communication unit
510... Shooting target range calculation unit
511 subject list acquisition unit
512: Ranking / point calculator
513 ... RAM
514 ... ROM
515 ... CPU
521: Terminal location information storage unit
522 ID storage unit
524: Map information storage unit
525: Entertainment Calendar Storage Unit

Claims (13)

An image management system that manages a photographic image including one or more moving objects as a subject in combination with the subject,
Shooting state obtaining means for obtaining a shooting state at the time of image shooting,
Subject position obtaining means for obtaining the position of each moving body at the time of image capturing,
A photographing space estimating means for calculating a photographing space to be photographed in a photographed image based on the photographing state;
A subject recognition unit that compares a shooting space calculated by the shooting space estimation unit with a position of each moving body obtained by the subject position obtaining unit, and recognizes a moving body in the shooting space as a subject;
Subject evaluation value calculation means for calculating an evaluation value according to a situation in the image of the recognized subject,
An image management system comprising: Subject management means for managing subjects included in each image according to a priority order based on the evaluation value;
Image search means for searching for an image including the subject according to the priority order;
The image management system according to claim 1, further comprising: The photographing state acquiring means acquires a camera position, a lens direction, a focal length, an angle of view, and an aperture value at the time of photographing as a photographing state,
The photographing space estimating means calculates a photographing space including a focus plane and a depth of field based on the instruction values of these photographing states.
The image management system according to claim 1, wherein: The subject evaluation value calculation means calculates an evaluation value based on the likelihood that the subject exists in the shooting space,
The image management system according to claim 1, wherein: The subject evaluation value calculation means calculates an evaluation value by giving a weight based on a shooting position error, a gaze direction error, and a position measurement error of the subject to the likelihood that the subject exists in the shooting space,
The image management system according to claim 4, wherein: Further provided is a position information use permission unit that obtains in advance use permission of position information from a moving body that can be a subject,
The subject position obtaining means obtains position information of a moving body for which use permission of position information has been obtained, and / or the subject recognition means performs a subject recognition process only for a moving body for which use permission of position information has been obtained.
The image management system according to claim 1, wherein: An image management method for managing a photographic image including one or more moving objects as a subject in combination with the subject,
A photographing state acquiring step for acquiring a photographing state at the time of photographing an image,
Subject position obtaining step of obtaining the position of each moving body at the time of image capturing,
A photographing space estimation step of calculating a photographing space to be photographed in a photographed image based on the photographing state;
A subject recognition step of collating a shooting space obtained in the shooting space estimation step with a position of each moving body obtained in the subject position obtaining step, and recognizing a moving body in the shooting space as a subject;
A subject evaluation value calculating step of calculating an evaluation value according to a situation in the image of the recognized subject;
An image management method, comprising: A subject management step of managing subjects included in each image according to a priority order based on the evaluation value;
An image search step of searching for an image including the subject according to the priority order;
The image management method according to claim 7, further comprising: In the shooting state obtaining step, a camera position, a lens direction, a focal length, an angle of view, and an aperture value at the time of shooting are obtained as a shooting state,
In the photographing space estimation step, a photographing space including a focus plane and a depth of field is calculated based on the instruction values of the photographing states.
The image management method according to claim 7, wherein: In the subject evaluation value calculation step, an evaluation value is calculated based on the likelihood that the subject exists in the shooting space,
The image management method according to claim 7, wherein: In the subject evaluation value calculation step, an evaluation value is calculated by giving a weight based on a shooting position error, a line-of-sight direction error, and a position measurement error of the subject to the likelihood that the subject exists in the shooting space,
The image management method according to claim 10, wherein: A position information use permission step of obtaining in advance use permission of position information from a moving body that can be a subject,
In the subject position obtaining step, position information of a moving body for which use of position information is permitted is obtained, and / or in the subject recognition step, subject recognition processing is performed only for a moving body for which use of position information is permitted.
The image management method according to claim 7, wherein: A computer program written in a computer-readable format to execute a process for managing a photographic image including one or more moving objects as a subject in combination with the subject on a computer system,
A photographing state acquiring step for acquiring a photographing state at the time of photographing an image,
Subject position obtaining step of obtaining the position of each moving body at the time of image capturing,
A photographing space estimation step of calculating a photographing space to be photographed in a photographed image based on the photographing state;
A subject recognition step of collating a shooting space obtained in the shooting space estimation step with a position of each moving body obtained in the subject position obtaining step, and recognizing a moving body in the shooting space as a subject;
A subject evaluation value calculating step of calculating an evaluation value according to a situation in the image of the recognized subject;
A computer program comprising:

Images