JP2008139911A - Information processing apparatus and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for facilitating consistency between an index used in index calibration and an index managed by an index tree.
When index information including position and orientation information of an index placed in a real space is input by a user operation, the input index information is registered in a memory as regular registration information. The position and orientation information of the index included in the captured image obtained by imaging the real space is obtained. If the index information including the obtained position and orientation information is not registered in the memory, the position and orientation information is obtained. Is registered in the memory as temporary registration information (S1404). The first display content related to the index corresponding to the regular registration information and the second display content related to the index corresponding to the temporary registration information are displayed in different display forms (S1402).
[Selection] FIG.

Description

  The present invention relates to a technique for managing information related to each index arranged in a real space.

  In recent years, research on mixed reality (MR) technology has been actively conducted. MR technology is a technology for seamlessly fusing a virtual space created by a computer to a real space. Among MR technologies, Augmented Reality (also referred to as AR, augmented reality, augmented reality) technology that displays a virtual space superimposed on a real space is attracting particular attention.

  An image display device used in the AR technology is realized by a video see-through method or an optical see-through method. The video see-through method is a method of displaying a composite image in which an image in a virtual space generated according to the position and orientation of the imaging device is superimposed on a real space image taken by an imaging device such as a video camera. That is. Here, the image in the virtual space is a virtual object or character information drawn by computer graphics. On the other hand, the optical see-through method is a method of displaying an image of a virtual space generated according to the position and orientation of the observer's viewpoint on an optical see-through display mounted on the observer's head.

  AR technology includes various types of support, such as surgery support that superimposes the state of the body on the patient's body surface, architectural simulation that superimposes and displays a virtual building on a vacant lot, and assembly support that superimposes and displays work procedures and wiring during assembly work. Application to the field is expected.

  One of the most important issues in AR technology is how to accurately align the real space and the virtual space, and many efforts have been made. The alignment problem in the AR technique results in a problem of obtaining the position and orientation of the imaging device in the scene (that is, in the reference coordinate system) in the video see-through method. As a method for solving this problem, the following method has been conventionally used. A plurality of indices whose coordinate values in the reference coordinate system are known are arranged or set in the scene. Then, the position and orientation of the imaging apparatus in the reference coordinate system are obtained from the correspondence between the projected position of the index in the image captured by the imaging apparatus and the position of the index in the reference coordinate system. In general, if the position of a plurality of known three-dimensional points (theoretically 3 points or more, 6 points or more for stable solving) on the captured image can be obtained, the position of the camera viewpoint from the corresponding relationship And the orientation posture can be obtained (solving the so-called PnP problem).

  When using the alignment method using such an index, it is necessary to prepare an index with a known world coordinate value in advance. The index may be artificially defined or may use a natural feature or the like, but it is desirable that the index is always arranged so that the index is photographed in the moving camera.

  On the other hand, a method for calculating the position and orientation of an imaging device from a set of three-dimensional coordinates and image coordinates of an index has been proposed for a long time in the field of photogrammetry (see Non-Patent Document 1 and Non-Patent Document 2).

  Further, not only indices (hereinafter referred to as point indices) as points in a plurality of three-dimensional spaces but also square-shaped indices (hereinafter referred to as A technique for calculating the position and orientation of an imaging apparatus using a square index has also been proposed. In addition, as disclosed in Non-Patent Document 7, for example, a method for calculating the position and orientation of an imaging device that combines a square index and a point index has been proposed. The point index has the merit that it can be set in a narrow place. The square index has the advantages of "easy identification" and "the position and orientation of the imaging device can be obtained from only one index because of the large amount of information in one index". can do.

  Here, the position and orientation of the index in the reference coordinate system can be measured manually by using a tape measure or a protractor, or by a surveying instrument. (Hereinafter referred to as index calibration).

  For example, the position measurement of the point index can be performed by a method called a bundle adjustment method. In the bundle adjustment method, a large number of point indices are photographed by an imaging device, and the position of the point index in the real space, the projection point on the image of the point index, and the viewpoint of the imaging device are present on the same straight line. Use conditions. Then, iterative calculation is performed so that the error (projection error) between the projection position where the index is actually observed on the image and the projection position calculated from the position and orientation of the imaging device and the position of the index is minimized. The position and orientation of the imaging device that captured the image and the position of the point index are obtained.

  Non-Patent Document 6 discloses an index calibration method for measuring the positions and orientations of a large number of square indexes arranged in a three-dimensional space. Non-Patent Document 6 takes a large number of images of a large number of square markers arranged in a three-dimensional space. Then, the position and orientation of the imaging device that captured each image by iterative calculation and the position and orientation of the square index are obtained so that the projection error is minimized.

  Also, an index calibration method for accurately obtaining index placement information that minimizes a projection error for a plurality of types of indices has been conventionally disclosed.

  Here, in order to obtain the index image position, it is necessary to perform a process of extracting a known index feature (in accordance with the index extraction algorithm) from the index image, and to extract the index portion. As the known feature, an actual index shape (index shape) or an index color (index image color) on an index image obtained by photographing the index with an imaging device is used.

  In order to capture and extract one or more indices placed in the three-dimensional real space and estimate camera parameters based on the camera calibration algorithm, definition information of the indices must be determined in advance in the information processing apparatus. There is. Here, the definition information of the index is what kind of index is in what position in the real space, and the color and shape for identifying the index. The index thus determined is hereinafter referred to as “registered index”.

  The index definition information is managed in a data format of a tree structure (also referred to as a tree structure or a hierarchical structure) in the information processing apparatus (hereinafter referred to as an index tree). The definition information related to these indices can be set by a user operating a graphical user interface (hereinafter, GUI) that operates on the information processing apparatus.

  FIG. 2 is a diagram showing a display example of a conventional GUI operated by the user.

  In the figure, reference numeral 280 denotes a tree view that graphically shows how indexes are managed in a data format having a tree structure, in which defined coordinate systems and indexes are hierarchically displayed. In the figure, a reference coordinate system and five indexes named YELLOW, ORANGE, GREEN, BLUE, and MAGENTA are defined. Then, the indicators YELLOW, ORANGE, GREEN, BLUE, and MAGENTA are shown belonging to the coordinate system world.

Reference numerals 230 to 238 denote text boxes for inputting / outputting definition information of the index selected in the tree view 280. In the tree view 280 of the figure, an index ORANGE is selected, and the state of the ORANGE character of the selected index being highlighted is shown. In 230 to 238, definition information related to the selected index ORANGE is displayed.
JP 2005-326274 A R. M.M. Haralick, C.I. Lee, K.M. Ottenberg, and M.M. Nole: "Review and analysis of solutions of the three point perspective positive estimation problem", Int'l. J. et al. Computer Vision, vol. 13, no. 3, pp. 331-356, 1994. M.M. A. Fischler and R.M. C. Bolles: “Random sample consensus: a paradigm for model fitting with applications to imaging analysis”, Comm. ACM, vol. 24, no. 6, pp. 381-395, 1981. Junichi Kokimoto: “Augmented Reality Construction Method Using Two-dimensional Matrix Code”, Interactive System and Software IV, Modern Science, 1996. Kato, M.M. Billinghurst, Asano, Tachibana: "Augmented reality system based on index tracking and its calibration", Transactions of the Virtual Reality Society of Japan, vol. 4, no. 4, pp. 607-616, 1999. A. State, G. Hirota, D.H. T.A. Chen, W .; F. Garrett and M.M. A. Livingston: “Superior augmented reality registration by integrating landmark tracking and magnetic tracking”, Proc. SIGGRAPH '96, pp. 429-438, 1996. G. Baratoff, A.M. Neubeck and H.M. Regenbrecht: “Interactive multi-marker calibration for augmented reality applications”, Proc. ISMAL2002, pp. 107-116, 2002. H. Kato, M .; Billinghurst, I.M. Poppyrev, K.M. Imamoto and K.M. Tachibana: “Virtual object manipulation on a table-top AR environment”, Proc. ISAR2000, pp. 111-119, 2000.

  Here, the index displayed in the index tree is an index registered in advance in the information processing apparatus, or an index input by the user through a GUI or the like. On the other hand, the index calibration handles the index extracted by the index extraction algorithm. An unregistered index may be processed by index calibration, or conversely, it may be registered in the index tree but not processed by index calibration. In other words, the managed index and the index handled by the index calibration may be different. Since it is difficult to achieve these consistency, improvement has been demanded.

  The present invention has been made in view of the above problems, and provides a technique for facilitating the consistency between an index used in index calibration or the like and an index managed by an index tree. Objective.

  In order to achieve the object of the present invention, for example, an information processing apparatus of the present invention comprises the following arrangement.

That is, index information acquisition means for acquiring index information regarding each of a plurality of indices,
Index registration status determination means for determining the registration status of the index;
Generating means for generating a user interface for editing the index information according to a user instruction based on the index information and the determination result of the index registration state determination means;
The registration status includes a temporary registration status and a regular registration status,
When the information automatically calculated by the index calibration is registered in the index information, the temporary registration state is set as the registration state of the index information.

  In order to achieve the object of the present invention, for example, an information processing method of the present invention comprises the following arrangement.

That is, a first registration step of registering, in the memory as regular registration information, index information including position information of an index placed in the real space, which is input by a user;
An acquisition step of acquiring a captured image of the real space imaged by the imaging device;
An index position calculating step of extracting an index from the captured image and calculating position information of the index;
A second registration step of registering the position information calculated by the index position calculation step in the memory as temporary registration information;
And a display step of displaying an index corresponding to the regular registration information and an index corresponding to the temporary registration information in different display forms.

  According to the configuration of the present invention, it is possible to easily achieve consistency between an index used in index calibration or the like and an index managed by the index tree.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment]
<System configuration>
First, the configuration of the system according to the present embodiment will be described. FIG. 1 is a block diagram showing a hardware configuration of a system according to the present embodiment. The system according to the present embodiment includes an information processing device 100, an imaging device 160, and a display device 170. Hereinafter, each device will be described.

  First, the information processing apparatus 100 will be described. The information processing apparatus 100 is configured by, for example, a general PC (personal computer) or WS (workstation). The information processing apparatus 100 includes a CPU 110, a memory 120, an HDD (hard disk drive device) 130, an image acquisition unit 140, an image generation unit 150, a keyboard 180, and a mouse 190. Note that the configuration is a list of the main ones, and other configurations may be added, and any configuration can be used as long as processing equivalent to the processing performed by the configuration can be performed. Also good.

  The CPU 110 controls the entire information processing apparatus 100 using programs and data loaded in the memory 120, and executes each process described later performed by the information processing apparatus 100.

  The memory 120 has an area for temporarily storing programs and data loaded from the HDD 130 and an area for temporarily storing image data acquired from the imaging device 160 by the image acquisition unit 140. Furthermore, the memory 120 also has a work area used when the CPU 110 executes various processes. That is, the memory 120 can provide various areas as appropriate.

  The HDD 130 stores an OS (Operating System), an index definition information management program 200, various information described as known information, and the like. Programs and data stored in the HDD 130 are appropriately loaded into the memory 120 under the control of the CPU 110 and are processed by the CPU 110.

  The index definition information management program 200 is for causing the CPU 110 to execute various processes related to the index. The index definition information management program 200 causes the CPU 110 to execute a process of displaying a tree view that graphically shows how indexes are managed in a data format having a tree structure. Further, the index definition information management program 200 causes the CPU 110 to execute processing for managing index definition information, displaying an index color area on the display device 170, and setting a recognition color area.

  The HDD 130 also stores data necessary for operating the index definition information management program 200, and this is also loaded into the memory 120 as appropriate under the control of the CPU 110.

  A keyboard 180 and a mouse 190 are used by the user to input various instructions.

  The image acquisition unit 140 receives the real space image of each frame sequentially sent from the imaging device 160, converts it into a data format that can be handled by the index definition information management program 200, and sequentially transfers it to the memory 120 or the HDD 130. To do.

  The image generation unit 150 generates a screen to be displayed on the display device 170.

  Next, the imaging device 160 will be described. The imaging device 160 captures a moving image in the real space where a plurality of indices are arranged, and sends an image of each frame to the image acquisition unit 140.

  Next, the display device 170 will be described. The display device 170 is configured by a CRT, a liquid crystal screen, or the like.

  The index definition information management program 200 will be described. First, the index definition information will be described.

  The index definition information includes index information of each index arranged in the real space. The index information exists for each index, and includes information unique to the index, the coordinate system to which the index belongs, the position (and orientation) of the index, the recognized color area of the index, the index shape, and the like.

  In some cases, the index definition information is only the position (and posture) of the index. In this case, the user can set information specific to the index, the coordinate system to which the index belongs, the recognition color area of the index, the index shape, and the like.

  The index definition information management program 200 provides a GUI for editing the index definition information, generates the edited result as a file (index definition information file) in a predetermined format, or reads an existing index definition information file Or

  The index definition information management program 200 displays the index corresponding to the temporarily registered index information and the index corresponding to the regularly registered index information in different display forms.

FIG. 10 is a diagram illustrating a configuration example of the index definition information file.
In the index definition information file of FIG. 10, a world coordinate system (world coordinate system) is defined as a coordinate system. In the world coordinate system, five indices having index names of YELLOW, ORANGE, GREEN, BLUE, and MAGENTA are defined.

  Furthermore, for each index (for each node), an arrangement position (position) and a recognition color area (the center and diameter of an ellipse on the color plane, and a luminance range) are defined.

  In the index definition information file of FIG. 10, the arrangement position and the recognition color area defined for each index are index information.

  The user can edit the index definition information file using the GUI displayed on the display device 170. The edited index definition information file is stored in the HDD 130 and can be read at any time.

  FIG. 9 is a diagram illustrating a GUI display example.

  Reference numerals 210 and 220 denote menus. When the mouse 190 is clicked, menu lists as shown in FIGS. 3 and 4 are displayed.

  FIG. 3 is a diagram showing a display example of a menu displayed on the display screen of the display device 170 when the “file” menu 210 is clicked. The “Open” menu 212 is an item for instructing a process of opening an index definition information file stored in the HDD 130 and expanding and displaying it on a tree view 280 or the like to be described later. The “save” menu 214 is an item for instructing processing for saving the index definition information being edited in the HDD 130 as an index definition information file. The “end” menu 216 is an item for instructing processing to end the index definition information management program 200.

  FIG. 4 is a diagram illustrating a display example of a menu displayed on the display screen of the display device 170 when the “edit” menu 220 is clicked. The “create new coordinate system” menu 222 is an item for instructing a process for newly creating a child coordinate system of the coordinate system selected in the tree view 280 and having the inverted color. By executing the “Create New Coordinate System” menu 222, a child coordinate system node (coordinate system icon and coordinate system name) for the selected coordinate system is added in the tree view 280.

  For example, when the tree view 280 indicates the “create new coordinate system” menu 222 while the coordinate system world is selected as shown in FIG. 12, a new coordinate system is created as a child coordinate system of the coordinate system world as shown in FIG. A coordinate system “NewCoordSys” is created. Then, a node indicating the created coordinate system “NewCoordSys” is displayed in the tree view 280.

  After that, this child coordinate system is selected in the tree view, and the position and orientation viewed from the parent coordinate system are input.

  Returning to FIG. 4, the “new index creation” menu 224 is an item for instructing a process for creating a new index belonging to the coordinate system selected in the tree view 280 and in the inverted color. When the “Create new index” menu 224 is selected, nodes (index icons and index names) indicating indexes are added to the tree view 280.

  The “delete” menu 228 is an item for instructing a process of deleting a coordinate system or index selected by clicking with the mouse and having an inverted color. The node instructed to be deleted is deleted from the tree view 280. When a coordinate system is deleted, all coordinate systems and indices belonging to that coordinate system are also deleted at the same time.

  Returning to FIG. 9, reference numeral 280 denotes a tree view that graphically displays the coordinate system and index tree structure managed in the index definition information file illustrated in FIG. In the tree view 280 of FIG. 9, defined coordinate systems and indices are displayed hierarchically in a tree structure. More specifically, a coordinate system named world (= world coordinate system) and indices named YELLOW, ORANGE, GREEN, BLUE, and MAGENTA are defined. In addition, indices YELLOW, ORANGE, GREEN, BLUE, and MAGENTA belong under the coordinate system world.

  230 to 238 are elements constituting the index information of the index selected in the tree view 280. The index information corresponding to the selected index is displayed. Then, the user can edit or newly set each element of the index information. A coordinate system icon and an index icon are associated with each coordinate system and each index in the tree view 280, respectively. For example, an index icon 282 indicating that it is an index is displayed side by side in association with the name of the index in the tree view 280 in the index ORANGE. Similar index icons are also displayed side by side with respect to the names of the indexes in the tree view 280 for the other indexes YELLOW, GREEN, BLUE, and MAGENTA.

  The index name 230 is the name of the index, and is identification information used to identify the index. In the figure, the type of color is used as identification information, but an arbitrary name such as MARKER01 or MARKER02 may be used.

  The index position 232 is a three-dimensional position in the coordinate system to which the index belongs.

  The luminance 234, the center 236, and the diameter 238 are parameters that define the recognition color region of the index. In this embodiment, the recognition color area of the index is the YUV444 color space (the luminance Y is 8 bits, the color difference Cb is 8 bits, and the color difference Cr is 8 bits). Determined by the elliptic cylinder. The major axis of this ellipse is constrained so that the extension line always passes through the CbCr plane center (Cb = 127, Cr = 127).

  The luminance 234 indicates the minimum and maximum values of Y, the center 236 indicates the center position of the ellipse, and the diameter 238 indicates the major axis and minor axis of the ellipse.

  When the values of these elements are edited by the user and the “Save” menu 214 is designated, the CPU 110 updates the index definition information file shown in FIG. 10 with the edited information.

  A color space image display area 250 represents a photographed image or a color area of a selected region of the photographed image in a YUV444 color space, and displays a color space image 254 that is projected onto a CbCr plane with luminance Y = 128. . The horizontal axis is the color difference Cb, and the vertical axis is the color difference Cr. The color space image display area 250 is also an area for setting a recognition color area. A part of the recognition color area (Cb value and Cr value range) can be specified in the area surrounded by the ellipse cursor 252. The position, shape and dimensions can be changed by dragging the center of the ellipse cursor 252 and the control point on the circumference with the mouse 190. In addition, when the ellipse cursor 252 is operated, the values of the center 236 and the diameter 238 change, and when the values of the center 236 and the diameter 238 are changed, the position, shape, and size of the ellipse cursor 252 change. Change.

  Reference numeral 260 denotes a live-action image display area, which is an area in which images of frames taken by the imaging device 160 are sequentially displayed. The photographed image display area 260 is also an area for setting a photographed image selection area. The region is selected by dragging the region to be selected on the image displayed in the photographed image display region 260 with the mouse 190.

  Reference numeral 270 denotes an extracted image display area. When the color in the photographed image displayed in the photographed image display area 260 is included in the recognition color area, only the pixel of that color is displayed in the representative color of the recognition color area. Note that the luminance range in the recognition color region is set by the luminance 234. The representative color of the recognition color region is the color (color difference Cb, color difference Cr) of the ellipse cursor 252 indicating the recognition color region when the luminance (Y) is 128.

  Similarly to the live-action image display area 260, the extracted image display area 270 is also an area for setting a live-action image selection area. The frame of the region selected in each of the photographed image display region 260 and the extracted image display region 270 is superimposed on the other in synchronization with the region selection. Further, when an area is selected, the area where the color space image 254 is selected is calculated as an input.

  In this way, the user can generate and update the index information by inputting a desired value in each text box in the GUI. Further, this index information can be included in the index definition information as regular registration information, and thereafter, this index definition information can be stored (registered) in the HDD 130 as an index definition information file.

  Next, registration of an index processed by index calibration will be described. A program for performing the index calibration is also stored in the HDD 130 and is executed by the CPU 110.

  As described in the prior art, the index calibration recognizes an index in a captured image captured by the imaging device 160 and calculates the position (posture) of each index in the space.

  Here, when an unregistered index (hereinafter referred to as an unregistered index) is recognized by the index calibration, it is preferably registered automatically in the index information. On the other hand, it is not preferable that information that is not originally an index, such as an index extracted due to a recognition error in index calibration, is arbitrarily registered in the index information.

  Therefore, in this embodiment, the index definition information management program 200 determines whether the index extracted by the index calibration is an already registered index (hereinafter referred to as a regular registration index) or an unregistered index. Judging. And when it is an unregistered parameter | index, it registers as a temporary registration parameter | index. The temporary registration index is displayed separately from the regular registration index. This index registration state is hereinafter referred to as an index registration state. In the present embodiment, there are “regular registration state” and “temporary registration state” as index registration states.

  5 to 8 are examples in which display methods are distinguished in the GUI tree view 280 according to the index state.

  In any of FIGS. 5 to 8, the two indexes whose index names are RED and YELLOW are temporary registration indexes, and the four indexes whose index names are GREEN, BLUE, CYAN and MAGENTA are regular registration indexes. It is.

  Here, in the tree view 280, the index name included in each index information included in the index definition information is displayed. The temporary registration index automatically registered by the index calibration does not have an index name at first. Therefore, the index name of the temporarily registered index is not displayed until the index name is set by the user. In FIGS. 5 to 8, description will be made assuming that the user has given index names of RED and YELLOW to each of the two temporary registration indices.

  In FIG. 5, a dotted circle icon surrounding the character “temporary” is displayed in the vicinity of the display positions of the index names of RED and YELLOW (left vicinity position in the figure). Also, a dotted circle icon surrounding the character “positive” is displayed in the vicinity of the display positions of the index names of GREEN, BLUE, CYAN, and MAGENTA (left vicinity position in the figure).

  In FIG. 6, index names of RED and YELLOW and index names of GREEN, BLUE, CYAN, and MAGENTA are listed and displayed at different positions in the direction orthogonal to the enumeration direction.

  Further, in FIG. 6, the branch display method is changed (temporary registration index: long dotted line, regular registration index: short solid line).

  Various display methods such as (color, size, font, line attributes) of (icon, index name, branch) as shown in FIGS. 5 and 6 may be changed.

  In FIG. 7, the index names RED and YELLOW are listed together under the heading “provisional registration marker”. In addition, index names of GREEN, BLUE, CYAN, and MAGENTA are listed together under the heading “regular registration marker”.

  In FIG. 8, when each element constituting the index information of the index whose index name is GREEN is displayed in the area 801, the character string “regular registration” indicating that GREEN is the regular registration information is displayed in the area 801. To do. On the other hand, when each element constituting the index information of the index whose index name is RED is displayed in the area 802, a character string “temporary registration” indicating that RED is temporary registration information is displayed in the area 802.

  Furthermore, in the present embodiment, the index state can be changed according to a user instruction in the tree view 280 illustrated in FIGS.

  Thus, since the index information is displayed in the tree view 280 according to the registration state of the index, it is possible to easily grasp whether each index is a regular registration index or a temporary registration index.

  Therefore, when the temporary registration index is an index to be registered, the user changes the index to regular registration, and the temporary registration index is a mark extraction error or the like. Can be done through the GUI.

  As shown in FIGS. 5 to 8, “display in different display forms” means a visually different display, and various forms can be considered for such display.

  FIG. 11 is a flowchart of processing performed by the CPU 110 executing the index definition information management program 200. In the first stage of the processing described below, the index definition information management program 200 is loaded in the memory 120, and a GUI as illustrated in FIG. 12 is displayed on the display screen of the display device 170. Shall.

  In step S1401, the index definition information file instructed using the keyboard 180 or the mouse 190 is loaded (read) from the HDD 130 into the memory 120, or the index definition information file loaded in the memory 120 in advance is referred to.

  In step S1402, a tree is displayed on the tree view 280 according to the read / referenced index definition information file. The display contents are as described above.

  Next, in step S1403, it is checked whether or not an instruction to perform arrangement position / orientation information for each index in the image acquired from the imaging apparatus 160, that is, an instruction to execute a so-called index calibration process is input using the keyboard 180 or the mouse 190. To do. Such an instruction may be an operation instruction for the GUI illustrated in FIG. 12 using the keyboard 180 or the mouse 190, or may be performed without using the GUI.

  If it is detected that such an instruction has been input, the process proceeds to step S1404. In step S <b> 1404, processing for obtaining arrangement position / orientation information is performed for each index included in the image from the imaging device 160. Then, for each index, index information composed only of arrangement position / orientation information is created, the above-described registration information is added to the created index information, and is included in the currently opened index definition information as temporary registration information. Alternatively, the arrangement position / orientation information added with registration information is created for each index as a separate file from the index definition information and registered in the HDD 130.

  However, before this registration, it is checked whether index information including the same arrangement position / orientation information as the arrangement position / orientation information obtained from the image is registered in the HDD 130 first. For example, even if the index information created by the user operation is registered in the HDD 130, the index corresponding to the index information is captured by the imaging device 160, and the index information about the index is obtained from the captured image, the same information is stored in the HDD 130. Because things exist, they are redundant. Therefore, in such a case, the index information obtained from the image is not registered in the HDD 130.

  Note that there is no particular limitation on the check processing for determining whether the arrangement position / orientation information obtained from the image is the same as the arrangement position / orientation information in the index information previously registered in the HDD 130. For example, the coordinate values x, y, and z are compared in each placement position and orientation information. The positional difference between the arrangement position / orientation information (x, y, z) of the index obtained from the image and the arrangement position / orientation information (x, y, z) in the index information previously registered in the HDD 130 is a threshold value. In the following cases, it is assumed that the index information is about the same index.

  On the other hand, only when the arrangement position / orientation information obtained from the image is different from the arrangement position / orientation information in any of the index information previously registered in the HDD 130, the index information composed only of the arrangement position / orientation information. Is added to the index definition information currently open as temporary registration information. That is, the temporary registration state is set as the registration state of the index information.

  Then, the process returns to step S1402, and as shown in FIGS. 5 to 8, the display of the index corresponding to the index information obtained from the image is the display of the index corresponding to the index information input by the user operation as described above. Have different display forms.

  On the other hand, if it is determined in step S1403 that an instruction to execute index calibration processing has not been input using the keyboard 180 or mouse 190, the process advances to step S1405.

  In step S1405, it is determined whether or not there is an operation input to the GUI as shown in FIG. If there is an operation input, the process proceeds to step S1406.

  In step S1406, processing according to the input operation is performed. For example, when the user selects a desired index displayed in the tree view 280 using the keyboard 180 or the mouse 190 in the GUI displayed on the display device 170, the CPU 110 detects that fact. The index information (index name, position, color, shape, etc.) is read out and output to the text boxes 230 to 238, respectively. Further, depending on whether or not the registration information is added to the index information, either “regular registration” or “temporary registration” is displayed.

  In the present embodiment, as described above, information indicating whether the index is provisional registration or regular registration is displayed in the display content of the index. Therefore, for example, by using the keyboard 180 or the mouse 190 to indicate the display portion “temporary registration”, the registration information added to the index information so that the index information as temporary registration information becomes regular registration information. Information may be deleted. The reverse is also conceivable. In this way, temporary registration information can be re-registered as regular registration information, and regular registration information can be re-registered as temporary registration information.

  In the case of FIG. 5, when the icon displayed next to the index name of the index corresponding to the regular registration information is designated using the keyboard 180 or the mouse 190, the registration information is added to the index information of the index, and the index information Is provisional registration information. Conversely, when the icon displayed next to the index name of the index corresponding to the provisional registration information is designated using the keyboard 180 or the mouse 190, the registration information added to the index information of the index is deleted, and The index information is regular registration information.

  In the case of FIG. 7, the index name of the index corresponding to the regular registration information is moved to the display position of the index name of the index corresponding to the temporary registration information in the direction orthogonal to the enumeration direction using the keyboard 180 and the mouse 190. By doing so, the registration information is added to the index information of the index. Thereby, the index information of the index is set as temporary registration information. Conversely, using the keyboard 180 and the mouse 190, the index name of the index corresponding to the temporary registration information is moved to the display position of the index name of the index corresponding to the regular registration information in the direction orthogonal to the enumeration direction. Then, the registration information added to the index information of the index is deleted. Thereby, the index information of the index is set as regular registration information.

  In FIG. 8, the index information in the area 801 is selected by instructing the text box 810 in the area 801 and performing an operation of selecting either regular registration or temporary registration using the keyboard 180 or the mouse 190. (Temporary registration information / regular registration information). The same applies to the region 820.

  As described above, it is possible to input an operation for changing each of the index as temporary registration information and the index as regular registration information to the index as regular registration information and the index as temporary registration information. Such an operation may be repeated any number of times.

  On the other hand, in step S1405, if there is no operation input to the GUI as shown in FIG. 12, the process proceeds to step S1407.

  Unless the user inputs an operation end instruction of the index definition information management program 200 using the keyboard 180 or the mouse 190 (unless the “end” menu 216 is selected with the mouse 190), the process proceeds to step S1402 via step S1407. Return. Then, the subsequent processing is repeated.

  Note that when the icon for another index is clicked with the mouse in the tree view 280, the index information 230 to 238, 250 to 254, and 270 corresponding to the selected icon is updated. Thereafter, a series of processing from step S1402 to step S1407 is performed on the index corresponding to the icon.

  As described above, according to the present embodiment, the display content of the index obtained from the image and the display content of the index corresponding to the index information input by the user operation are visually distinguished and displayed (different from each other). Display in display form). Further, the effect enables the user to intuitively determine the correspondence between the GUI index and the actual index placed in the three-dimensional space.

  In the present embodiment, the captured image is described as being acquired from the imaging device 160, but the present invention is not limited to this. For example, an image stored in the HDD 130 in advance may be used.

  Further, in the present embodiment, information indicating that the index information is temporarily registered (registration information) is added to the index information including only the arrangement position and orientation information of the index extracted from the image. The index definition information is included in the HDD 130 and registered. However, the method of registering the index information as temporary registration information is not limited to the registration after adding the registration information, and various methods are conceivable.

  For example, each index information includes an attribute bit indicating whether it is regular registration information or temporary registration information. When registering the index information, the value of the attribute bit is set according to whether the index information is regular registration information or temporary registration information, and then the index information is included in the index definition information.

[Second Embodiment]
In the present embodiment, a case will be described in which the index arranged in the real space is a polygonal shape such as a triangle. In the following, a triangle is used as an example of a polygon, but the following description is substantially the same even when an n-gon (n> 3) is used. Needless to say, not only circular and polygonal shapes but also ellipses and barcodes (one-dimensional and two-dimensional) may be used as the shape of the index.

  In the present embodiment, the menu shown in FIG. 14 is used instead of the menu shown in FIG. 4 in order to use the triangular index. FIG. 14 is a diagram showing a display example of the menu according to the present embodiment displayed on the display screen of the display device 170 when the “edit” menu 220 is clicked.

  In the menu shown in FIG. 14, a “new indicator creation (triangle)” menu 229 is provided in addition to the items provided in the menu of FIG. When the user selects this “new indicator creation (triangle)” menu 229 using the keyboard 180 or the mouse 190, a triangle indicator is added to the indicator information definition file and the tree view 280. The “new indicator creation (circular)” menu 224 is an item having the same function as the “new indicator creation” menu 224 in the menu of FIG.

  FIGS. 15 and 16 are diagrams showing examples of an index information definition file and a tree view 280 when a triangular index is added.

  As shown in FIG. 15, when a new triangular index is created, definition information defined as “TriangleMarker” in the index information definition file is created.

  Also, as shown in FIG. 16, when the index TRIANGLE is created, an index icon 283 is added next to the index name “TRIANGLE”. The index icon 283 has a triangular shape, and is different in shape from other circular index icons. In this way, the shape of the icon of each index in the tree view 280 can correspond to the shape of the actual index. That is, the user can easily determine the correspondence between each index in the tree view 280 and the actual index.

  Other configurations are the same as those in the first embodiment.

  Here, even with the same triangular shape index, it is conceivable to handle triangular shape indexes having different colors. In that case, as in the first embodiment, it goes without saying that the color of each index may be managed by the index information definition file, and an icon reflecting the color of each index may be displayed.

  As described above, according to the present embodiment, the GUI for managing the index information is displayed separately according to the registration state of the index information. Thereby, the display content can be visually distinguished and displayed (displayed in different display forms) between the index obtained from the image and the index corresponding to the index information input by the user operation.

It is a block diagram which shows the hardware constitutions of the system which concerns on the 1st Embodiment of this invention. It is a figure which shows the example of a display of the conventional GUI which a user operates. FIG. 10 is a diagram showing a display example of a menu displayed on the display screen of the display device 170 by clicking the “File” menu 210. 12 is a diagram showing a display example of a menu displayed on the display screen of the display device 170 by clicking the “edit” menu 220. FIG. In tree view 280, it is a figure which shows a mode that the display content about the parameter | index corresponding to the parameter | index information produced by user operation and the display content about the parameter | index extracted from the captured image are displayed on a different display form. In tree view 280, it is a figure which shows a mode that the display content about the parameter | index corresponding to the parameter | index information produced by user operation and the display content about the parameter | index extracted from the captured image are displayed on a different display form. In tree view 280, it is a figure which shows a mode that the display content about the parameter | index corresponding to the parameter | index information produced by user operation and the display content about the parameter | index extracted from the captured image are displayed on a different display form. In tree view 280, it is a figure which shows a mode that the display content about the parameter | index corresponding to the parameter | index information produced by user operation and the display content about the parameter | index extracted from the captured image are displayed on a different display form. It is a figure which shows the example of a display of GUI. It is a figure which shows the structural example of an index definition information file. The CPU 110 is a flowchart of processing performed by executing the index definition information management program 200. It is a figure explaining addition of the node which shows a coordinate system. It is a figure explaining addition of the node which shows a coordinate system. It is a figure which shows the example of a display of the menu which concerns on the 2nd Embodiment of this invention displayed on the display screen of the display apparatus 170 by clicking the "edit" menu 220. FIG. It is a figure which shows the example of the parameter | index information definition file at the time of adding the triangle-shaped parameter | index. 5 is a diagram illustrating an example of a tree view 280. FIG.

Claims (8)

Index information acquisition means for acquiring index information regarding each of a plurality of indexes,
Index registration status determination means for determining the registration status of the index;
Generating means for generating a user interface for editing the index information according to a user instruction based on the index information and the determination result of the index registration state determination means;
The registration status includes a temporary registration status and a regular registration status,
An information processing apparatus, wherein when the index information is automatically calculated by index calibration and is registered in the index information, the temporary registration state is set as the registration state of the index information. The information automatically calculated by the indicator calibration is the three-dimensional position of the indicator,
The index calibration acquires a captured image of a real space in which an index is arranged, extracts an index from the captured image, and calculates a three-dimensional position of the index based on the image position of the extracted index The information processing apparatus according to claim 1. The user interface includes a first display area that displays a list of the plurality of indices, and a second display area that displays details of index information of an index selected from the plurality of indices.
The information processing apparatus according to claim 1, wherein a display method of the index in the first display area is different according to a determination result of the index registration state determination unit.   4. The information processing apparatus according to claim 3, wherein an indicator display method in the first display area is different in an indicator icon display method according to a determination result of the indicator registration state determination unit. The list display of the plurality of indicators in the first display area is displayed in a tree structure based on the indicator information,
The information processing apparatus according to claim 3, wherein a display method of the branches of the tree structure differs according to a determination result of the index registration state determination unit.   The information processing apparatus according to claim 1, wherein the registration state can be changed according to a user instruction in the user interface.   The program for implement | achieving the information processing apparatus of any one of Claim 1 thru | or 6 using a computer. A first registration step of registering in the memory index information including the position information of the index placed in the real space, which is input by the user, as regular registration information;
An acquisition step of acquiring a captured image of the real space imaged by the imaging device;
An index position calculating step of extracting an index from the captured image and calculating position information of the index;
A second registration step of registering the position information calculated by the index position calculation step in the memory as temporary registration information;
An information processing method comprising: a display step of displaying an index corresponding to the regular registration information and an index corresponding to the temporary registration information in different display forms.

Images