JPH11144024A - Image combining apparatus, image combining method, and medium - Google Patents

Abstract

(57) [Summary] [Problem] A problem that a composite image can be generated only by a combination of a real object and a virtual object stored in advance. A camera for imaging a document on which information is written.
A pointing position detecting unit 2 for detecting a pointing position on a document by a pen or the like, a pointing information detecting unit 3 for detecting a characteristic amount of pointing information written at the pointing position, and a characteristic amount of the pointing information. Additional image generation unit 4 for generating an image to decorate the instruction information, additional image generation unit 5 for storing the generated additional image, and information position-specific information addition method for determining the information addition method according to the indicated position An appropriate additional image is generated in accordance with the position indicated by the pen or the like, the indicated area, and the characteristic amount of the indicated information, by the image forming apparatus including the determining unit 6 and the image synthesizing unit 7 for synthesizing the image of the camera 1 and the additional image. Then, it is synthesized and displayed on the display unit 8.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image synthesizing apparatus, an image synthesizing method, and a medium which can be used for an apparatus for creating a synthetic image of a real object and a virtual object in a computer.

[0002]

2. Description of the Related Art Conventionally, a real object photographed by a camera is
An information processing method and apparatus have been proposed as a system that identifies information by a color code added to the real object and superimposes information related to the real object on a camera image in a superimposed manner.

For example, an information providing system can be realized in which such a device is mounted on a monitor-integrated video camera and a picture to which a color code is added is displayed on the monitor.

As a system for performing interaction between a virtual object and a real object, Proceedings of UIST '91,
ACM, pp.27-33 and Communications of the ACM, vol
36, no. 7, pp. 86-96, 1993, "Digital Desk" is known.

In this system, a camera and a projector are provided above a desk. Then, for example, from the information written on the paper (real object) on the desk, the operator wants to extract as an image. For example, when a range of a numerical value sequence or a picture is designated by a finger or a pen, the numerical value is obtained. Images such as columns are automatically extracted. On the other hand, as an image of another picture (virtual object) stored in advance by the system, for example, an image of a small computer on a desk is projected on another paper (real object) on a desk by a projector. Then, the image of the small computer on the desk and the image of the extracted numerical value sequence are synthesized and projected on the same paper. This projected image is
It is a composite image that represents a state as if the numerical sequence was input by operating the keys of the small computer on the desk being projected, and the numerical sequence was displayed. in this way,
This system can be used as an image combining device for combining an image of a real object and an image of a virtual object.

[0006] Proceedings of CHI'92, ACM, pp.5
87-592 discloses that a tablet is mounted on a part of the desk in order to give a more accurate instruction than a finger.

[0007]

However, in the above-described conventional information processing method and apparatus, it is disclosed that a real object is identified using a color code and an image of information related thereto is displayed in a superimposed manner. However, it is not possible to automatically extract an image of desired information from a position specified by an operator with respect to a real object such as an actual document, process the image with a virtual object image, and display the processed image. There was a problem that.

That is, in the above-described conventional system, an image of information desired by an operator cannot be automatically extracted from a real object, so that a composite image is generated only by a combination of a real object and a virtual object stored in advance. There was a problem that could not be done.

[0009] On the other hand, in a conventional system, for a target object to be extracted as an image, for example, by inputting a finger or a pen, the image of a plurality of objects is independently obtained unless a rough area is specified by the number of the objects and information. Could not be extracted. For this reason, when the number of objects and the number of images to be processed are large, there is a problem that the process of designating the objects to be extracted as an image places a heavy burden on the operator of the apparatus.

The present invention takes into account the problems of the above-described conventional apparatus, and is an image synthesizing apparatus for combining a real object and a virtual object, which can be applied to, for example, a presentation using an overhead camera. If a part of the information to be emphasized is pointed out by a pointing stick or the like, the operator can automatically identify the area of the information to be emphasized and whether the information to be emphasized is text or a photograph. It is an object of the present invention to provide an image synthesizing apparatus, an image synthesizing method, and a medium which can automatically detect and automatically perform decoration processing such as underlining or shading according to the characteristics of the information.

Further, the present invention takes into consideration such problems of the above-mentioned conventional system. For example, by placing a plurality of objects to be extracted on a display and giving only one closed curve including all of them, It is an object of the present invention to provide an image synthesizing apparatus, an image synthesizing method, and a medium having a contour extracting function capable of independently extracting an image of an object.

[0012]

In order to achieve the above object, according to the present invention, there is provided the present invention, wherein a predetermined image pickup means and a position of instruction information given on a surface on which information is displayed are stored in the image pickup means. Pointing position detecting means for detecting from the image captured by the pointing device, and the area of the pointing target information and / or the pointing target information displayed on the surface corresponding to the pointing position detected by the pointing position detecting means. Pointing target information detecting means for detecting a characteristic amount of information; additional image generating means for generating an additional image to be added to the image of the pointing target information based on the area and / or the characteristic amount; An image synthesizing apparatus comprising an image synthesizing means for synthesizing the image of the instruction target information and the generated additional image.

According to a fifteenth aspect of the present invention, there is provided a photographing apparatus, comprising: a predetermined image pickup means; and information and / or a plurality of objects which are picked up by the image pickup means and which are displayed on a predetermined surface. Closed curve information detecting means for detecting closed curve information surrounding a plurality of extraction target candidates to be candidates; and the plurality of extraction targets enclosed based on the detected closed curve information and information on the image of the extraction target candidate. An image synthesizing apparatus comprising: an individual extracting unit that individually extracts an image of each extraction target candidate from a candidate image; and an image synthesizing unit that synthesizes an image using the individually extracted image. is there.

The present invention as defined in claim 16, wherein the closed curve information for detecting closed curve information surrounding a plurality of pieces of pointing target information to be pointed out of the information captured by the predetermined imaging means and the imaging means. Detecting means, based on the detected closed curve information and the image information of the pointing target information, from the image of the surrounded plurality of pointing target information,
An individual extracting unit for individually extracting an image of each piece of instruction target information; and an area and / or an area of the individually extracted image.
Or an instruction information detecting unit for detecting a characteristic amount of the image, and an additional image for generating an additional image to be added to the image of the instruction target information based on the detected area of the image and / or the characteristic amount of the image. An image synthesizing apparatus comprising: a generating unit; and an image synthesizing unit that synthesizes the image of the instruction target information obtained by the imaging unit and the generated additional image.

Further, according to the present invention, an input display integrated type display, an image pickup means for picking up an image of at least a part of a display section of the display, and a plurality of real objects placed on the display are provided. Among them, closed curve information detection means for detecting closed curve information surrounding a plurality of objects to be candidates for image extraction, the detected closed curve information, and the plurality of enclosed objects imaged by the imaging means Based on the information of the image, from the images of the plurality of enclosed objects, individual extraction means for individually extracting an image of each object, and using the individually extracted images, An additional image generation unit that generates an additional image to be displayed, and a display method selection unit that selects a display method of the generated additional image on a display.
An image synthesizing apparatus comprising: an image synthesizing unit that synthesizes the additional image and the image of the real object based on the selected display method.

According to the present invention, there is provided an input / output integrated display, a camera for capturing at least a part of a display section of the display, and an image of a plurality of real objects placed on the display. Closed curve information detecting means for detecting closed curve information surrounding a plurality of objects to be candidates for extraction, information on the detected closed curve information, and information on images of the plurality of enclosed objects imaged by the imaging means Individual extracting means for individually extracting an image of each object from the plurality of images of the enclosed object based on An additional image generation unit that generates an image, a display method selection unit that selects a method of displaying the generated additional image on a display, and the selected display method And the additional image displayed on said display based, it is a body of reality placed on the display an image synthesis apparatus and a control unit for imaging in the camera.

According to a nineteenth aspect of the present invention, an input / display integrated display, imaging means for imaging at least a part of a display section of the display, and a real object placed on the display are indicated. A pointing position detecting unit for detecting pointing position information for, based on the detected pointing position information, a real object extracting unit that extracts an image of the real object from an image captured by the imaging unit, An additional image generation unit that generates an additional image to be displayed on the display using the extracted image; a display method selection unit that selects a display method of the generated additional image on a display; An image synthesizing apparatus comprising: an image synthesizing unit that synthesizes the additional image and the image of the real object based on a display method.

According to a twenty-fifth aspect of the present invention, a predetermined position of a surface on which information is indicated is indicated by instruction information, and the information represented on the surface and the instruction information are imaged. Detecting a pointing position indicated by the pointing information from an imaging result; detecting a region and / or a feature amount of the pointing target information displayed on the surface corresponding to the pointing position; And generating an additional image to be added to the image of the instruction target information based on the region and / or the feature amount of the instruction target information, and synthesizing the image of the instruction target information and the generated additional image.

According to a twenty-seventh aspect of the present invention, an image and / or a plurality of objects represented on a predetermined surface are imaged, and an image is extracted from the imaged information and / or the plurality of objects. Providing a closed curve surrounding a plurality of extraction target candidates to be candidates for the target, detecting the given closed curve information, and performing the enclosing based on the detected closed curve information and information on the image of the extraction target candidate. An image synthesis method for individually extracting images of the respective extraction target candidates from the plurality of extracted extraction target candidate images, and performing image synthesis using the individually extracted images.

[0020]

Embodiments of the present invention will be described below with reference to the drawings. (Embodiment 1) FIG. 1 is a block diagram showing a first embodiment of the image synthesizing apparatus of the present invention.

In FIG. 1, reference numeral 1 denotes a camera for capturing a document on which information is written, preferably a color CCD camera;
Reference numeral denotes an indicated position detection unit that detects an indicated position on a document by the pen 100 using the camera 1. Reference numeral 3 denotes an instruction information detection unit that detects the area and characteristic amount of the instruction information described in the instruction position, and reference numeral 4 denotes an additional image generation unit that generates an additional image according to the characteristic amount of the detected instruction information. Reference numeral 5 denotes an additional image storage unit that stores the additional image generated by the additional image generation unit 5, and 6 denotes an additional method determination unit that determines the method of combining and displaying the additional image based on the detected specified position. is there. Reference numeral 7 denotes an image synthesizing unit for synthesizing the video of the camera 1 and the additional image, and reference numeral 8 denotes a display unit for displaying the image synthesized by the image synthesizing unit 7, preferably a projector or a display. Further, the pen 100 preferably has a marker 100a having a predetermined RGB value and a designated position detection start switch 100b for facilitating detection of the designated position. Reference numeral 1000 denotes a control unit which generally includes a memory such as a ROM or a RAM and a CPU, and is connected to each of the units 1 to 8, and controls the operation of each unit and the exchange of data between the units. Here, the pen 100, the marker 100a, and the like correspond to the pointing means of the present invention, and the pointing information detecting unit 3 corresponds to the pointing target information detecting means of the present invention. Next, an embodiment of the image synthesizing method of the present invention will be described at the same time, while describing the operation of the present embodiment.

FIG. 2 is a flowchart showing the operation of the image synthesizing apparatus according to the present embodiment.

In order to explain the operation of the present embodiment more easily, an example in which the present invention is mounted on an overhead camera will be described here. Overhead camera
It is usually used for lectures and presentations at universities and the like. At this time, as shown in FIG. 3, characters are written on a paper or a document, or a photograph or a real three-dimensional object is presented. This corresponds to a case where the image captured by the camera 1 is displayed on the display unit 8 such as a monitor as it is.

As described above, the present embodiment can be used, for example, to support a lecture or the like. In this document, this is also simply referred to as instruction information), various additional images (virtual objects) are generated in accordance with the regions and characteristics, and the additional images are combined with the image of the camera to decorate the document with decorations. It is something to give.

In the present embodiment, in the most common use of writing black characters on white paper, by pointing a part of the entry portion to be emphasized with the pen 100, the area of the entry portion is detected. Further, the characteristic amount of the information of the entry portion is detected, the characteristic of the information of the entry portion whether the entry portion is a character or a photograph is determined, and a decoration suitable for the feature is determined. The operation procedure of the present embodiment will be described below with reference to an example in which the image is applied to a partial image and the control of the additional image according to the position indicated by the pen 100. [STEP 1a] Image capture: The camera 1 captures a document and a pointing stick 100 as shown in FIG. 3 as an image I (x, y). [STEP 2a] Detecting the state of the designated position detection start switch: If the designated position detection start switch 100b is OFF, an image captured in STEP 8a described later is displayed on the display unit 8. If the switch 100b is ON, the process proceeds to STEP3a. [STEP 3a] Detection of a designated position by the pen marker 100a: The designated position detection unit 2 uses the designated position detecting unit 2 to determine a predetermined color (RGB value in a predetermined range of the marker 100a) from the image I (x, y). The area is detected as the designated position.

For example, for each pixel I (x, y), an RGB value within a predetermined range representing the color of the marker (for example, when the marker 100a is reddish, 180 <R <255 and 120 <G <140 And 60
<B <120), and if it ’s the color of the marker,
For an area of a predetermined size centered on the pixel, pixels having RGB values in a predetermined range representing the color of the marker are extracted. A circumscribed rectangle that includes all the extracted pixels is obtained, and if the rectangle is a predetermined size, it is determined that the rectangle is a marker, and the designated position is the marker center (mx, my).
If no marker has been detected, proceed to STEP7a. [STEP4a] Determination of information addition method based on designated position: Here, it is determined whether or not the image is near the edge of the image. If the designated position is at the edge of the image (document), for example, the image size is vertical x horizontal : Marker center (mx, my) when 480 x 640
When satisfies mx <30, my <30, 610 <mx, 450 <my, a frame image is selected in STEP4a_1. [STEP4a_1] Selection of a frame image: The additional image storage unit 5 stores at least one frame image including the affiliation of the person giving the presentation and the date and time of the lecture. As shown in FIG.
The frame image is superimposed on the image of the camera 1 and displayed on the display unit 8. When a plurality of frame images are stored, a different image is selected each time the edge of the document is specified. In this way, the affiliation, date and time, and the frame to be given when the normal OHP sheet is created can be given on the spot to a sheet of paper on which normal nothing is written during presentation. [STEP 5a] Determination of information addition method based on designated position: Here, it is determined whether or not undecorated information has been designated, and if so, the presence / absence of a black pixel near the marker center is determined by the designated information detecting unit 3. Find out. When there is a black pixel, it is assumed that information such as characters is described, and the feature amount is calculated in STEP5a_1, and a decorative image corresponding to the feature amount is generated and stored in STEP5a_2. [STEP 5a_1] Calculation of the area and the characteristic amount of the instruction target information: First, the area of the instruction target information is obtained by the instruction information detecting unit 3. Label black pixels within a predetermined range from the marker center (mx, my). Labeling is performed recursively for each labeled black pixel. When there are no more black pixels to be labeled, labeling ends.

As shown in FIG. 5, for example, when a character string is expressed in parallel to the x-axis, the minimum and maximum values of the x- and y-coordinates of the labeled pixels are xl, yl, xr, If it is yr, the rectangle (xl, yl)-(xr, yr) is the area of the instruction target information.

Next, as a feature amount for identifying whether a character string or a photograph, for example, a ratio of pixels having white (RGB values in a predetermined range) among all pixels in the designated area is calculated. I do. When the designated area is a photograph, there are relatively few white pixels. On the other hand, in the case of a character string, black pixels and white pixels appear relatively alternately, so that the ratio of white pixels is larger than that in a photograph area. Therefore, if the ratio of white pixels is small according to the predetermined threshold value, it can be determined that the region is a photograph region, and if the ratio is large, it can be determined that the region is a character string region. [STEP5a_2] Generation and storage of decorative image: additional image generation unit 4
Thus, an additional image for decoration is generated based on the region calculated in STEP5a_1 and its feature amount.

For example, if a photograph area is emphasized by shading, it becomes difficult to see. Therefore, it is bordered and a character string is emphasized by shading. The hatched image indicates that, for the designated area detected in STEP 5a_1, the RGB values of the pixels other than black are changed to RGB of a predetermined color.
It can be generated by replacing it with a value. FIG. 6 shows the decoration result when the ratio of white pixels in the designated area is 15% or less as a photograph, and when the ratio is more than 15% as a character string. The decorated area is stored as an existing decoration area. [STEP6a] Determining information addition method by pointing position: Pen 1
It is determined whether the position designated by 00 is within the existing decoration area stored in STEP 5a_2, and if it is determined that the existing decoration information is indicated, STEP 7a displays the decoration in accordance with the position. To change. Otherwise, proceed to STEP1a. [STEP 7a] Changing the display method of the existing decoration area: The information addition method determination unit 6 for each designated position divides the existing decoration area 700 into a plurality of areas as shown in FIG. 7 (the front part 701 in FIG. 7). And two parts of the rear part 702), for example, if the front part 701 is designated, the decoration is erased and the rear part 702 is deleted.
02 is indicated, the display method is changed such that the decoration is blinked, and the display unit 8 is synthesized with the image of the camera 1.
To be displayed.

As described above, if the marker 100a is not detected in STEP 3a, the process proceeds to STEP 7a. In this case, the display of all additional images is stopped.
Thus, by pressing the switch 100b of the pen 100 outside the field of view of the camera before starting writing characters on a different paper, all the additional images can be erased at once. [STEP 8a] Image composition / display: The image composition unit 7 combines the image of the camera 1 and the decoration image generated in the additional image storage unit 5 by, for example, superimposing and displays the image on the display unit 8. I do.

According to the first embodiment, according to the procedure described in [STEP1a] to [STEP8a], the presenter specifies the part of the information that he / she wrote by using the wand. Just then, the features of that part are automatically extracted and the appropriate decorative image is superimposed.

Therefore, there is no need to switch between pens of various colors and use them properly. In addition, the decoration method can be easily changed depending on the position indicated by a pen or the like. For example, if the user touches the edge of the document, a frame, affiliation, date and time, etc. are automatically added, and the information part already decorated is Can be turned off or blinked depending on the designated position, so that an effective presentation can be made on the spot without preparing presentation materials in advance.

In the above procedure, the additional image is generated in accordance with the region of the pointing object and the characteristic amount at the position indicated by the pen or the like. May be superimposed and displayed by superimposition or the like. This can be done, for example, by placing a textbook on an overhead camera in an English lesson or the like to indicate the position of accents of English words, or in a video conference or personal computer conference, by using real objects such as materials at hand or models. This is effective when the user wants to pay attention to a part or the part that is just described.

In the above procedure, the image I (x, y)
Has been described in the example of being always digitized, but may be digitized only when the pointing position detection switch 100b is turned on. (Embodiment 2) Next, a second embodiment of the image synthesizing apparatus according to the present invention will be described.
An embodiment will be described.

In the first embodiment, there is only one type of decoration method for a character string and a photograph. However, in the second embodiment, consideration is given so that a plurality of types of decoration methods can be used more easily. It is.

FIG. 8 is a block diagram of the second embodiment.
The same components as those in the first embodiment shown in FIG. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted.

The main difference between the second embodiment and the first embodiment is that the designated position detecting unit 2 is a marker type / designated position determining unit 9 for identifying the type of a plurality of markers attached to a pen or the like. Thus, the additional image generation unit 4 is replaced with a different marker-compatible additional image generation unit 10 that generates a different additional image according to the type of marker. Here, the marker type / pointed position determination unit 9 corresponds to the pointed position detection unit of the present invention, and the heterogeneous marker corresponding additional image generation unit 10
Corresponds to the additional image generation means of the present invention.

The operation of the image synthesizing apparatus according to the second embodiment configured as described above will be described with reference to the flowchart of FIG. However, since a new procedure is added to the flowchart of FIG. 2 representing the operation of the first embodiment, the differences [STEP 3b], [STEP 5b], [STEP 3]
7b_2] alone will be described.

In the present embodiment, as shown in FIG. 10, four types of markers 100a-1 and 100a-4 are provided on the tip of the pointing rod, and 100a-1 to 100a-
4, an example in which a plurality of decoration methods can be easily used by identifying which marker indicates a document from an image captured by the camera 1 will be described. [STEP 3b] Determination of marker color: The marker type / pointed position determination unit 9 determines the marker color assigned to the pen 100 and facing the camera 1.

More specifically, in the image I (x, y), the number of pixels is counted for each predetermined color (in this embodiment, the red and blue colors of the marker shown in FIG. 10), and The color is recognized as the pen marker color facing the camera 1. For example, here, the number of red pixels and the number of blue pixels are checked, and the numbers of these pixels are compared. Thereby, even if the pen 100 is slightly inclined and two types of colors are detected at the same time, the color determined to have a larger number of pixels is determined as the color of the marker used for the instruction. More flexible and error-free judgment can be made. [STEP5b]
Determination of marker type: The marker type / pointed position determination unit 9 uses the shading markers (100a-1, 100a).
-2) or underline marker (100a-3, 1)
00a-4).

More specifically, the pen 10 detected in STEP 4b
The number of pixels of a predetermined color included in the marker area on 0 is counted. The shaded marker has a smaller number of pixels of a predetermined color than the underline marker, so that it is possible to identify which marker is a predetermined threshold.
Note that the number of pixels of a predetermined color included in the marker area is determined in STEP.
It is the number of pixels of a predetermined color identified in 3b. [STEP7b_2] Generation of decorative image based on marker type: The different-marker-corresponding additional image generation unit 10 generates a decorative image based on the marker type identified in STEP5b and the feature amount obtained in STEP7b_1. For example, FIG.
Shows an example of a decoration image of red shading (marker 100a-1 in FIG. 11) and blue underline (marker 100a-4 in FIG. 11) when the instruction target information is a character string. ing. If the instruction target information is a photograph, only the color of the border is changed.

As described above, according to the third embodiment, the pen 1
Since the decoration method can be switched by identifying a plurality of types of markers assigned to 00 or the like, it is possible to save the trouble of holding and writing with a plurality of types of color pens, and to facilitate the presentation. (Embodiment 3) Next, a third embodiment of the image synthesizing apparatus according to the present invention will be described.
An embodiment will be described.

The third embodiment is different from the first and second embodiments in that the existing decoration image generated so far can be reused by copying or the like.

FIG. 12 is a block diagram of the third embodiment. The same components as those of the first embodiment shown in FIG. 1 are denoted by the same reference numerals, and detailed description is omitted.

The main difference between the third embodiment and the first embodiment is that a list of existing decorative images stored in the additional image storage unit 5 is combined with the video of the camera 1. Existing additional image copying unit 1 for displaying the selected image with a pen 100 or the like, and combining and displaying the selected image at another designated position on the document.
That is, 1 is added.

The operation of the image synthesizing apparatus according to the third embodiment configured as described above will be described with reference to the flowchart of FIG.

As shown in FIG. 13, the operation procedure of the third embodiment is similar to the operation procedure [STEP3a] and [S3] of the first embodiment.
Since [STEP1c] to [STEP3c] are inserted between [TEP4a], only that portion will be described. [STEP1c] Display of existing decoration image list: First, if at least one decoration image has been generated by the additional image generation unit 4 so far, as shown in the upper left of FIG. An image with the heading “List” is displayed in superimposition on the video of the camera 1.

Next, if the designated position detected in the same manner as in STEP 3a of the first embodiment is within this area, for example, as shown in FIG. The image of the camera 1 is superimposed on the image of the camera 1 and the image of the “extraction of a real object” and the underlined image of “4. [STEP2c] Selection of existing decoration image: As shown in FIG.
The existing decoration images listed in STEP 1c are detected by the pen 100 in the same manner as in STEP 3a of the first embodiment, and the existing image at the specified position is selected. [STEP3c] Copy of existing decoration image: As shown in FIG.
The image selected in STEP 2c is superimposed and displayed at the position indicated by the pen 100 detected in the same manner as in STEP 3a of the first embodiment.

According to the third embodiment, according to the above-described procedure, a list of existing decorative images can be displayed in a superimposed manner, and these can be copied by designating them with a pen or the like. However, it is not necessary to write the same character string many times on documents, and the presentation can be further smoothly performed. (Embodiment 4) Next, the fourth embodiment according to the image synthesizing apparatus of the present invention will be described.
An embodiment will be described.

In the first to third embodiments, the description has been made assuming that the document is fixed to the camera. However, in the fourth embodiment, when writing characters on the document, the document is replaced with the original. Even in a case where the decoration image is shifted from the position, the existing decoration image is considered so as to be superimposed at a correct position in accordance with the movement of the document.

FIG. 17 is a configuration diagram of the fourth embodiment. The same components as those of the first embodiment shown in FIG. 1 are denoted by the same reference numerals, and detailed description is omitted.

The main difference between the fourth embodiment and the first embodiment is that a moving parameter estimating unit 12 for estimating a moving position of a document and an existing decoration image are also estimated based on the estimated moving parameters. That is, an additional image deforming unit 13 to be deformed is added. Here, the fourth configured as described above
The operation of the image synthesizing apparatus according to this embodiment will be described with reference to the flowchart in FIG. The operation of the fourth embodiment is as follows.
The following STEP1d to STEP4d are inserted between STEP1a and STEP2a in the operation of the first embodiment. [STEP1d] Presence or absence of existing decoration area: The movement parameter estimation unit 12 refers to the additional image generation unit 5, and proceeds to STEP2d if there is an existing decoration image, and executes STEP2a if there is no existing decoration image. I do. [STEP2d] Selection of partial image from existing decorative image: From an arbitrary existing decorative image, a small area as shown in FIG. 19 is selected as a template to perform template matching between two temporally consecutive images. .

The two temporally continuous images are images before and after the movement of the document, and are images taken by the camera 1. [STEP3d] Matching of partial images: template matching is performed between two temporally consecutive images to obtain a movement parameter. In this case, the movement of documents is mainly parallel movement
Since (xd, yd) and the rotational movement θ are included, for example, the template is set to 0 <θ <360 °, (xd, yd) ∈R (predetermined search range R:
(See FIG. 19), and θ, (xd, yd) at which the sum of squares of the difference between the luminance values is minimized is set as a movement parameter. [STEP 4d] Deformation of decorative image: All images stored in the additional image generation unit 5 are deformed by the movement parameters obtained in STEP 3d.

As described above, according to the fourth embodiment, the decoration image is deformed by obtaining the movement parameter of the document by performing template matching between consecutive frames using a part of the decoration image. Even if the document is moved, the part to be emphasized can be correctly decorated. (Embodiment 5) Next, a fifth embodiment according to the image synthesizing apparatus of the present invention will be described.
An embodiment will be described.

In the first embodiment, a part to be emphasized one by one with a pen or the like is specified, and the decoration is erased.
In that case, the part was pointed in the same way as in the case of decoration. The fifth embodiment is designed so that emphasis and erasure of a plurality of locations can be performed more easily and intuitively.

FIG. 20 is a configuration diagram of the fifth embodiment. The same components as those of the first embodiment shown in FIG. 1 are denoted by the same reference numerals, and detailed description is omitted.

The main difference between the fifth embodiment and the first embodiment is that, in addition to the pen type indicating means such as the pen 100,
A point using the installation instruction means 200 for indicating a part to be decorated by placing it on a document, and a pointing position detection unit 2
The point that it is replaced by a different type plural pointing means pointing position detecting unit 14 for detecting a plurality of different pointing means, and that the pointing information detecting unit 3
To the area or feature amount of the information specified by the specifying means,
The point that it is replaced by the instruction information detecting unit 15 corresponding to the different type plural instruction means that detects according to the type of the instruction means, and further, the additional image generation unit 4 is replaced by the different type plural instruction that generates a different additional image according to the type of the instruction means. This is the point of replacement in the means corresponding additional image generation unit 16. Here, the indicating means of the present invention is a means including a pen-type indicating means such as the pen 100 and an installation indicating means 200. The pen-type indicating means such as the pen 100 and the installation instruction means 200 correspond to the indicating member of the present invention. In addition, the different type plural pointing means pointing position detecting unit 14 corresponds to the pointing position detecting means of the present invention, and the different type plural pointing means corresponding indicating information detecting unit 15 corresponds to the pointing target information detecting means of the present invention. Further, the additional image generation unit 16 corresponding to different types of plural instruction means corresponds to the additional image generation means of the present invention.

The operation of the image synthesizing apparatus according to the fifth embodiment configured as described above will be described with reference to the flowchart of FIG. Here, only the steps STEP2e, STEP2e_1, STEP2e_2, and STEP8e which are different from the operation of the first embodiment will be described.
Here, as the installation instruction means 200, the first, second,
An example using a yellow marker different from the marker of the pen 100 described in the embodiment will be described. As shown in FIG. 22, here, as shown in the second embodiment, a shading marker 200 is used as the installation instruction means.
An example will be described in which two types of markers, ie, a and an underline marker 200b are used. Here, these two types of markers 200a and 200b are formed by printing different patterns as shown in FIG. 22 on the surface of a round shape such as a coin or a button formed of, for example, glass or plastic material. This can be easily realized.
Further, separately from this, the back surface is an adhesive surface, and a seal member having a different pattern printed on the front surface as shown in FIG. 22 can be easily realized. Further, the seal member may be attached to a plain surface of the button-shaped member. [STEP2e] Detection of the installation instruction means: The different installation instruction means indicated position detection unit 14 performs the predetermined installation from the image I (x, y) captured in STEP1a of the first embodiment in the same manner as in STEP3a. Color of the marker for use (RGB values in a predetermined range, here, for example, yellow 180 <R <255 and 180 <G <255 and 60 <B <12
An area of a predetermined size having 0) is detected as an installation marker. However, since a plurality of installation markers are usually installed, all areas of a predetermined size having a predetermined color are extracted.

The position of the installation marker is determined by the set of vertices (xl (i), xl (i), yr (i)) at the upper left (xl (i), yl (i)) and lower right (xr (i), yr (i)) of the circumscribed rectangle. yl (i))-(xr (i), yr (i)). Here, i represents the number of markers. Further, two types of markers 200a and 200b for shading and for underlining are identified based on the number of pixels of a predetermined color included in the area. [STEP2e_1] Calculation of designated area / feature quantity based on installation instruction means: As shown in FIG. 22, the rectangular area (xr (i), yl (i))-(image width, yr (i))
A black pixel having the largest x coordinate is searched for, and the x coordinate is set to x_max (i). At this time, the search is performed in the x-axis direction for each y coordinate, but if no black color is detected for a predetermined length, the search is stopped. This prevents erroneous extraction of information even if there is information after the stain on the document or the part to be emphasized. Here, the rectangle (xr (i), yl (i))-(x
_max (i), yr (i)) is the designated area.

Next, as in STEP 5a_1 of the first embodiment, whether the designated area is a character or a photograph is identified. [STEP2e_2] Generation of additional image for installation instruction means: Similar to STEP5a_2 of the first embodiment, based on the region / feature calculated in STEP1e_1, an additional image used for shading or underlining is generated. Generate. [STEP8e]: The image synthesizing unit 7 determines the image I (x, y) in STEP1e.
Is acquired at predetermined time intervals, the display of the image on the display unit 8 is updated. At this time, the additional image is re-synthesized according to the position of the installation marker. Thus, if the installation marker is removed as shown in FIG. 23, the additional image is not re-synthesized, and the decoration disappears.

As described above, according to the fifth embodiment, an installation marker is detected at a predetermined time interval, so that an intuitive and easy-to-understand operation method can be provided in which the marker is placed and the decoration is removed, and the marker is removed and removed. (Embodiment 6) Next, a sixth embodiment according to the image synthesizing apparatus of the present invention will be described.
An embodiment will be described.

In the fourth embodiment, the existing pointing area by the pen-shaped pointing means is displaced when writing characters on the document and moves from the original position. Explained that they were superimposed in the correct position.

On the other hand, in the sixth embodiment, the case where the document moves when the installation instruction means used in the fifth embodiment is used is considered.

FIG. 24 is a block diagram of the sixth embodiment. The same components as those of the fourth embodiment shown in FIG. 17 and the fifth embodiment shown in FIG. The detailed description is omitted.

The main difference between the sixth embodiment and the fourth and fifth embodiments is that the movement parameter estimating unit 12 determines the rotation movement, that is, the inclination angle, of the document movement parameters. Is replaced by the installation marker use movement parameter estimating unit 17 which is calculated from the inclination of the indicated portion. Thereby, if the instruction information by the pen-type instruction means and the instruction information by the installation marker are written in parallel, the partial image for template matching is regarded as the inclination of the instruction information by the installation marker as the inclination of the document. It can be rotated and used to calculate the parameters of the translation of the document. Here, the installation marker-based movement parameter estimation unit 17 corresponds to the movement parameter calculation unit of the present invention.

The operation of the image synthesizing apparatus according to the sixth embodiment configured as described above will be described with reference to the flowchart of FIG. The operation according to the sixth embodiment is performed between STEP1e and STEP3e of the operation according to the fifth embodiment.
EP6f is inserted. [STEP1f] Calculation of Instruction Information Inclination Angle Using Installation Marker: FIG.
As shown in FIG. 6, first, the value of each y coordinate in the rectangular area (xr (i), yl (i))-(image width, yr (i)) adjacent to the position where the marker is detected A black pixel having the maximum x coordinate is searched for every y (k), and the x coordinate is set to x_max (k). At this time,
In the search in the x-axis direction for each y coordinate, if black is not detected for a predetermined length, the search is stopped. Thereby, in the example of FIG. 26, only the candidate point above the tilted instruction information is detected. Next, a least-squares approximate straight line A is obtained for the candidate point above the entry portion. The angle between this straight line and the x-axis of the camera coordinate system is defined as the inclination θ of the instruction information. It should be noted that y (k) does not necessarily need to be incremented by one as long as at least two candidate points at the top of the entry portion can be obtained. [STEP2f] Detection of tilt correction instruction area using installation marker: least square line A obtained in STEP1f and x in the camera coordinate system
Since the angle formed by the axes is the inclination of the written portion, this straight line A is translated in a direction away from the origin and in a direction approaching the origin, and a straight line when no black pixel exists on the straight line A Are defined as straight lines B and B ′, respectively. A black pixel having the maximum x coordinate x_max (i) is searched for in an area surrounded by the straight lines B and B ′ and the straight lines x = xr (i) and x = (width of image). When the inclination θ of the straight line A is positive, the intersection B1 of the straight line B and x = xr (i), the coordinate position B2 of the perpendicular foot from the intersection B1 to the straight line B ′, x = x_max (i) and the straight line B The area surrounded by the four points of the intersection B3 and the coordinate position B4 of the foot of the perpendicular from the intersection B3 to the straight line B 'is the area of the instruction information. Here, as shown in FIG. 26, when a tilt θ occurs in the clockwise direction with respect to the camera coordinate system (xy coordinate system) of the camera 1, it is assumed that the tilt θ is positive. [STEP3f] Presence or absence of existing decoration area by pen-type indicating means:
With reference to the additional image generation unit 5, if there is an existing decoration image by the pen-type indicating means, STEP4f is executed; otherwise, ST4 is executed.
Execute EP3e. [STEP4f] Deformation of partial image by inclination angle: The installation marker-based movement parameter calculation unit 17 deforms an existing decoration image according to the movement of a document.
A partial image for template matching described in EP2d is selected, and the image is rotationally deformed at the inclination angle of the document obtained in STEP1f. [STEP5f] Calculation of translation parameters by matching partial images: Using the template created in STEP4f,
As described above, matching is performed between two temporally continuous images to obtain a movement parameter. Since the template is rotated by the rotation parameter θ of the document, the required movement parameter is only translation (xd, yd). [STEP 6f] Deformation of the decoration image: All the images stored in the additional image generation unit 5 are deformed by the movement parameters obtained in STEP 5f.

As described above, according to the sixth embodiment, even when a document is moved when the installation marker is used, the inclination of the instruction information and its area are detected. Even if there is an existing decoration image, the existing decoration image is transformed and synthesized by calculating the movement parameter, so even if the document moves during the presentation using the installation marker, the emphasis part is correctly decorated. can do. (Embodiment 7) Next, the seventh embodiment according to the image synthesizing apparatus of the present invention will be described.
An embodiment will be described.

In the fifth embodiment, it has been described that the decoration of the instruction information can be easily performed simply by placing or removing the installation marker. In the seventh embodiment, sound effects are introduced together with the decoration by the installation markers to make it easier to confirm the operation and to further enhance the presentation effect.

FIG. 27 is a diagram showing the configuration of the seventh embodiment. The same components as those of the fifth embodiment shown in FIG. 20 are denoted by the same reference numerals, and detailed description is omitted.

The main difference between the seventh embodiment and the fifth embodiment is that an installation instruction means for selecting predetermined audio data by increasing or decreasing the number of installation markers detected at predetermined time intervals. That is, a number increase / decrease detection unit 18 is added. Reference numeral 19 denotes an audio data storage unit for storing audio data;
A speaker 0 reproduces the selected audio data. Here, the installation instruction member number increase / decrease detection unit 18 corresponds to the installation instruction member increase / decrease detection unit of the present invention.

The operation of the image synthesizing apparatus according to the seventh embodiment configured as described above will be described with reference to the flowchart of FIG.

The operation of the seventh embodiment is the same as the operation of the fifth embodiment except that the following STEPs 1g to 1ST are set between STEP8e and STEP9e.
EP4g is inserted. [STEP1g] Comparison of the number of installation markers between frames: The number of installation markers detected by the installation instruction means increase / decrease detection unit 18 is compared between the frames.
Execute 2g, STEP3g in case of decrease, and STEP9e in case of no change. [STEP2g] Selection of audio data for increase: Select audio data for increase, and proceed to STEP4g. [STEP3g] Selection of audio data for reduction: Select audio data for reduction, and proceed to STEP4g. [STEP4g] Output of sound effect: The control unit 1000 transmits the audio data selected in STEP2g or STEP3g to the speaker 2
Output from 0.

According to the above procedure, only by examining the change in the number of installation markers, when a marker is newly placed on a document or removed, the sound effect is simultaneously added to the decoration of the instruction information. Can be output.

As described above, according to the seventh embodiment, the case where a marker is added or removed from a document can be identified by increasing or decreasing the number of markers, and a sound effect corresponding to the marker can be added. In addition to facilitating confirmation, a more effective presentation is possible. (Eighth Embodiment) Next, an eighth embodiment according to the image synthesizing apparatus of the present invention will be described.
An embodiment will be described.

In the first to seventh embodiments, examples have been described in which a designated portion is detected and decorated from a relatively simple document including bullet points and photographs. In the eighth embodiment, furthermore, by recognizing the shapes and associations of a plurality of indicating means, it is considered that a document in which characters and figures are mixed can be decorated.

FIG. 29 is a block diagram of the eighth embodiment. The same components as those of the fifth embodiment shown in FIG. 20 are denoted by the same reference numerals, and detailed description is omitted.

The main difference between the eighth embodiment and the fifth embodiment is that the pointing means shape recognizing section 21 for recognizing at least two kinds of shapes of the pointing means, and the pointing means shape recognizing section 21
Instruction unit correspondence determining unit 2 that determines the association of the instruction units of a predetermined combination from the shape of the instruction unit recognized by
2 is added. Here, the pointing means shape recognition unit 21 corresponds to the pointing member shape recognition means of the present invention, and the pointing means correspondence determining unit 22 corresponds to the pointing member correspondence determining means of the present invention.

The operation of the image synthesizing apparatus according to the eighth embodiment thus configured will be described with reference to the flowchart of FIG. In order to explain the operation of the present embodiment more easily, a document in which a character string and a graphic are mixed as shown in FIG. 31 is associated by recognizing the key shape using two types of key-shaped markers. An example of decorating an area will be described. Such a marker may be marked in advance in a textbook, for example, by a school teacher as a preparation for a lesson. The indication by the marker may be affixed with a sticker or the like, or may be drawn on a document with a highlighter pen or the like.

The operation procedure of the eighth embodiment will be described below. The operation of the eighth embodiment is such that the following STEP1h to STEP5h are inserted between STEP1e and STEP2e of the operation of the fifth embodiment. [STEP1h] Detection of a key-shaped marker: A region of a predetermined size having a predetermined color (RGB value in a predetermined range) representing a key-shaped marker is detected as a marker from the image by the different-type plural-pointing unit pointing position detection unit 14. I do. [STEP2h] Recognition of marker shape: pointing means shape recognition unit 2
The type of the marker detected by step 1 is recognized. Here, the start marker and the end marker as shown in FIG. 32 are recognized. Specifically, the region of the detected marker is shown in FIG.
2, the number of pixels having a predetermined color (RGB value) representing a marker is counted for regions 0 and 3, and if the number is larger in region 0, the start marker is used, and vice versa. If it is, it recognizes it as an end marker. [STEP3h] Marked part detection by mapping markers:
The instruction means correspondence determination unit 22 determines which start marker corresponds to which end marker and a region is designated. First, for all the start markers, the distances to all the end markers at the lower right of the user are calculated. next,
Correspond from the marker with the smallest distance. [STEP4h] Detection of characteristic amount of instruction information: The instruction information detecting unit 15 corresponding to different plural instruction means obtains, for example, a ratio of white pixels in an instruction area as a characteristic amount of instruction information. In general,
The ratio of white pixels in the designated area is (drawing) <(character string) <
(Photos) so that they can be identified by an appropriate threshold. [STEP5h] Decoration of entry portion: An image obtained by replacing the RGB values of pixels other than black with RGB values of a predetermined color in the area of the instruction information detected in STEP3a by the additional image generation unit 16 corresponding to different plural instruction means. It is stored in the additional image generation unit 5 as an additional image.

According to the above procedure, the key-shaped markers are correctly associated with each other, and the designated area is decorated as shown in FIG.

As described above, according to the eighth embodiment, it is possible to decorate not only a bulleted character string but also a more complicated document including a figure in which characters and figures are mixed. (Embodiment 9) Next, a ninth embodiment according to the image synthesizing apparatus of the present invention will be described.
An embodiment will be described.

In contrast to the first to eighth embodiments, in the ninth embodiment, even when the camera is further zoomed,
The emphasis was placed on the correct display.

FIG. 34 is a block diagram of the ninth embodiment. The same components as those of the first embodiment shown in FIG. 1 are denoted by the same reference numerals, and detailed description is omitted.

The main difference between the ninth embodiment and the first embodiment is that a zoom magnification detecting section 23 for detecting the zoom magnification of the camera 1 and a decorative image based on the detected zoom magnification are further provided. Decorative image enlargement / reduction unit 24 for enlargement / reduction
Is added.

The operation of the image synthesizing apparatus according to the ninth embodiment configured as described above will be described with reference to the flowchart of FIG. The operation of the ninth embodiment includes the following steps STEP1i to ST1 between STEP1a and STEP2a of the operation of the first embodiment.
EP3i is inserted. [STEP1i] Presence or absence of existing decoration area: The additional image enlarging / reducing unit 24 refers to the additional image generation unit 5, and if there is an existing additional image, proceeds to STEP2i, and if not, executes STEP2a. [STEP 2i] Detection of zoom magnification: The zoom magnification detection unit 23 detects the zoom signal of the camera 1 via the control unit 1000 and sends it to the additional image enlargement / reduction unit 24. [STEP3i] Enlargement / reduction of decorative image: All images stored in the additional image generation unit 5 are enlarged or reduced by the zoom magnification detected in STEP2i.

According to the above-described procedure, according to the ninth embodiment, even when the camera is zoomed, the additional image is enlarged or reduced in accordance with the zoom magnification, so that the portion to be emphasized can be correctly decorated. You can expand the range of your presentation. (Embodiment 10) Next, a tenth embodiment according to the image synthesizing apparatus of the present invention will be described.

In the first to ninth embodiments, an example has been described in which an area designated by a plurality of pointing means such as a pen and an installation marker is decorated with a color or an underline. The tenth embodiment further considers that a designated area colored with a predetermined color is not displayed.

FIG. 36 is a block diagram of the tenth embodiment. The same components as those of the first embodiment shown in FIG. 1 are denoted by the same reference numerals, and detailed description is omitted.

The main difference between the tenth embodiment and the first embodiment is that a colored designated area erasing unit which detects an area colored or included in a predetermined color and erases the display of the information of the area. 25 is added.

The operation of the image synthesizing apparatus according to the tenth embodiment configured as described above will be described.

The designated colored area erasing section 25 raster-scans the image, and approaches the area having a predetermined RGB value (for example, 8
(Nearby) Pixels are assigned the same label as the same designated area. Next, for a group of a predetermined number or more of pixels having the same label, all colors of pixels of the same group labeled with a predetermined color (RGB value) are replaced.

According to the tenth embodiment, the information of the area designated by the predetermined color is not displayed on the display unit according to the above procedure. By painting the answer and comment part in a predetermined color, it is possible to prevent unnecessary information from being shown to the listener. (Eleventh Embodiment) Next, an eleventh embodiment according to the image synthesizing apparatus of the present invention will be described.

In the first to tenth embodiments, an image that embellishes or emphasizes a designated portion is generated and synthesized and displayed by simply pointing a document or the like by a predetermined pointing device. I explained what you can do,
The eleventh embodiment further considers expansion as an efficient educational system.

FIG. 37 is a block diagram of the eleventh embodiment. The same components as those of the first embodiment shown in FIG. 1 are denoted by the same reference numerals, and detailed description is omitted.

A major difference between the eleventh embodiment and the first embodiment is that a browser-compatible file for storing an image synthesized by the image synthesizing unit 5 in a format that can be displayed on an Internet browser is further provided. That is, the updating unit 26 is added. Reference numeral 27 denotes a server which manages a home page and is connected to a plurality of terminals via a network or the like.

The operation of the image synthesizing apparatus according to the eleventh embodiment configured as described above will be described.

For example, a case where a school teacher uses the image synthesizing apparatus according to the present embodiment in a class will be described as an example.
The control unit 1000 includes a memory for storing an image. For example, by pressing a predetermined key of a keyboard connected to the control unit 1000, the image synthesized by the image synthesis unit 5 is stored. Alternatively, as in the third embodiment, for example, a method of superimposing an image “save image” on a video of the camera 1 and instructing the superimposition with a pen or the like can be adopted. This allows the teacher to save, to the system, images that are useful to the students among the images created in the lesson. Next, the browser-compatible file creating / updating unit 26 converts the saved image into a GIF or JPEG format, and creates or updates, for example, an HTML file capable of referring to the server 27. As a result, students can view the lecture materials from off-campus or the like.

According to the above-described procedure, according to the eleventh embodiment, the created composite image can be registered on the Internet homepage, so that it is possible to participate in a class at a school such as a university from outside the university via a terminal or to give a lecture. It is possible to realize an efficient educational support system such as reusing materials and giving teachers assignments for reports. (Twelfth Embodiment) Next, a twelfth embodiment according to the image synthesizing apparatus of the present invention will be described.

In the first to eleventh embodiments, the superimposition of the additional image synthesized according to the feature is displayed by simply pointing the actual information described in the document or the like with a pen or the like. Although it was shown that an effective presentation system can be realized, in the twelfth embodiment, as an additional image to be used for synthesis, a plurality of real objects are easily captured and processed, and the synthesized image is transmitted by FAX or FAX. This is designed to allow transmission to a TV or the like.

FIG. 38 is a block diagram showing the twelfth embodiment. The same components as those in the first embodiment shown in FIG. 1 are denoted by the same reference numerals and detailed description thereof will be omitted.

The main difference between the twelfth embodiment and the first embodiment is that the display unit 8 is replaced with an input display integrated type display 28, and an input position detection unit for detecting the input position to the display 28. 29, the camera 1 is installed so as to capture at least a part of the display unit of the display 28, and a real object extracting unit 30 and a real object extracting unit 30 for extracting a contour of a real object placed on the display unit of the display 28 A decoration image generation unit 31 that generates an additional image to be displayed on a display for decoration or emphasis by processing the image of the object extracted by the processing, and an additional image storage unit 32 that stores the generated additional image. A decoration function selection unit 33 for selecting a display method of the added image on the display, and a real object on the display 28 and a display image on the display 28. Image receiving unit 34 for transmitting or receiving an image synthesized by the image synthesizing unit 7 to the FAX and TV
It is to have. Further, the control unit 1000 includes a ROM
And a memory such as a RAM and a CPU, in which processing corresponding to the input from the display 26 is stored.

Next, the operation of this embodiment will be described.

FIG. 39 is a flowchart showing the operation of the image synthesizing apparatus according to the twelfth embodiment. FIG. 40 shows an example of an imaging range of the input display-integrated display and the camera according to the present embodiment. In order to explain the operation of the present embodiment more clearly, a case will be described here as an example in which a message image of thanks to a person who has received a clock for joining the company is combined and transmitted to a FAX.

The operation procedure of this embodiment will be described below. [STEP1j] Installation of a real object: In the imaging area of the camera 1 arranged above the input display integrated display 28,
Set up the real thing you want to use for the message. here,
As shown in FIG. 40, an example will be described in which a clock received from a partner who intends to transmit a message and a name tag obtained by joining the company and assigned to the company are provided on the display 28.
Here, if an ultrasonic surface acoustic wave type or electromagnetic induction type object is used as the input surface of the display 28, it is not erroneously determined that an input has been made on the display 28 even when a hard object such as a watch is placed. In addition, even a pressure-sensitive type or a resistive type is not erroneously determined as long as it is light. The control unit 1000 manages the presence or absence of an input to the display 26. [STEP2j] Extraction of a real object: As shown in FIG. 41, functions that can be selected by the decoration function selection unit 33 are preferably displayed as icons on the display 28. If the user presses the extraction icon 401 of a real thing, a mode for selecting a real thing to be captured is set. A real object (clock and name tag in this case) to be extracted is surrounded by one closed curve by pen input. Control unit 10
00 receives a pen input end signal, first captures an image of a shooting area by the camera 1, and then activates the real thing extraction unit 30 to extract a real thing based on the captured image. Here, the start and end of pen input are determined by the pen input icon 4
This is done by pressing 11. The real thing extraction unit 30 is configured as shown in FIG.
As shown in FIG. 2, one closed curve is automatically divided into only the number of objects (here, two), and a clock and a name tag are independently extracted. This method will be described later in detail.
Note that pen input can be made with a soft finger or rubber if it is an ultrasonic surface acoustic wave method, and it can be input with a finger as well as a hard one such as a normal ball-point pen cap if it is a pressure-sensitive type or a resistive type. [STEP3j] Selection of decoration mode: The control unit 1000 activates the decoration image generation unit 31, changes the color and shape of the extracted real image, generates a decoration image, and displays the decoration image storage unit 32 To memorize. There are various methods for this. Here, reduction will be described as an example. The control unit 1000 displays the decoration image generated by the decoration image generation unit 31 (displayed as the extracted reduced image 422 of the real object in FIG. 42) and the decoration mode button 421 on the display 28 as shown in FIG. . Here, the selection of the decoration image and the decoration mode is performed by touching an image or a button icon displayed on the display 28 with a pen or the like. These input instructions are processed by the input position detector 29.

Various decoration modes can be considered.
Here, the description will be made as three modes of “background”, “edge”, and “free”. The "background" mode displays the decorative image on the entire display area of the display 28, the "rim" mode displays the decorative image along the inside of the display area, and the "free" mode displays the decorative image at an arbitrary place touching the display 28 with a pen or the like. . FIG.
Is a reduced image 422 of the extracted real object as a decoration image
Is the result of selecting a clock image from among the above and selecting the button 421-a corresponding to the mode of “edge” with a pen or the like.
The control unit 1000 corresponds to the “edge” mode, and displays the clock image 422 reduced along the inner edge of the display area of the display 28. [STEP4j] Message input: Enter the message you want to convey to the other party by pen input. For example, a message as shown in FIG. 44 is input. Pen input is performed by changing the display on the display at the position touching the input surface with the pen to a predetermined color (for example, black). [STEP 5j] Transmission of synthesized image: As shown in FIG. 44, when a “Send” button 402 displayed on the display 28 is pressed, the control unit 1000 first activates the image synthesis unit 7 and captures the image by the camera 1. The image data of the real object on the display 28 and the data of the decoration image and the message data displayed on the display 28 are stored as a composite image in, for example, a bitmap format. Next, the stored image is converted into a data format corresponding to the device to be transmitted by the image transmitting / receiving unit 34, and the composite image is transmitted. For example, when transmitting an image to a facsimile, the image may be converted to a run-length format.

The image data of the real object on the display 28 captured by the camera 1 described above is processed by the real object extracting unit 30, sent to the image synthesizing unit 7, and sent from the decorative image storage unit 32. The image is synthesized in the image synthesizing section 7 together with the output data of the decorative image and the like.

Further, as another combining method other than the above, the following method may be adopted.

That is, the real object on the display 28 and the decorative image or message displayed on the display 28 are imaged by the camera 1 as they are. In this case, the composite image from the camera 1 can be directly sent to the image transmitting / receiving unit 34. Also, in this case, unlike the above-described combining method, the quality of an image such as a decorative image or a message displayed on the display 28 is degraded.

Now, a method of extracting a real object on the display by the real object extracting unit 30 in [STEP2j] will be described in detail.

FIG. 45 is a detailed block diagram of the real thing extracting unit 30. In FIG. 45, reference numeral 35 denotes an image storage unit that stores an image captured by the camera 1; and 36, a closed curve input to the display 28 is discretized as a candidate contour point of an object in the image stored in the image storage unit 35. An outline model storage unit 37 for storing the outline model, and an outline model transformation unit 37 for moving the outline candidate points stored in the outline model storage unit 36 in accordance with a predetermined rule. 38 is a closed curve connecting the outline candidate points in a predetermined combination. A contour model intersection judging unit 39 for judging the presence / absence of intersection, 39 is a contour model splitting unit for classifying contour candidate points into a plurality of sets when an intersection is detected by the intersection judging unit 38, and 40 is no intersection by the intersection judging unit 38 A contour candidate point generation / deletion unit 41 for newly adding or deleting a contour candidate point when it is determined that the extraction is completed. That.

[0111] The real object extraction unit 30 configured as described above.
Will be described with reference to the flowchart of FIG. The image storage unit 35 is usually constituted by a RAM,
An image captured by the camera 1 is digitized and stored. The pixel data of the digitized image is stored, for example, by expressing its luminance in 8 bits. The operation procedure of the real thing extraction unit 30 will be described below. [STEP2j_1] Initial placement of contour candidate points: [STE
P2j], the n points constituting the closed curve (contour model) input to the display 1 with a pen are converted into initial contour candidate points vi (xi
(0), yi (0)) (i = 1, 2,..., N) are stored in the contour model storage unit 36. i represents the connection order of the contour candidate points for forming a closed curve (contour model) connecting the contour candidate points. The contour model storage unit 36 is usually constituted by a RAM. [STEP2j_2] Deformation of contour model: contour model transformation unit 37
Moves the positions of all the contour candidate points once.
Thereby, the closed curve connecting the contour candidate points is deformed.
The closed curve is obtained by connecting the contour candidate points according to a predetermined connection order. The deformation of the contour model is based on the energy Esnake
This is performed by moving each contour candidate point in a direction that minimizes (vi). This defined energy will be described later.

For example, when each contour candidate point is temporarily moved within a predetermined range, it may be moved to a position where Esnake (vi) is minimized.

[0113]

(Equation 10)

The steepest descent vector of Esnake (vi) expressed by the above equation (10) is obtained for each contour candidate point vi (xi, yi) (i = 1, 2,..., N), and is moved by the following equation. Is also good.

[0115]

(Equation 1)

Here, t (i = 0, 1,...) Represents the number of movements of each point. Kx> 0 and Ky> 0 are constants for controlling the movement amount of the contour candidate point. The movement of each contour candidate point according to Equation 1 may be performed sequentially along the contour or independently in parallel for each point. However, if the energy Esnake (vi) increases due to the movement, the energy is not moved. At this time, the presence or absence of the movement is stored in a flag F (vi) (i = 1, 2,..., N). F (vi) = 1 for moved contour candidate points, F (vi) = 0 for contour candidate points that have not moved
And As energy Esnake (vi) of the contour model,
For example, the sum of the respective energy terms represented by the following equations may be used. Note that various types of energy can be considered according to the characteristics of the target object. (1) Espline representing the smoothness of the contour model:

[0117]

(Equation 2)

The moving direction of the contour candidate point that minimizes Equation 2 is a direction in which the contour model contracts. Here, i, i-1, and i + 1 in vi, vi-1, and vi + 1 are subscripts of v, respectively. The same applies to the following mathematical expressions. (2) Eare equivalent to the area of the closed area surrounded by the contour model
a:

[0119]

(Equation 3)

Here, it is assumed that xn + 1 = x1, yn + 1 = y1. The moving direction of the contour candidate point that minimizes Equation 3 is a direction perpendicular to the contour model. If Earea is used, the contour of a concave-shaped object can be extracted. Where xi, xi
I and i + 1 in +1 are subscripts of x, and i and i + 1 in yi and yi + 1 are subscripts of y. The same applies to the following mathematical expressions. (3) Edist for averaging the distance between contour candidate points:

[0121]

(Equation 4)

Here, dav is the average distance between the contour candidate points. (4) The magnitude Eedge of the gradient of the image luminance I (vi):

[0123]

(Equation 5)

The moving direction of the contour candidate point that minimizes Equation 5 is the direction in which the luminance gradient increases, that is, the direction toward the contour portion of the object. (5) Image luminance value Eintens:

[0125]

(Equation 6)

The moving direction of the contour candidate point that minimizes Equation 6 is a direction in which the luminance decreases (dark) when Wntens> 0, and a direction in which the luminance increases (bright) when Wintens> 0. Also, Wsp1, Wsp2, Warea, Wdist, Wedg
e, Wintens> 0 are weighting factors of each energy term, respectively.

If the contour model is deformed so as to minimize the sum of the energy terms of Equations (2) to (6), the initial contour contracts at first, but the vector for moving the contour candidate point by each energy term in the contour part of the object. It stops because it is balanced. At this time, a polygon in which the contour candidate points are connected in a predetermined order is extracted as the contour of the object. [STEP2j_3] Intersection determination of contour model: The contour model intersection determination unit 38 determines the presence / absence of intersection for all combinations of two different sides with respect to a closed curve (polygon) in which contour candidate points are connected in a predetermined connection order. However, two connected sides are excluded. The intersection may be determined, for example, as follows. If the line segment vivi + 1 and the line segment vjvj + 1 have an intersection, there are real numbers p (0 ≦ p ≦ 1) and q (0 ≦ q ≦ 1) that satisfy the following expression.

[0128]

[Mathematical formula-see original document] p (vi + 1-vi) + vi = q (vj + 1-vj) + vj Equation 7 is a simultaneous equation relating to p and q, and the determinant det calculated by the following equation is not 0 Have a solution, then a line segment vi
vi + 1 and the line segment vjvj + 1 intersect at only one point. (However, d
When et = 0, the line segment vivi + 1 may coincide with the line segment vjvj + 1, and the intersection in this case is determined by matching the contour candidate points before calculating the determinant. )

[0129]

[Equation 8] det = (xi-xi + 1) (yj + 1-yj)-(xj + 1-xj)
(yi-yi + 1) Therefore, when det ≠ 0, p, q calculated by the following equation
However, when 0 ≦ p ≦ 1, 0 ≦ q ≦ 1, it is determined that polygons (contour models) in which the contour candidate points are connected in a predetermined connection order intersect, and STEP4a is executed. Otherwise, STEP5a is executed. Execute.

[0130]

## EQU9 ## p = ((yj + 1-yj) (xi-xj) + (xj-xj + 1) (y
i-yj)) / det q = ((yi + 1-yi) (xi-xj) + (xi-xi + 1) (yi-yj)
) / det [STEP2j_4] Division of contour model: contour model division unit 39
Divides the contour model into multiple parts, and returns to STEP2a. The contour model is split at contour candidate points forming a line segment having an intersection. FIG. 47 shows an example of a typical intersection that occurs when Earea is used as the energy of the contour model. Eare
a takes a positive or negative value, but in this example, the contour model is divided into two by connecting the contour candidate points vi and vj + 1 and vj and vi + 1, respectively, so that the sign does not change. Break up. That is, the connection order is {v1, ..., vi, vj + 1, ..., vn}
A set of contour candidate points of {vi + 1,..., vj} is newly split into independent contour models. The intersection can occur at a plurality of places at the same time, but the procedure of splitting into two may be recursively repeated to split into a plurality of contour models. [STEP2j_5] Generation / deletion determination of contour candidate points: The contour candidate point generation / deletion unit 40 generates and deletes new contour candidate points. For example, contour candidate points may be generated and deleted in the following cases. The distance between adjacent contour candidate points is | v
When i + 1-vi |> DTH is satisfied, a new discrete point is generated between two points vi and vi + 1. Here, DTH is a predetermined maximum distance between contour candidate points. If there are generated discrete points, Gflag = 1; otherwise, Gflag = 0. Also, cosθ> θTH
(Θ is an angle between vivi-1 and vivi + 1). Here, θTH is a predetermined threshold for determining the sharpness of the contour model. If there are discrete points that have disappeared, Dflag = 1; otherwise, Dflag = 0. By additionally generating the contour candidate points, the number of contour candidate points per one object reduced by division can be compensated, and the contours of a plurality of objects can be correctly extracted. Further, according to the disappearing procedure, the contour candidate points stopped at positions other than the contour of the target object are deleted, and erroneous contour extraction can be avoided. [STEP2j_6] Extraction completion determination: When all the outline candidate points have not moved (∀i; F (vi) = 0) and no new outline candidate point has been generated or deleted by the extraction completion determination unit 41 (Gflag =
0 and Dflag = 0), and the extraction of the contour ends. Otherwise ST
Return to EP2a.

As described above, [STEP1] to [STEP5] and [STEP2j_1]
According to the twelfth embodiment, the procedure described in [STEP2j_6] places a plurality of real objects to be combined on a display and gives one closed curve that includes them to reduce the deformation of the closed curve. By detecting the accompanying self-intersection and dividing it into a plurality of pieces, they can be extracted independently, so that a composite image of a real object and a virtual object can be easily created. Further, even when there are many synthesized images to be created, it is possible to reduce the burden on the user in extracting a real object.

Note that the above [STEP1j] to [STEP4j] are not necessarily performed in this order. For example, after the [STEP4j] message is input first,
P1j] to [STEP3j] can combine images in any combination, such as taking in a real object into a computer and generating a decoration image. It goes without saying that images can be synthesized by repeatedly executing the above [STEP1j] to [STEP4j] in an arbitrary order. However, when displaying a decorative image or a message by pen input on the display and then extracting a new real object placed on the display, the pen 1 is used only while the image is captured by the camera 1 in conjunction with the closed curve input by the pen. The partial area in the display area of the display 28 including the input closed curve is changed to a predetermined color (for example, white or black). As a result, only the real object can be extracted without being affected by the virtual object.

Further, it goes without saying that the embodiment of the present invention can be easily extended to combine with a snapshot by connecting to a digital camera, or to add a voice message by connecting to a microphone. No. (Thirteenth Embodiment) Next, a thirteenth embodiment according to the image synthesizing apparatus of the present invention will be described.

In the twelfth embodiment, for example, when inputting a message with a pen, pressing a “pen” button determines whether the input is a message or a closed curve for selecting a real object. The command that distinguishes the above is described in a form in which the user inputs to the system. The thirteenth embodiment further reduces the input burden on such a system and uses a miniature model such as a real pen, eraser, or FAX that is usually used instead of the user pressing the selection button. The present invention provides an interface used as a three-dimensional icon for intuitively understanding its function, and is designed to make it easier to compose an image. At this time, even when the virtual object is displayed on the display 28 and includes the same color as a plurality of real objects to be identified, the virtual object is considered so as to be able to stably identify which physical object it is.

FIG. 48 is a block diagram of the thirteenth embodiment. The same components as those of the twelfth embodiment shown in FIG. 38 are denoted by the same reference numerals, and detailed description is omitted.

The main difference between the thirteenth embodiment and the twelfth embodiment is that the twelfth embodiment differs from the twelfth embodiment in that a real object such as a pen used for input to the system or a three-dimensional icon for registering a three-dimensional icon is registered. That is, a registration unit 42 and a three-dimensional icon identification unit 43 for identifying what the input is when the input is made on the display 1 are added.

The operation of the image synthesizing apparatus according to the thirteenth embodiment configured as described above will be described only for components different from those in the twelfth embodiment.

First, the operation of the three-dimensional icon registration unit 42 will be described. The three-dimensional icon registration unit 43 is activated by the user by pressing an icon displayed on the display 28 by the decoration function selection unit 33 before combining images. When the three-dimensional icon registration unit 42 is activated, a list of functions of the system such as “character input” and “delete” is displayed on the display 1. The user shoots a real object such as a ball-point pen and an eraser that he or she normally uses with the camera 1 and registers the feature amount in this list. The list may be saved as a file in a text format, for example, and the information may be displayed on the display 28 when the three-dimensional icon registration unit 42 is activated. As the feature amount, for example, R
The frequency distribution of each GB value is stored. When taking an image with a camera, it is desirable to take a picture in a state where the camera itself is actually used. In addition, the manufacturer attaches a dedicated pen, an eraser, a facsimile three-dimensional icon, or the like at the time of shipment, stores the function corresponding to the color, and does not necessarily have to be set by the user.

Next, the operation of the three-dimensional icon identification section 43 will be described. The three-dimensional icon identification section 43
Fired every time there is input to The control unit 1000 manages the presence / absence of input to the display 28 and the position coordinates. First, the three-dimensional icon identification unit 43
000, the coordinates of an input position to the display 28 are received from the camera 1 and an area of a predetermined color (for example, white) and size (for example, a square of 120 pixels) corresponding to the size of the three-dimensional icon is displayed at that position. Import images.
Next, a frequency distribution is created by examining the RGB values of each pixel only in the previous region of the captured image, and a three-dimensional icon having the highest similarity to the frequency distribution registered in the three-dimensional icon registration unit 42 is used. Is determined to be
As the similarity, for example, the sum of the absolute values of the differences between the classes of the frequency distribution is used. As a result, if it is identified as a pen, the locus of movement of the pen is displayed on the display 28 as a character input, and if it is identified as an eraser, the portion of the locus of movement of the eraser is erased. In the case of a FAX miniature icon, a composite image is transmitted. However,
When a facsimile icon or the like is used, the viewing angle of the camera is set wider than the image synthesis area, and the camera is placed in that area so as not to affect the synthesized image.

By examining only the feature amount of the real object by the above procedure, it is possible to correctly identify the three-dimensional icon without being affected by the display image, so that the three-dimensional icon registered stably can be used. Images can be combined with more intuitive operations.

As described above, according to the thirteenth embodiment, after displaying an area of a predetermined color and size at the input position on the display, the image is taken in and the color is examined only in that area. Only the features of the real object can be examined without being affected, and even if the virtual object displayed on the display contains the same color as multiple real objects to be identified, it is possible to correctly identify the real object. Can be identified,
You can use three-dimensional icons that have been stably registered,
Images can be synthesized with a more intuitive operation. (Embodiment 14) Next, a fourteenth embodiment according to the image synthesizing apparatus of the present invention will be described.

In the twelfth embodiment, when a message is input with a pen, the character color is described as black.
Generally, in the case of the pen input system, the user selects a desired color from a menu or the like. For example, if the destination of the synthesized image is a monochrome FAX, it is easier to see the pen color black when entering characters into a bright image and white when a dark image is input. The fourteenth embodiment is designed to automatically change the color of the pen according to the brightness of the display image, to further reduce the mode change input load on the system, and to make it possible to synthesize a more easily viewable image. It is.

FIG. 49 is a block diagram of the fourteenth embodiment. The same components as those of the twelfth embodiment shown in FIG. 38 are denoted by the same reference numerals, and detailed description is omitted. The main difference between the fourteenth embodiment and the twelfth embodiment is that
In the second embodiment, a display image brightness calculation unit 44 that calculates the brightness of a display image, and a pen input color that changes the color of the pen input according to the brightness calculated by the display image brightness calculation unit 44 That is, a change unit 45 is added.

The operation of the image synthesizing apparatus according to the fourteenth embodiment configured as described above will be described only for the display image brightness calculation unit 44 and pen input color changing unit 45 different from the twelfth embodiment. .

The display image brightness judging section 44 is activated every time there is a pen input to the display 28, and calculates the average luminance of the entire display image. The determination as to whether or not a pen input is performed is based on the method of the thirteenth embodiment. The pen input color changing unit 45 refers to the average brightness calculated by the display image brightness calculation unit 44, and determines, for example, that the pen input color is black if the average brightness is 180 or more of 256 gradations, and white otherwise. . As a result, the pen color is automatically changed to black when characters are input in a bright image, and white in a dark image, without making the user aware of the pen input color selection mode.
An image in which characters are easy to see can be synthesized.

As described above, according to the fourteenth embodiment, the color of the pen input is changed according to the brightness of the display image.
It is possible to compose an image in which the input characters are easier to see. (Embodiment 15) Next, a fifteenth embodiment according to the image synthesizing apparatus of the present invention will be described.

In the twelfth embodiment, the decoration image in which the size and the color of the extracted image are converted is displayed. Also, the first
In the fourth embodiment, the color of the pen input is changed according to the brightness of the display image, and an image in which the character information is easy to see is synthesized.
In the fifteenth embodiment, further consideration is given to changing the color and brightness of a display image according to the color and brightness of a real object placed on the display 1 so as to synthesize a more easily viewable image. It is.

FIG. 50 is a block diagram of the fifteenth embodiment. The same components as those of the twelfth embodiment shown in FIG. 38 are denoted by the same reference numerals, and detailed description is omitted.

The main difference between the fifteenth embodiment and the twelfth embodiment is that the twelfth embodiment differs from the twelfth embodiment in that the brightness and color of the display image are changed according to the brightness and color of the real object. That is, an image correction unit 46 is added.

The operation of the image synthesizing apparatus of the fifth embodiment configured as described above will be described only for the display image correcting section 46 different from that of the twelfth embodiment.

The display image correction section 46 is activated, for example, when the synthesis of the images is completed and the transmission button is pressed. First, the display image correction unit 46 activates the real object extraction unit 30 and extracts all the real objects on the display by setting a rectangle having the same size as the imaging area of the camera 1 as an initial contour model. Next, the display image correction unit 46 calculates the average luminance of all the extracted objects. Finally, the luminance of the display image on the display 28 is changed according to the value of the average luminance. For example, when the real object is bright (for example, average luminance) or more, the display image is also brightened (for example, the luminance of each pixel value is increased by 20%), so that a combined image that is easy to see as a whole is obtained.

As described above, according to the fifteenth embodiment, since the color and brightness of the displayed image are changed according to the color and brightness of the real object on the display, it is possible to synthesize a more easily viewable image.

A program for causing a computer to execute all or some of the steps (or processing operations) of each step (or each means) described in any one of the above embodiments is recorded. A magnetic recording medium, an optical recording medium, or the like may be created, and the same operation as described above may be realized using the recording medium.

The individual extracting means of the present invention is a real thing extracting unit in the above-described embodiment, and the case where the individual extracting means is applied to an input display integrated display has been described. However, the present invention is not limited to this. A simple image synthesizing device may be used.

In other words, the image synthesizing apparatus in this case has an image pickup means such as a camera for picking up an image of a surface such as a document on which a plurality of pieces of information such as a photograph and an illustration are represented and / or an object, and the image pickup means. Closed curve information detecting means for detecting closed curve information surrounding extraction target candidates, such as a plurality of illustrations to be image extraction candidates, among information such as a plurality of photographs or illustrations or a plurality of objects, and the detected closed curve Individual extraction means for individually extracting images such as illustrations from the images of the extraction target candidates such as the plurality of enclosed illustrations based on the information and the information of the image of the extraction target candidate such as the illustration, An image synthesizing means for synthesizing an image with another image such as a message using the image such as the illustration extracted individually. Thereby, information represented on a predetermined surface and / or a plurality of objects are imaged, and a plurality of extraction targets to be candidates for image extraction among the imaged information and / or the plurality of objects are extracted. Providing a closed curve surrounding the candidate, detecting the given closed curve information, based on the detected closed curve information and the information of the image of the extraction target candidate, from the image of the plurality of extraction target candidates surrounded, Each candidate image is extracted individually,
Image synthesis can be performed using the individually extracted images.

With this configuration, as in the case described above, even when there are a plurality of image extraction targets, the effect of reducing the work load for designating those extraction targets is reduced as compared with the related art. That is, the designation of the designated area can be performed more easily than in the past.

Alternatively, the following image synthesizing apparatus may be used.

That is, the image synthesizing device in this case is
An imaging unit that captures a surface on which information such as an illustration and a character string is displayed, and, among the plurality of pieces of information captured by the imaging unit, detect closed curve information that surrounds a plurality of pieces of pointing target information to be pointing targets. Closed curve information detecting means, and individually extracting an image of each indicated target information from the surrounded plurality of indicated target information images based on the detected closed curve information and the image information of the indicated target information. An instruction extracting unit that detects an area of the individually extracted image and / or a feature amount of the image, and an instruction information detecting unit that detects the area of the image and / or the feature amount of the image. An additional image generation unit that generates an additional image to be added to the image of the instruction target information, an additional image storage unit that stores the generated additional image, and an image obtained by the imaging unit. An image referents information, a configuration in which an image combining unit for combining the additional image that is generated. This configuration may be considered, for example, as a modification of the image synthesizing apparatus described in the eighth embodiment. That is, FIG.
As shown in FIG. 8, when two extraction targets are a set of polygonal figures and a photograph, as described in the eighth embodiment, instead of individually instructing each object, both of them are designated. It gives a closed curve that surrounds the object. The configuration of the individual extraction means is the same as that described above.

With this configuration, as in the case described above, even when there are a plurality of image extraction targets, the effect of designating those extraction targets can be further reduced as compared with the related art.

Further, in the above embodiment, the image synthesizing apparatus of the present invention has been described as applied to an input display integrated type display having a real object extracting unit. However, the present invention is not limited to this. A synthesizer may be used.

That is, the image synthesizing apparatus comprises an input-display integrated display, an image pickup means for picking up an image of at least a part of a display section of the display, and an instruction for indicating a real object placed on the display. A pointing position detecting unit that detects position information, and based on the detected pointing position information, an image captured by the imaging unit,
A real object extracting unit for extracting an image of the real object, an additional image generating unit for generating an additional image to be displayed on the display using the extracted image, and an additional unit for storing the generated additional image An image storage unit, a display method selection unit that selects a display method of the generated additional image on a display, and an image synthesis unit that synthesizes the additional image and the image of the real object based on the selected display method. And a unit. As a result, there is an effect that a composite image of a real object and a virtual object can be easily created in an unconventional manner.

As described above, according to the first embodiment, the presenter can specify the area or feature of the part by simply pointing the part to be emphasized of the information he wrote with the wand. Since the image for decoration is automatically superimposed and superimposed, there is no need to switch and use different colored pens. Furthermore, the decoration method can be easily changed according to the position indicated by the pen or the like.
It is possible to add the date and time, etc., or to erase or blink the information part that has already been decorated according to the designated position, so that it is effective on the spot without creating an OHP sheet for presentation in advance. Presentation is possible.

That is, there is an advantage that processing such as decoration or emphasis based on the image area, feature amount, and the like can be performed more easily.

Further, according to the second embodiment, the decoration method can be switched by identifying a plurality of types of markers attached to pens or the like, so that it is troublesome to switch between pens of a plurality of colors and write. Can be omitted, and the presentation can proceed smoothly.

According to the third embodiment, existing decoration images are displayed in a superimposed list and can be copied by designating them with the pointing stick. Therefore, even when a plurality of documents are used, the same character string can be used. There is no need to write on the document many times, and the presentation can proceed smoothly.

According to the fourth embodiment, the decoration image is transformed by obtaining template movement parameters by performing template matching between successive frames using a part of the decoration image as a template. Even if it moves, the part to be emphasized can be correctly decorated, and the user can give a presentation without worrying about the movement of the document.

According to the fifth embodiment, an intuitive operation such as decorating by placing a marker and erasing by removing a marker can be performed intuitively, so that anybody can easily give an effective presentation.

According to the sixth embodiment, since the entry portion and the inclination of the document are detected by using the installation markers, the number of the movement parameters of the document to be estimated can be reduced.
Since the additional image corresponding to the movement of the document can be synthesized at a higher speed, it is possible to reduce a sense of discomfort due to a delay in displaying the synthesized image.

According to the seventh embodiment, by detecting an increase or decrease in the number of markers, it is possible to discriminate between a case where a marker is placed and a case where it is removed, and to level a sound effect in accordance with that. You can give an effective presentation.

According to the eighth embodiment, by recognizing a plurality of marker shapes and associating them, it is possible to decorate a complicated document in which graphics and characters are mixed, and the presentation effect is further enhanced.

According to the ninth embodiment, the decorative image is enlarged or reduced in accordance with the zoom magnification of the camera. Therefore, even when the camera is zoomed, the correctly emphasized portion can be decorated, and the width of the presentation can be further increased. Can be spread.

According to the tenth embodiment, only the part that is not desired to be displayed is colored with a highlighter or the like, and the colored part is erased from the image. Therefore, the part that the user does not want to show to the listener can be hidden. it can.

According to the eleventh embodiment, the created composite image can be registered on the Internet homepage, so that it is possible to participate in a class at a school such as a university from outside the university via a terminal, or to reuse a lecture memo. And the teacher can provide an efficient educational support system, such as giving a report assignment.

According to the twelfth embodiment, a plurality of real objects to be synthesized are placed on a display, and only one closed curve including them is provided, and the self-intersection accompanying the deformation of the closed curve is detected. By dividing the image into a plurality of objects, they can be extracted independently, so that a composite image of a real object and a virtual object can be easily created.

According to the thirteenth embodiment, since the color for identifying a three-dimensional icon is checked only in a predetermined area corresponding to the input position on the display, the display image is not affected.
Since the three-dimensional icon can be correctly identified, the three-dimensional icon registered stably can be used, and the image can be synthesized by a more intuitive operation.

According to the fourteenth embodiment, the color of the pen input is changed according to the brightness of the displayed image, so that an image in which the input characters are more easily viewable can be synthesized.

According to the fifteenth embodiment, since the color and brightness of the display image are changed according to the color and brightness of the real object on the display, a more easily viewable image can be synthesized.

[0178]

As is apparent from the above description, the present invention has an advantage that the restriction on creating a composite image is smaller than that of the related art.

Also, the present invention has an advantage that a composite image of a real object and a virtual object can be created more easily.

[Brief description of the drawings]

FIG. 1 is a block diagram of a first embodiment of the present invention.

FIG. 2 is a flowchart showing an operation procedure according to the first embodiment of the present invention;

FIG. 3 illustrates an example of a document according to the present embodiment.

FIG. 4 is a diagram illustrating an example of decoration by a frame image according to the present embodiment;

FIG. 5 is a diagram illustrating an example of detection of a pointing position and a pointing region by a pen-type pointing device according to the present embodiment;

FIG. 6 is a diagram illustrating an example of decoration of a character area and a photograph area by the pen-type pointing device according to the embodiment;

FIG. 7 is a diagram showing an example of a method of changing the display of an existing decorative area by the pen-type indicating means according to the embodiment;

FIG. 8 is a block diagram of a second embodiment of the present invention.

FIG. 9 is a flowchart showing an operation procedure according to the second embodiment of the present invention;

FIG. 10 is a diagram showing an example of a plurality of markers attached to the pen-type indicating means of the embodiment and an attaching method.

FIG. 11 is a diagram illustrating an example of decoration by recognition of a marker type according to the present embodiment;

FIG. 12 is a block diagram of a third embodiment of the present invention.

FIG. 13 is a flowchart showing an operation procedure according to the third embodiment of the present invention.

FIG. 14 is a diagram illustrating an example of a method for displaying an existing decoration image according to the present embodiment;

FIG. 15 is a diagram showing an example of a method for selecting an existing decoration image according to the embodiment;

FIG. 16 is a diagram illustrating an example of a method for copying an existing decorative image according to the present embodiment;

FIG. 17 is a block diagram of a fourth embodiment of the present invention.

FIG. 18 is a flowchart illustrating an operation procedure according to the fourth embodiment of the present invention.

FIG. 19 is a diagram illustrating an example of estimating document movement parameters by template matching according to the present embodiment;

FIG. 20 is a block diagram of a fifth embodiment of the present invention.

FIG. 21 is a flowchart showing an operation procedure according to the fifth embodiment of the present invention.

FIG. 22 is a diagram showing an example of decoration by the installation marker of the present embodiment.

FIG. 23 is a diagram showing an example of erasing a decorative image by removing an installation marker according to the present embodiment.

FIG. 24 is a block diagram of a sixth embodiment of the present invention.

FIG. 25 is a flowchart showing an operation procedure according to the sixth embodiment of the present invention.

FIG. 26 is a diagram illustrating an example of detection of an instruction information area in which the inclination of a document has been corrected using the installation instruction means according to the present embodiment;

FIG. 27 is a block diagram of a seventh embodiment of the present invention.

FIG. 28 is a flowchart showing an operation procedure according to the seventh embodiment of the present invention;

FIG. 29 is a block diagram of an eighth embodiment of the present invention.

FIG. 30 is a flowchart showing an operation procedure according to the eighth embodiment of the present invention.

FIG. 31 is a diagram showing an example of a decoration area designating method by the key shape designating means of the present embodiment.

FIG. 32 is a diagram illustrating an example of a key shape indicating unit according to the present embodiment.

FIG. 33 is a diagram showing an example of a decoration result by the key shape indicating means of the present embodiment.

FIG. 34 is a block diagram of a ninth embodiment of the present invention.

FIG. 35 is a flowchart showing an operation procedure according to the ninth embodiment of the present invention;

FIG. 36 is a block diagram of a tenth embodiment of the present invention.

FIG. 37 is a block diagram of an eleventh embodiment of the present invention.

FIG. 38 is a block diagram of a twelfth embodiment of the present invention.

FIG. 39 is a flowchart showing an operation procedure according to the twelfth embodiment of the present invention;

FIG. 40 is a diagram showing an example of installation of a real object on a display according to the present embodiment.

FIG. 41 is a diagram illustrating an example of selection of a real object by pen input according to the present embodiment;

FIG. 42 is a diagram showing an example of a real object extraction result according to the present embodiment;

FIG. 43 is a view showing an example of decoration by a real object taken in the present embodiment;

FIG. 44 is a diagram showing an example of message entry by pen input according to the present embodiment;

FIG. 45 is a block diagram showing a detailed configuration of the real thing extracting unit 30 of the present embodiment.

FIG. 46 is a flowchart showing a physical object extraction procedure according to the embodiment.

FIG. 47 is a diagram showing an example of intersection of a contour model according to the present embodiment;

FIG. 48 is a block diagram of a thirteenth embodiment of the present invention.

FIG. 49 is a block diagram of a fourteenth embodiment of the present invention.

FIG. 50 is a block diagram of a fifteenth embodiment of the present invention.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 camera 2 pointing position detecting unit 3 pointing information detecting unit 4 additional image generating unit 5 additional image storage unit 6 pointing position information adding method determining unit 7 image synthesizing unit 8 display unit 9 marker type, pointing position determining unit Additional image generating unit 11 Existing additional image copying unit 12 Moving parameter estimating unit 13 Additional image deforming unit 14 Heterogeneous plural instruction means indicating position detecting unit 15 Heterogeneous plural instruction means corresponding instruction information detecting unit 16 Heterogeneous plural instruction means corresponding additional image generating unit 17 Installation marker use movement parameter calculation unit 18 Installation instruction means number increase / decrease detection unit 19 Voice data storage unit 20 Speaker 21 Instruction unit shape recognition unit 22 Instruction unit correspondence determination unit 23 Zoom magnification detection unit 24 Additional image enlargement / reduction unit 25 Coloring Designated area deletion unit 26 HTML file creation / update unit 27 Server and terminal 8 Input display integrated display 30 Real thing extraction unit 32 Decoration image storage unit 33 Decoration function selection unit 34 Image transmission / reception unit 36 Outline model storage unit 37 Outline model deformation unit 38 Outline model intersection determination unit 39 Outline model splitting unit 40 Outline candidate point Generation / disappearance unit 41 Extraction completion determination unit 42 3D icon registration unit 43 3D icon identification unit 44 Display image brightness calculation unit 45 Pen input color change unit 46 Display image correction unit 100 Pen, pointing rod 100a Marker 100b Pointed position detection start switch 200 Installation instruction means 300 Key shape instruction means 1000 Control unit

──────────────────────────────────────────────────の Continuation of the front page (51) Int.Cl. ⁶ Identification code FIG06F 15/66 470A According to the proviso of Article 64, Paragraph 2 of the Patent Act Figures 4, 5, 6, 7, Some of Figures 11, 14, 15, and 16 are not shown.

Claims (28)

[Claims] 1. A predetermined image pickup means, an instruction position detection means for detecting a position of instruction information given to a surface on which information is displayed, from an image taken by the image pickup means, and a detection by the indication position detection means Pointing target information detecting means for detecting a region of the pointing target information and / or a feature amount of the pointing target information displayed on the surface corresponding to the specified pointing position; and the region and / or the feature. Additional image generation means for generating an additional image to be added to the image of the instruction target information based on the amount; image synthesis for synthesizing the image of the instruction target information obtained by the imaging means and the generated additional image Means for synthesizing an image. 2. Based on the detected indicated position,
An information adding method determining unit for each designated position that determines a method of combining and / or displaying the additional image, wherein the image combining unit uses a determination result by the information adding method determining unit at the time of the combining; 2. The image synthesizing apparatus according to claim 1, wherein: 3. The apparatus according to claim 1, wherein the additional image is an image for decorating and / or enhancing the image of the pointing target information. 4. An apparatus according to claim 1, further comprising: an instruction unit having one or a plurality of types of instruction information, the instruction unit indicating the instruction target, wherein the instruction position detecting unit is further obtained from the image captured by the imaging unit. The type of the instruction information is determined using the instruction information, and the additional image generation unit generates the additional image in consideration of the type of the determined instruction information. The image synthesizing device according to claim 1. 5. An additional image storage unit for storing the generated additional image, and an additional image stored in the additional image storage unit is combined with an image obtained by the imaging unit to display a list.
The image compositing device according to claim 1, further comprising: an existing additional image copying unit that composites and displays the additional image selected from the list display at the designated position. apparatus. 6. A moving parameter estimating unit for estimating a position of the surface on which the information is displayed after the movement, and an additional image deforming unit for deforming the additional image based on the estimated moving parameter. The image synthesizing apparatus according to claim 1, wherein: 7. The pointing means has a plurality of pointing members, the pointing position detecting means detects the pointing position for each of the plurality of pointing members, and the pointing target information detecting section comprises: The detection according to any one of claims 1 to 6, wherein the detection is performed for each of the plurality of pointing members, and the additional image generation unit performs the generation for each of the plurality of pointing members. Image synthesis device. 8. At least one of the plurality of instruction members is an installation instruction member that indicates the instruction target by being installed on a surface on which the information is displayed. The image synthesizing device according to claim 7. 9. The pointing target information detecting means estimates a tilt angle of a region of the pointing target information with respect to a coordinate system of the image pickup means based on a setting position of the setting indicating member, and the estimated tilt angle Calculating a feature amount of the pointing target information and / or a region of the pointing target information, and using the estimated inclination angle to estimate a translation parameter of a plane on which the information is expressed. 9. The image synthesizing apparatus according to claim 8, further comprising: a movement parameter calculating unit that performs a moving parameter calculation. 10. An installation instruction member increase / decrease detection means for detecting an increase / decrease in the number of said installation instruction members arranged on a surface on which said information is displayed, and corresponding to an increase / decrease in the number of said installation instruction members. 9. An audio data storage unit for storing audio data, and a speaker for outputting audio data of the audio data storage unit in accordance with an increase or decrease in the number of the installation instruction members. Or the image synthesizing device according to 9. 11. The pointing means has pointing members of different shapes, the pointing member shape recognizing means for identifying the pointing members of different shapes, and the pointing members of different shapes based on a predetermined combination criterion. 11. The image synthesizing apparatus according to claim 7, further comprising: an instruction member correspondence determining unit that associates the two. 12. The image pickup means is a camera, a zoom magnification detection unit for detecting a zoom magnification of the camera, and an additional image enlargement / reduction unit for enlarging / reducing the additional image based on the detected zoom magnification. The image synthesizing apparatus according to any one of claims 1 to 11, further comprising: 13. A color designation area erasing section for replacing a predetermined color area on a surface on which the information is displayed with a color different from the predetermined color.
The image synthesizing device according to any one of the above. 14. A browser-compatible file creating / updating unit for saving an image synthesized by the image synthesizing unit in a format that can be displayed on an Internet browser. The image synthesizing apparatus according to any one of the above. 15. A predetermined imaging unit, and a plurality of extraction target candidates to be image extraction targets among information and / or a plurality of objects which are imaged by the imaging unit and are displayed on a predetermined surface. A closed curve information detecting means for detecting closed curve information surrounding the plurality of extraction target candidates based on the detected closed curve information and the information of the image of the extraction target candidate. An image synthesizing apparatus, comprising: an individual extracting unit that individually extracts a candidate image; and an image synthesizing unit that performs image synthesizing by using the individually extracted image. 16. Predetermined imaging means, closed-curve information detection means for detecting closed-curve information surrounding a plurality of pieces of pointing target information to be specified from information captured by the imaging means, and the detected closed curve An individual extraction unit for individually extracting an image of each piece of instruction target information from the surrounded plurality of pieces of instruction target information based on information and the image information of the instruction target information; An instruction information detecting unit that detects a region of the detected image and / or a characteristic amount of the image, and adds the instruction target information to the image based on the detected region of the image and / or a characteristic amount of the image. An additional image generating unit that generates an additional image; and an image synthesizing unit that synthesizes the image of the instruction target information obtained by the imaging unit and the generated additional image. Image synthesizing apparatus according to claim. 17. An input display-integrated display, imaging means for capturing at least a part of a display unit of the display, and a plurality of real objects placed on the display.
Closed-curve information detection means for detecting closed-curve information surrounding a plurality of objects to be candidates for image extraction, and the detected closed-curve information and images of the plurality of enclosed objects imaged by the imaging means An individual extraction unit for individually extracting an image of each of the objects from the images of the plurality of enclosed objects based on the information; and displaying the images on the display using the individually extracted images. An additional image generation unit that generates an additional image; a display method selection unit that selects a display method of the generated additional image on a display; and an image of the additional image and the real object based on the selected display method. An image synthesizing apparatus, comprising: 18. An input display-integrated display, a camera for capturing at least a part of a display unit of the display, and a plurality of real objects placed on the display.
Closed-curve information detection means for detecting closed-curve information surrounding a plurality of objects to be candidates for image extraction, and the detected closed-curve information and images of the plurality of enclosed objects imaged by the imaging means An individual extraction unit for individually extracting an image of each of the objects from the images of the plurality of enclosed objects based on the information; and displaying the images on the display using the individually extracted images. An additional image generation unit that generates an additional image; a display method selection unit that selects a display method of the generated additional image on a display; and the additional image displayed on the display based on the selected display method. And a control unit that causes the camera to image a real object placed on the display,
An image synthesizing apparatus comprising: 19. An input display integrated type display, an image pickup means for picking up an image of at least a part of a display section of the display, and an instruction for detecting indicated position information for indicating a real object placed on the display. A position detection unit, based on the detected indicated position information, a real object extraction unit that extracts an image of the real object from an image captured by the imaging unit, using the extracted image, An additional image generation unit that generates an additional image to be displayed on the display, a display method selection unit that selects a display method of the generated additional image on a display, and the additional image based on the selected display method. An image synthesizing device, comprising: image synthesizing means for synthesizing an image of a real object. 20. An image storage unit that stores an image captured by the imaging unit, an outline model storage unit that stores the closed curve information as a candidate outline, and an individual extraction unit that is stored in the outline model storage unit. A contour model deforming unit that deforms the candidate contour according to a predetermined rule; a contour model intersection detecting unit that detects the presence / absence of intersection of the candidate contour; and a contour when the intersection detecting unit determines that there is an intersection. A contour model splitting unit that splits the candidate contour stored in the model storage unit into a plurality of pieces, and an extraction completion determining unit that determines the completion of the contour extraction based on the splitting result. 18. The image synthesizing apparatus according to claim 15, wherein an image surrounded by the outline generated by the division is extracted. 21. A three-dimensional icon registration unit for registering a feature amount and a function of a real object used for input to the display, and a real object used for the input when the input is made on the display. 19. A three-dimensional icon identification unit for identifying a three-dimensional icon according to
Or the image synthesizing device according to 19. 22. A display image brightness calculation unit for calculating brightness of a display image on the display, and a pen input color for changing a pen input color according to the brightness calculated by the display image brightness calculation unit 20. The image synthesizing device according to claim 17, further comprising a changing unit. 23. A display device according to claim 1, further comprising a display image correction unit for changing the brightness and color of a display image according to the brightness and color of a real object placed on said display.
20. The image synthesizing device according to 7, 18 or 19. 24. A medium having recorded thereon a program for causing a computer to execute the functions of all or part of each of the means described in any one of claims 1 to 23. 25. A predetermined position of a surface on which information is represented is indicated by instruction information, the information represented on the surface and the instruction information are imaged, and an instruction is given by the instruction information from the imaging result. Detecting the indicated position corresponding to the indicated position, detecting an area and / or characteristic amount of the information of the indicated object corresponding to the indicated position, and And generating an additional image to be added to the image of the instruction target information, and synthesizing the image of the instruction target information and the generated additional image. 26. The method according to claim 26, wherein when synthesizing the images, a method of synthesizing and / or displaying the additional image is determined based on the designated position, and the synthesis is performed using a result of the determination. Item 30. The image synthesizing method according to Item 25. 27. An image of information and / or a plurality of objects represented on a predetermined surface, and a plurality of images and / or a plurality of objects to be candidates for an image extraction target among the imaged information and / or the plurality of objects. Providing a closed curve surrounding the extraction target candidate, detecting the given closed curve information, and, based on the detected closed curve information and information on the image of the extraction target candidate, images of the plurality of extraction target candidates surrounded A method for individually extracting images of the respective extraction target candidates from the above, and performing image synthesis using the individually extracted images. 28. A medium having recorded thereon a program for causing a computer to execute all or some of the steps according to claim 25.