CN1186991A - Image synthesizing device, image synthesizing method and medium used - Google Patents

Abstract

An image synthesizing device that generates an appropriate additional image based on a pointing position of a pen, etc., pointing area, and feature value of pointing information, and synthesizes and displays it on a display unit. The image synthesizing device includes a camera for photographing a document on which information is recorded, a detection unit for detecting a point indicated by a pen etc. An additional image forming unit that forms an image of the decoration indication information based on the characteristic quantity, an additional image generating unit that stores the formed additional image, a determination unit that determines an information addition method based on the indicated position, and combines the image captured by the camera with the Additional unit for image synthesis.

Description

Image synthesizing device, image synthesizing method and medium used

The present invention relates to an image synthesizing device, an image synthesizing method and a medium used therefor which can be used in an apparatus for forming synthetic images of real objects and hypothetical objects in a computer.

JP Unexamined Publication No. Hei 9-6798 once disclosed an "information processing method and device therefor". The information processing method and device therefor are used as the following system. The actual object is identified, and then the information related to the above-mentioned actual object is synthesized on the image of the camera in a superimposed manner.

If the above-mentioned existing device is placed in a video camera integrated with a monitor, and a painting to which color codes are added is written, the following information providing system can be realized in which information related to the painting The description is displayed on the monitor.

In addition, as a system that assumes that objects interact with actual objects, in "Procceedings of UIST'91, ACM, pp.27-33" and "Communications of the ACM, vol 36, no.7, pp.86-96, 1993 "Digital Desk" is disclosed in .

In the above system, a camera and a projector are installed above the device. Then, based on the information written on the paper (actual object) of the device, the range of the numerical sequence or drawing that the operator intends to extract as an image is designated by a finger or a pen, and the image of the numerical sequence or the like can be automatically extracted. On the other hand, as another drawing image (virtual object) stored in advance by the system, an image of a desktop small computer, etc. is projected on another paper (real object) of the device by a projector. In this way, the image of the desktop computer and the extracted numerical sequence image are synthesized and projected on the same paper. The projected image has a composite image as follows. The above-mentioned numerical sequence is input by operating the keys of the projected desktop computer, and this composite image shows a state as if the above-mentioned numerical sequence is displayed. The above-mentioned system can be used as an image synthesizing device of an image of an actual object and an image of a hypothetical object in the above-mentioned manner.

In addition, the document "Procceedings of CHI'92, ACM, pp.587-592" proposes that an induction plate is provided at a part of the device so that high-precision instructions can also be performed by fingers.

Although the above-mentioned existing information processing method and the device used therein point out that the actual object can be distinguished by color coding, and the information image related to it can be displayed in a superimposed manner, but the above-mentioned device has the following problem, that is, it cannot be viewed from the position specified by the operator. Therefore, compared to real objects such as actual books, the image of the required information is automatically extracted, and the extracted image cannot be processed and displayed together with the image of the hypothetical object.

That is, in the above-mentioned existing system, the operator cannot automatically extract the required information image from the actual object, thereby having the following problem, that is, it is impossible to form a composite formed by combining the actual object and the hypothetical object stored in advance. image.

According to the conventional system, it is not possible to designate basic regions equivalent to the number of objects or information among objects extracted as images by inputting with a finger or a pen, so images of a plurality of objects cannot be individually extracted. Therefore, there arises a problem that, when the number of objects or the number of images to be processed is large, the processing of specifying objects to be extracted as images imposes a large burden on the operator of the apparatus.

In view of the problems of the above-mentioned existing devices, the object of the present invention is to provide an image synthesis device, an image synthesis method and the medium used therefor, as an image using a suspended photographic camera for displaying actual objects and hypothetical problems The synthesis device, for example, in the information recorded in books, can automatically detect the above-mentioned information by indicating the information part that needs to be emphasized with an indicator rod, etc. The information refers to the information area that the operator intends to emphasize, or its intended The emphasized information is a text or a photo, and it can be automatically underlined or screened, etc., which have its information characteristics.

In addition, in view of the above-mentioned problems in the above-mentioned existing systems, another object of the present invention is to provide an image synthesis device, an image synthesis method and the media used, the image synthesis device includes the function of extracting the contour, and this extraction The contour function is to extract the corresponding objects by placing multiple objects to be extracted on the display to provide only one closed curve surrounding all these objects.

In order to achieve the above object, the present invention according to claim 1 relates to the following image synthesizing device, characterized in that the image synthesizing device comprises: a given photographing mechanism; The object is used to detect the position of the indication information given on the surface where the information is displayed; the indication object information detection mechanism is used to detect the indication displayed on the surface corresponding to the indication position detected by the indication position detection mechanism. The area of the object information and/or the characteristic value of the information indicating the object; the additional image forming mechanism is used to form an additional image based on the above-mentioned area and/or characteristic value, and add it to the above-mentioned image indicating the object information; An image synthesis unit, which synthesizes the image of the above-mentioned pointing object information obtained by the above-mentioned shooting unit and the formed additional image.

The present invention according to claim 15 relates to the following image synthesis device, characterized in that the image synthesis device comprises: a given shooting mechanism; a closed curve information detection mechanism for detecting closed curve information, the closed curve Surrounding the following multiple extraction target candidates, they can constitute the target candidates for image extraction of the information and/or inside a plurality of objects photographed by the above-mentioned photographing mechanism and displayed on the specified surface; The detected closed curve information and the image information of the above-mentioned extraction target information candidates are used to extract the corresponding extraction target candidate images from the plurality of extraction target candidate images surrounded by the above-mentioned; The images extracted separately are subjected to image synthesis.

The present invention according to claim 16 relates to the following image synthesis device, characterized in that the image synthesis device includes: a given photographing mechanism; a closed curve information detection mechanism for detecting closed curve information, the closed curve Surrounding the information shot by the photographing mechanism, a plurality of pointing object information that can constitute the pointing object; respectively extracting means for extracting from the above-mentioned surrounded multiple points according to the measured closed curve information and the image information of pointing objects Respectively extract the image of the corresponding pointing object information from the pointing object information image; the pointing information detection unit detects the above-mentioned respectively extracted image region and/or the feature quantity of its image; the additional image forming unit is used for according to the above The measured image area and/or image feature quantity form an additional image, which is added to the image of the above-mentioned pointing object information; the image synthesis mechanism is used to combine the image of the pointing object information obtained by the above-mentioned shooting mechanism Composite with the additional image formed above.

Furthermore, the present invention according to claim 17 relates to the following image synthesizing device, characterized in that the image synthesizing device includes: an integrated display for input display; At least a part of them is photographed; the closed curve information detection mechanism is used to detect the closed curve information, and the closed curve surrounds the following multiple objects, and these objects can constitute the image extraction objects in the multiple actual objects placed on the above-mentioned display Candidates; respectively extracting mechanism for extracting corresponding curves from the images of the above-mentioned surrounding objects according to the above-mentioned measured closed curve information and the image information of the above-mentioned surrounding objects photographed by the shooting mechanism. The image of the object; the additional image forming unit, through the images extracted separately, the additional image is formed and displayed on the above-mentioned display; the display method selection unit is used to select the display method of the above-mentioned additional image formed on the display ; The image synthesis mechanism is used to synthesize the above additional image and the image of the actual object according to the above selected display method.

In addition, the present invention according to claim 18 relates to an image synthesis device, which is characterized in that the image synthesis device includes: an input and display integrated display; a camera for photographing at least a part of the display unit of the above-mentioned display; closed curve information detection A mechanism for detecting closed curve information, the closed curve enclosing the following multiple objects, these objects can constitute image extraction target candidates in a plurality of actual objects placed on the above-mentioned display; The closed curve information obtained, and the image information of the above-mentioned multiple objects surrounded by the camera are taken, and the images of the corresponding objects are respectively extracted from the images of the above-mentioned multiple objects surrounded; the additional image forming unit is used An additional image is formed and displayed on the above-mentioned display through the above-mentioned respectively extracted images; a display method selection unit is used to select a display method of the above-mentioned additional image formed on the display; a control unit is used for displaying according to the above-mentioned selection In the method, the additional image displayed on the above-mentioned display is photographed by the camera on the actual object placed on the above-mentioned display.

The present invention according to claim 19 relates to an image synthesis device, characterized in that the image synthesis device comprises: an integrated display for input display; a photographing mechanism for photographing at least a part of the display portion of the display; and pointing position detection The unit is used to detect the actual object placed on the display indicated by the indicated position information; the actual object extraction mechanism is used to extract the actual object from the image taken by the shooting mechanism according to the indicated indicated position information measured above image; the additional image forming unit is used to form an additional image through the above-mentioned extracted image and is displayed on the above-mentioned display; the display method selection unit is used to select the display method of the above-mentioned additional image formed on the display ; The image synthesis mechanism is used to synthesize the above additional image and the image of the actual object according to the above selected display method.

The present invention according to claim 25 relates to the following image synthesis method, characterized in that the method includes the steps of: using instruction information to indicate a designated position on a surface on which information is displayed; and photographing the information displayed on the above-mentioned surface and the above-mentioned instruction information. ; According to the shooting result, detect the indicated position indicated by the above-mentioned indicating information; detect the area and/or feature value of the information indicating the object displayed on the above-mentioned surface according to the above-mentioned indicating position; according to the above-mentioned indicating object information area and/or feature amount , forming an additional image to be added to the image of the pointing object information; combining the image of the pointing object information with the formed additional image.

In addition, the present invention described in claim 27 relates to the following image synthesis method, which is characterized in that the method includes the steps of: photographing information and/or a plurality of objects displayed on a predetermined surface; The closed curve of the object candidate, the plurality of extracted object candidates constitute the candidates of the above-mentioned shooting information and/or image extraction objects in a plurality of actual objects; detect the closed curve information provided above; extract the closed curve according to the above-mentioned measured closed curve and For the image information of the target candidate, the images of the corresponding extraction target candidates are respectively extracted from the surrounding images of the plurality of extraction target candidates, and image synthesis is performed using the above-mentioned separately extracted images.

Embodiments of the structure of the present invention will be described below with reference to the accompanying drawings.

Fig. 1 is the structural block diagram of the 1st embodiment of the present invention;

Fig. 2 is a flowchart of the action steps of the first embodiment of the present invention;

FIG. 3 is a schematic diagram showing an example of books in the above embodiment;

Fig. 4 is a schematic diagram showing an example of decorating with a frame portrait in the foregoing embodiment;

Fig. 5 is a schematic diagram showing an example of the detection of the indicated position and the indicated area by the pen-shaped indicating mechanism of the above-mentioned embodiment;

Fig. 6 is a schematic diagram showing an example of the decoration of the text area and the photo area of the pen-shaped pointing mechanism of the above-mentioned embodiment;

FIG. 7 is a schematic diagram showing an example of a conventional decorative area display transformation method for the pen-shaped pointing mechanism of the above-mentioned embodiment;

Fig. 8 is a structural block diagram of the second embodiment of the present invention;

Fig. 9 is a flowchart of the action steps of the second embodiment of the present invention;

Fig. 10 is a schematic diagram showing an example of a plurality of markers and an installation method thereof provided on the pen-type indicating mechanism of the above-mentioned embodiment;

Fig. 11 is a schematic diagram showing an example of decorating with mark type identification in the above embodiment;

Fig. 12 is a structural block diagram of the third embodiment of the present invention;

Fig. 13 is a flowchart of the action steps of the third embodiment of the present invention;

Fig. 14 is a schematic diagram of an example of an existing decorative portrait display method in the above embodiment;

Fig. 15 is a schematic diagram of an example of an existing decorative portrait selection method in the above-mentioned embodiment;

Fig. 16 is a schematic diagram of an example of an existing decorative portrait copying method in the foregoing embodiment;

Fig. 17 is a structural block diagram of the fourth embodiment of the present invention;

Fig. 18 is a flowchart of the action steps of the fourth embodiment of the present invention;

Fig. 19 is a schematic diagram of an example of estimating book movement parameters by means of template comparison in the above embodiment;

Fig. 20 is a structural block diagram of the fifth embodiment of the present invention;

Fig. 21 is a flowchart of the action steps of the fifth embodiment of the present invention;

Fig. 22 is a schematic diagram of an example of decoration with setting marks in the above embodiment;

Fig. 23 is a schematic diagram of an example of canceling a decorative portrait by setting and removing a mark in the above embodiment;

Fig. 24 is a structural block diagram of the sixth embodiment of the present invention;

Fig. 25 is a flowchart of the action steps of the sixth embodiment of the present invention;

Fig. 26 is a schematic diagram of an example of the detection indication information area where the inclination of books is corrected by setting the indication mechanism in the above embodiment;

Fig. 27 is a structural block diagram of the seventh embodiment of the present invention;

Fig. 28 is a flowchart of the action steps of the seventh embodiment of the present invention;

Fig. 29 is a structural block diagram of the eighth embodiment of the present invention;

Fig. 30 is a flowchart of the action steps of the eighth embodiment of the present invention;

Fig. 31 is a schematic diagram of an example of a method for indicating a decorative area by a hook mark in the above embodiment;

Fig. 32 is the schematic diagram of the example of the hook mark in the above-mentioned embodiment;

Fig. 33 is the schematic diagram of the decoration result example that carries out with the hook mark in above-mentioned embodiment;

Fig. 34 is a structural block diagram of the ninth embodiment of the present invention;

Fig. 35 is a flowchart of the action steps of the ninth embodiment of the present invention;

Fig. 36 is a block diagram of the tenth embodiment of the present invention;

Fig. 37 is a block diagram of the eleventh embodiment of the present invention;

Fig. 38 is a structural block diagram of the twelfth embodiment of the present invention;

Fig. 39 is a flowchart of the action steps of the twelfth embodiment of the present invention;

Fig. 40 is a schematic diagram of an example of setting an actual object on the display in the above embodiment;

Fig. 41 is a schematic diagram of an example of selecting an actual object by pen input in the above embodiment;

Fig. 42 is a schematic diagram of an example of the result of taking out the actual object in the above-mentioned embodiment;

Figure 43 is a schematic diagram of an example of decorating an actual object in the above-mentioned embodiment;

Fig. 44 is a schematic diagram of an example of recording information by means of pen input in the above-mentioned embodiment;

Fig. 45 is a block diagram of the specific structure of the extracted actual object part 30 in the above-mentioned embodiment;

Figure 46 is a flow chart of the steps of extracting the actual object in the above embodiment;

Fig. 47 is a schematic diagram of an example of contour model intersection in the above embodiment;

Fig. 48 is a structural block diagram of the thirteenth embodiment of the present invention;

Fig. 49 is a structural block diagram of the fourteenth embodiment of the present invention;

Fig. 50 is a block diagram showing the structure of a fifteenth embodiment of the present invention.

first embodiment

FIG. 1 is a structural diagram of a first embodiment of an image synthesis device according to the present invention.

Reference numeral 1 in Fig. 1 represents the camera that takes pictures of the books on which information is recorded, preferably a color CCD camera, and reference numeral 2 represents the pointing position detection unit that detects the pointing position of the pen 100 on the books by the camera 1. Reference numeral 3 denotes an indication information detection unit for detecting the indication information area and the feature quantity recorded at the above indicated position. Reference numeral 4 denotes an additional image forming unit which forms an additional image corresponding to the detected characteristic amount of the index information. Reference numeral 5 denotes an additional image storage unit for storing an additional image formed by the additional image forming section 5 . Reference numeral 6 denotes a pointing position other information adding method determining unit, which can determine a composite display method of the above-mentioned additional image based on the above-mentioned detected pointing position. Reference numeral 7 represents an image synthesis unit that synthesizes the image in the camera 1 with the above-mentioned additional image, and reference numeral 8 represents a display unit for displaying the image synthesized by the image synthesis unit 7, preferably a projector or a display.

In addition, it is preferable that the pen 100 is provided with an indicator mark 100a having predetermined RGB values and a pointing position detection start switch 100b, which are easy to detect the pointing position. Reference numeral 1000 represents a control unit composed of a memory such as a general ROM or RAM and a CPU, connected to the parts indicated by the above reference numerals 1 to 8, and controls the operation of each part and the data exchange between each part. Here, components such as the pen 100 and the marker 100a correspond to the pointing means of the present invention, and the pointing information detecting unit 3 corresponds to the pointing object information detecting means of the present invention.

The action of this embodiment will be described below, and an embodiment of the image synthesizing method of the present invention will be described.

Fig. 2 is a flow chart of the working process of the image synthesis device of this embodiment.

In order to illustrate the operation of this embodiment in an easy-to-understand manner, an example in which the present invention is provided in an overhead camera is described here. Pendant cameras are typically used for lectures or presentations at universities. As shown in FIG. 3 , at this time, words are written on paper or books, or photos or actual three-dimensional objects are presented. This case corresponds to the case where the image captured by the camera 1 is displayed as it is on the display unit 8 such as a monitor.

In the manner described above, this embodiment can be used as a device for assisting handouts, etc., where you want to emphasize the position in the handout, etc., based on the pointing object information with the pointing position or pointing object with the pen ("pointing object information" in this manual) Various additional images (hypothetical objects) are formed in areas or features, and the additional images are combined with images in the camera to decorate books and the like.

According to the above-described embodiment, in normal use of writing black characters on white paper, the pen 100 indicates a part of the writing part to be emphasized, and detects the area of the writing part. The following embodiments are described below, and while the control of the additional image corresponding to the pointing position of the pen 100 is described, the action steps of this embodiment are described. This embodiment specifically refers to the above-mentioned writing part Detect the information characteristic quantity of the information, judge the information characteristics of the entry part whether the entry part is a text or a photo, and carry out decoration suitable for the feature in the image of the entry part.

Step 1a image acquisition:

The books and the indicator rod 100 shown in FIG. 3 are photographed by a camera as an image l(x, y).

Step 2a instructs position detection to start state detection of the switch:

An image obtained by step 8a to be described later is displayed on the display unit 8 when the pointing position detection start switch 100b is turned off. When the above-mentioned switch 100b is turned on, step 3a is performed.

Step 3a Detection of the indicated position of the pen marker 100a:

From the image l(x, y), the pointing position detecting unit 2 detects, as a pointing position, an area having a predetermined size and a predetermined color (RGB values in a predetermined range that the mark 100a has).

For example, as far as each pixel l (x, y) is concerned, it is detected in order whether it meets the specified range of RGB values representing the color of the mark (in the case of a reddish mark, such as 180<R<255 and 120<G< 140 and 60<B<120), when the color of the mark is selected, a pixel having an RGB value representing a predetermined range of the mark color is extracted from an area of a predetermined size centered on the pixel. Calculate the circumscribed rectangle including all the extracted pixels, and if the rectangle has a predetermined size, it is judged as a mark, and the indicated position is at the center (mx, my) of the mark. In case no label is detected, proceed to step 7a.

Step 4a indicates the determination of the information addition method of the location:

Here, it is judged whether to indicate near the end of the image. In the case where the indicated position is at the end of the image (book), for example, when the image size is length×width: 480×640, and the center of the indicator (mx, my) When any relationship among mx<30, my<30, 610<mx, 450<my is satisfied, the image is selected in step 4a-1.

Step 4a-1 selection of frame image:

The additional image storage unit 5 stores at least one frame image, which includes the position of the person who demonstrates or the time of the speech. When the end of the instruction book is shown, as shown in Figure 4, the frame image is superimposed by the image synthesis unit 7 on the image of the camera 1 and display it on the display unit 8 . When there are multiple frame images, a different image is selected each time the end of the book is indicated. Thus, the position, time, or frame provided can be displayed when forming a normal paper sheet (OHP sheet), and at the same time, the above-mentioned content can be provided at the above-mentioned position on the paper that is usually not written.

Step 5a indicates the determination of the information addition method of the location:

Here, it is judged whether the undecorated information indicates, and if there is an indication, the indication information detection unit 3 detects whether there are black pixels near the center of the indicator. In the case of black pixels, store information such as characters, calculate its feature value through step 5a-1, and form and store a decorative image corresponding to the feature value through step 5a-2.

Step 5a-1 indicates the calculation of object information area and feature quantity:

First, the pointing object information area is calculated by the pointing information detection unit 3 . Markers are set in black pixels located within a prescribed range relative to the center (mx, my) of the pointer. A flag is set in a recursive manner in each black pixel having the above flag. If there is no black pixel with a mark, setting of the mark is terminated.

As shown in Figure 5, for example, in the case of displaying a character string in a manner parallel to the x-axis, if the corresponding minimum and maximum values in the x-coordinates and y-coordinates of the marked pixel are set to xl, xr, yr, then The rectangle (xl, yl)-(xr, yr) refers to the object information area.

The following calculation is used to distinguish whether it belongs to a character string or a feature quantity belonging to a photo, such as the proportion of white (RGB value in a given range) pixels in all pixels in the indicated area. In the case where the indicated area is a photograph, there are fewer white pixels. On the other hand, in the case of a character string, since black pixels and white pixels are displayed alternately, the proportion of white pixels is larger than that of the photo area. Therefore, if the ratio occupied by white pixels is smaller than a predetermined limit value, it is determined to be a photo area, and if it is greater than the above-mentioned limit value, it is determined to be a character string area.

Step 5a-2 Formation and storage of decorative images:

The additional image for decoration is formed by the additional image forming unit 4 based on the region and its feature value calculated in step 5a-1.

For example, since it is not easy to see clearly by emphasizing the photo area in the way of adding texture, the border is formed on the photo area, and the character string is emphasized in the way of shading. A shaded image is formed by replacing RGB values of pixels other than black with RGB values of a predetermined color in the indicated area detected in step 5a-1. FIG. 6 shows the decoration results for the case of forming a photo when the ratio of white pixels in the indication area is less than 15%, and for the case of forming a character string when the above ratio is greater than 15%. Additionally, the decorated area is stored in the existing decorated area.

Step 6a indicates the determination of the information addition method of the location:

It is judged whether the pointing position of the pen 100 is located in the existing decoration area stored in step 5a-2, if it is judged to point to the existing decoration area, then by step 7a, the display method of decoration is changed according to its position. Otherwise, proceed to step 1a.

Step 7a Changes to the display method of the existing decoration area:

As shown in FIG. 7 , another information addition method determining unit 6 indicating a position changes the display method in such a manner that the existing decoration area 700 is divided into a plurality of areas (in FIG. ), for example, if the front part 701 is indicated, its decoration will be eliminated; if the rear part 702 is indicated, its decoration will blink, and it will be synthesized with the image in the camera 1 and displayed on the display unit 8 .

In addition, in the case where the mark 100a is not detected in the step 3a as described above, the process proceeds to the step 7a, and at this time, the display of all additional images is stopped. Thus, all additional images can be temporarily erased by pressing the switch 100b of the pen 100 out of the field of view of the camera before starting to write on a different paper again.

Synthesis and display of step 8a image:

The image synthesizing unit 7 synthesizes the image in the camera 1 and the decorative image formed by the additional image storage unit 5 by, for example, superimposing, and displays it on the display unit 8 .

The situation described above is to implement the first embodiment according to the sequence described in step 1a to step 8a, then the person performing the demonstration uses the indicator rod to indicate the part to be emphasized in the information written by the person, and automatically extracts the characteristics of this part, and then Superimpose suitable decorative images.

Therefore, there is no need to switch pens to use each color separately. In addition, because the decoration method can be easily changed according to the pointing position of the pen, if the point is near the end of a book, for example, the border, job title, time, etc. are automatically provided, or in the decorated information part, it is removed by the pointing position Decoration, or making the decoration flash, like this, even if the material for presentation is not preformed, still can carry out effective presentation.

In addition, in the above-mentioned steps, an additional image is formed based on the pointing object area or feature quantity at the pointing position of the pen, etc., but it is also possible to synthesize and display only the following images by superimposition or the like. An image such as an "arrow" is stored in advance at a point indicated by a pen or the like. The above-mentioned method is effective for situations such as placing a textbook on a suspended camera during an English lecture to indicate the accent position of an English word, or indicating the material at hand or during a videoconference or computer conference. Models and other places that should be emphasized separately or parts to be explained at present.

In the above steps, although the image l(x, y) is usually digitized as an example, digitization may be performed only when the pointing position detection switch 100b is turned on.

2nd embodiment

Next, a second embodiment of the image synthesizing device of the present invention will be described.

In the first embodiment, there is only one method of decorating character strings and photos, but the second embodiment is conceived in such a way that multiple decoration methods can be used more easily.

FIG. 8 is a structural diagram of the second embodiment. In this embodiment, the same structural parts as those of the first embodiment shown in FIG. 1 are given the same reference numerals, so their detailed descriptions are omitted.

The main difference between the second embodiment and the first embodiment is that the marking type and pointing position judging unit 9 is used to replace the pointing position detection unit 2 for identifying the types of multiple marks provided to the pen, etc., and different marks are used to correspond to the attached drawings The image forming unit 10 replaces the additional image forming unit 4, and is used to form an additional image that changes depending on the type of pointer. Here, the mark type and pointing position judging unit 9 corresponds to the pointing position detecting means of the present invention, and the different mark corresponding additional image forming unit 10 corresponds to the additional image forming means of the present invention.

Next, the operation of the image synthesizing apparatus of the second embodiment constructed as above will be described with reference to the flow chart of FIG. 9 . However, since new steps are added to the flow chart of Fig. 2 and Fig. 9 showing the operation of the first embodiment, only the different steps 3b, 5b and 7b will be described here.

In this embodiment, the following example will be described. This example is constructed in such a manner that, as shown in FIG. It is possible to easily use various decoration methods for image recognition whether one of the four kinds of marks 100a-1 to 100a-4 indicates whether a book is to be used.

Judgment of the color of the mark in step 3b:

The color of the pointer facing the camera 1 provided to the pen 100 is judged by the marking type and pointing position judging unit 9 .

Specifically, in the image l(x, y), the number of pixels is calculated for each specified color (in this embodiment, including the red and cyan colors of the mark shown in Figure 10), and there will be The color of the pixel is identified as the color of the pen marking towards the camera 1. For example, the number of red pixels and the number of cyan pixels are detected here, and the numbers of these pixels are compared. Thus, it is possible to more flexibly determine that there is no error, because even if some pens 100 are detected to have two colors at the same time, the color with a larger number of pixels is still considered to be determined as the pointer for pointing. color.

Judgment of indicator type in step 5b:

The mark type designation position judgment unit 9 judges whether it is a mark for shading (100a-1, 100a-2) or a mark for underlining (100a-3, 100a-4).

Specifically, the number of pixels of the specified color of the marking area on the pen 100 measured in step 4b is calculated. Since the underlined mark has fewer pixels of a predetermined color than the underlined mark, it can be determined which mark it belongs to based on a predetermined limit value. In addition, the number of pixels of the predetermined color included in the marking area is the number of pixels of the predetermined color determined in step 3b.

Step 7a2 forms the decorative image according to the type of the index mark:

The different marks correspond to the mark types determined by the additional image forming unit 10 in step 5b, and the feature values obtained in step 7b1 to form a decorative image. For example, FIG. 11 shows an example of a decorative image of red shading (mark 100a-1 in FIG. 11) and cyan underline (mark 100a-4 in FIG. 11) in the case where the indicated object information is a character string. . Also, in the case where the pointing target information is a photograph, only the color of the border is changed.

According to the structure of the above-mentioned second embodiment, since the decoration method can be changed by recognizing various marks provided to the pen 100, the time for writing can be saved by changing pens of various colors, so that the presentation can be carried out smoothly. .

3rd embodiment

Next, a third embodiment of the image synthesizing device of the present invention will be described.

The third embodiment is conceived in such a way that compared with the first and second embodiments, the existing decorative image formed is reused by duplication or the like.

Fig. 12 is a structural diagram of the third embodiment. In this embodiment, the same structural parts as those of the first embodiment shown in FIG. 1 are given the same reference numerals, so their detailed descriptions are omitted.

The main difference between the 3rd embodiment and the 1st embodiment is: the existing additional image reproduction unit 11 that is also added, it will be stored in the existing decorative image list in the additional image storage unit 5 and the shadow of the camera 1. The images are synthesized and displayed, selected by the pen 100, etc., and then synthesized and displayed at another indicated position on the book.

Next, the operation of the image synthesizing apparatus of the third embodiment constructed as above will be described with reference to the flow chart of FIG. 13. FIG.

As shown in FIG. 13, since the operation process of the third embodiment is to insert step 1c to step 3c between step 3a and step 4a in the operation program of the first embodiment, only the inserted part will be described here.

Display of the list of existing decoration images in step 1c:

At first, if so far, for example, an image for decoration is formed by the additional image forming unit 4, as shown in the upper left part of FIG. In the memory.

Next, if the indication position measured in the same manner as step 3a of the first embodiment is located in this area, then as shown in Figure 14, the "3. The image of "actual object" and the underlined image of "4. Prototype system function" are displayed in the image of camera 1.

Step 2c Selection of Existing Decoration Image:

As shown in FIG. 15, among the existing decorative images displayed in a list in step 1c, the pointing position of the pen 100 is detected in the same manner as step 3a in the first embodiment, and the current pointing position at the pointing position is selected. There are images.

Step 3c Copy of existing decoration image:

As shown in FIG. 16, the image selected in step 2c is superimposed on the pointing position of the pen 100 detected in the same manner as in step 3a in the first embodiment.

By above-mentioned steps, then owing to implementing the 3rd embodiment can display existing decoration image in the form of list, and it is copied with the mode of indicating by pen etc., even in the situation of using many books like this, still need not Write the same string to any extent in the book class, so that the presentation can be performed more smoothly.

4th embodiment

Next, a fourth embodiment of the image synthesizing device of the present invention will be described.

Embodiments 1 to 3 are described for the case where the books are kept fixed relative to the camera, but the fourth embodiment is based on the case where the books move relative to their original positions even if writing is written in the books. Decorative images are superimposed in the correct position as the book class is moved.

Fig. 17 is a structural diagram of the fourth embodiment. In this embodiment, the same structural parts as those of the first embodiment shown in Fig. 1 use the same reference numerals, so their detailed descriptions are omitted here.

The main difference between the fourth embodiment and the first embodiment is that an additional movement parameter estimation unit 12 is used to estimate the movement position of books, and an additional image deformation unit 13 transforms the existing decorative image by the estimated movement parameters. Like deformation.

Next, referring to the flow chart of FIG. 18, the operation process of the image synthesizing apparatus of the fourth embodiment constructed as above will be described. The operation of the fourth embodiment is performed by inserting the following steps 1d to 4d between the steps 1a and 2a of the operation of the first embodiment.

Step 1d with or without an existing trim area:

When referring to the additional image forming unit 5, if there is an existing decoration image, the movement parameter estimation unit 12 proceeds to step 2d, and if there is no existing decoration image, the estimation unit 12 proceeds to step 2a.

Step 2d selects a partial image from an existing decoration image

In order to realize superimposition between two temporally continuous images from any existing decoration image, the small area shown in Fig. 19 is selected as a template.

In addition, the above-mentioned two consecutive images in time are images before and after the movement of the book, which are captured by the camera 1 .

Comparison of step 3d partial images:

Simultaneously with template matching between two temporally consecutive images, the motion parameters are calculated. Since the movement of books in this case is mainly parallel movement (xd, yd) and rotational movement θ, for example, when the template is 0<θ<360°, (xd, yd)∈R (the specified search range R: refer to Figure 19 ) between the movement, and the sum of the squares of the differences in brightness values is the smallest θ, (xd, yd) as the moving parameter.

Deformation of step 4d decoration image:

All the images stored in the additional image forming unit 5 are deformed by means of the movement parameters calculated in step 3d.

According to the above-mentioned fourth embodiment, by performing template comparison between consecutive borders of a part of the decoration image, calculating the movement parameters of the book and deforming the decoration image, even if the book moves during the presentation, it can still be displayed. Decorate the emphasized part correctly.

fifth embodiment

Next, a fifth embodiment of the image synthesizing device of the present invention will be described.

In the first embodiment, the highlighted part of each place is designated by a pen or the like to delete its decoration, which is the same as the case of decoration, indicating this part. The idea of the fifth embodiment is to emphasize or delete multiple points in a simpler and more intuitive manner.

FIG. 20 is a structural diagram of the fifth embodiment. In this embodiment, the same structural parts as those of the first embodiment shown in FIG. 1 are given the same reference numerals, so their detailed descriptions are omitted here.

The main difference between the fifth embodiment and the first embodiment is that, in addition to the pen-shaped indicating mechanism such as the pen 100, the setting indicating mechanism 200 is also used to indicate the part to be decorated by placing it on the book; A plurality of different indicating mechanism position detection units 14 replace the indicating position detecting unit 2, and are used to detect a plurality of different indicating mechanisms; the following multiple different indicating mechanism corresponding indicating information detecting units 15 are used to replace the indicating position detecting unit 2, The detection unit 15 is used to detect the information area or the feature quantity indicated by the indicating mechanism according to the type of the indicating mechanism; the corresponding additional image forming unit 16 of a plurality of different indicating mechanisms replaces the additional image forming unit 4, for according to the indicating mechanism Types form different additional images. Here, the pointing mechanism of the present invention includes a pen-type pointing mechanism such as the pen 100 and the setting pointing mechanism 200 . In addition, the pen-shaped pointing mechanism such as the pen 100 or the setting pointing mechanism 200 corresponds to the pointing member of the present invention, respectively. In addition, a plurality of pointing mechanism pointing position detection units 14 correspond to the pointing position detection mechanism of the present invention, and a plurality of different pointing mechanisms corresponding pointing information detection units 15 correspond to the pointing object information detection mechanism of the present invention. Also, a plurality of different indicating means corresponding to the additional image forming unit 16 corresponds to the additional image forming means of the present invention.

Next, the operation of the image synthesizing apparatus of the fifth embodiment constructed as above will be described with reference to the flowchart of FIG. 21. FIG. Here, only step 2e, step 2e1, step 2e2, and step 8e, which are different from the operations of the first embodiment, will be described. Here, the setting pointing mechanism 200 will be described by using an example using a yellow mark different from the pointer having the pen 100 described above in the first and second embodiments. As shown in FIG. 22, the setting indicating mechanism will be described in the same manner as in the second embodiment. This example uses two types of indicators for shading 200a and underlining 200b. Here, the two kinds of marks 200a, 200b can be realized simply by printing, for example, on the surface of a circular object such as a coin or a button formed of glass or plastic, as shown in FIG. 22 . different forms. In addition, the above-mentioned indicator can also be simply formed from a layered member whose inner surface constitutes an adhesive surface and whose outer surface is printed with a pattern different from that shown in FIG. 22 . In addition, the above-mentioned layered member may be pasted on the plain surface of the above-mentioned button-shaped object.

Step 2e sets up the detection with the indicating mechanism:

From the image l(x, y) obtained in the step 1a of the first embodiment by a plurality of different pointing mechanism position detection units 14, in the same manner as in the step 3a, there will be marked colors (within the specified range Among the RGB values, here, for example, yellow (180<R<255, and 180<G<255, and 60<B<120) an area of a predetermined size is detected as a mark for setting. However, since a plurality of installation indicators are generally provided, all regions of a predetermined size having a predetermined color are extracted.

The upper left (xl(i), yl(i)) of the circumscribed rectangle, and the vertex group (xl(i), yl(i)) of the lower right (xr(i), yr(i))-(xr( i), yr(i)) stores the position of the marking for setting. However, the reference j indicates the number of markers. Also, depending on the number of pixels of a predetermined color included in the area, two types of marks 200a, 200b for shading and for underline can be identified.

Step 2e-1 calculates the indication area and feature quantity with the indication mechanism according to the setting:

As shown in Figure 22, by a plurality of different pointing mechanism corresponding instruction information detecting unit 16, search for the rectangular area (xr(i), yl(i))-(image width, yr(i) adjacent to the detected mark position )) has the largest x-coordinate of the black pixel, and this x-coordinate is xmax(i). At this time, the search is performed for each y-coordinate along the x-axis direction, but if black with a predetermined length is not detected, the search is terminated. As a ^result , even if information is formed at a trace that deviates from book stains or a desired inscribed portion, the information will not be taken by mistake. Here, the rectangle (xr(i), yl(i))-(xmax(i), yr(i) constitutes the indicated area.

Next, in the same manner as step 5a of the first embodiment, it is judged whether the indicated area is a character or a photograph.

Step 2e-2 Formation of additional images for setting pointing mechanisms:

In the same manner as in step 5a-2 of the first embodiment, an additional image for shading or underlining is formed based on the region and feature value calculated in step 1e-1.

Step 8e:

Since the image l(x, y) is acquired at predetermined time intervals in step 1e, the image synthesis unit 7 can update the image display on the display unit 8. At this time, an additional image is synthesized again corresponding to the position of the setting mark. Therefore, if the marking for setting is removed as shown in FIG. 23, since the additional image is not synthesized again, the decoration is cleared.

As described above, the fifth embodiment is implemented. Since the setting indicator is detected at predetermined time intervals, it is possible to provide an intuitive and easy-to-understand operation method of decorating when setting a mark and deleting the decoration when removing a mark.

sixth embodiment

Next, a sixth embodiment of the image synthesizing device of the present invention will be described.

The 4th embodiment is the situation described, even if the book is shifted and moved relative to its original position when writing characters etc. in the book, the existing indication area of the pen-shaped indicating mechanism is still in the Stacks in the correct position according to the way books move.

On the other hand, the idea of the sixth embodiment is that the books are moved when the pointing mechanism for setting used in the fifth embodiment is used.

Fig. 24 is a structural diagram of the sixth embodiment. In this embodiment, the same structural parts as those of the fourth embodiment shown in Fig. 17 and the fifth embodiment shown in Fig. 20 use the same symbols, so their specific details are omitted here. illustrate.

The main difference between the sixth embodiment and the fourth and fifth embodiments is that the movement parameter estimation unit 12 is replaced by the movement parameter estimating unit 17 of the following marking for setting. degrees to calculate the rotational movement in the book movement parameter, that is, the inclination angle. Thus, if the instruction information of the stylus-type pointing mechanism and the instruction information of the marking for setting are written in parallel, the partial image for demonstration is rotated with the inclination of the indicating information of the marking for setting as the inclination of the book, From this rotation, the parameters of the parallel movement of the book can be calculated. Here, the movement parameter estimating unit 17 for setting flags corresponds to the movement parameter calculation unit of the present invention.

Next, the operation of the image synthesizing apparatus of the sixth embodiment constructed as above will be described with reference to the flow chart of FIG. 25. FIG. The operation of the sixth embodiment is to insert the following steps 1f to 6f between the steps 1e and 3e operated in the fifth embodiment.

Step 1f calculates the inclination angle of the indication information of the indicator for setting:

As shown in Figure 26, according to the corresponding indication information detection unit 15 of a plurality of indicating mechanisms, firstly, in the rectangular area (xr(i), yl(i))-(image width, yr( In i)), find the black pixel with the largest x-coordinate corresponding to each y-coordinate value y(k), and use the x-coordinate as x-max(k). At this time, in the process of searching for each y-coordinate along the x-axis direction, if black within a predetermined length is not detected, the search is terminated. Thus, according to the embodiment shown in FIG. 26, candidate points located only in the upper part of the inclined instruction information can be detected. Next, the least squares approximate straight line A is obtained for the upper candidate points of the written part. The included angle between the straight line and the x-axis in the camera coordinate system is the inclination angle θ of the indication information. In addition, y(k) does not have to be increased by one, and the number of upper candidate points of the write-in part may be two or more.

Step 2f sets the detection of the indicated area with the tilt correction of the pointer:

Since the angle between the minimum 2-multiplied straight line A calculated in step 1f and the x-axis in the camera coordinate system is the inclination angle of the entered part, the straight lines in the following states constitute straight lines B and B' respectively, and this state refers to the straight line A Move in parallel along the direction away from the origin and the direction close to the origin, and there is no black pixel on the straight line A. Find the black pixel with maximum x-coordinate x-max(i) in the area surrounded by straight line B, B' and straight line x=xr(i), x=(image width). When the inclination angle θ of the straight line A is a positive value, the following areas constitute the indication information area, that is, the intersection B1 of the straight line B and x=xr(i), and the vertical foot coordinate position B2 of the vertical line from the intersection B1 to the straight line B' , x=x-max(i) and the intersection point B3 of the straight line B, and the vertical foot coordinate position B4 of the vertical line from the intersection point B3 to the straight line B'. Here, with the camera coordinate system (x-y coordinate system) of the camera 1 as a reference, as shown in FIG. 26 , the inclination angle θ generated in the clockwise direction is a positive value.

Step 3f Presence of Existing Decorative Areas for Pen Type Indicating Mechanisms:

Refer to the additional image forming unit 5 to set the indicator using the movement parameter estimation unit 17. When there is an existing decorative image formed by the pen-shaped pointing mechanism, go to step 4f. If there is no above-mentioned image, go to step 3e.

Step 4f partial image is deformed according to the inclination angle:

Since the existing decorative image is deformed with the movement of the book, the setting pointer uses the moving parameter estimation unit 17 to select the partial image for template comparison described in step 2d in the fourth embodiment, and the image is passed through the step. The inclination angle rotation deformation of books calculated by 1f.

The parallel movement parameter calculation of step 5f partial image comparison:

In the above-mentioned manner, step 4f is used to form a template, which is to compare two consecutive images in time to calculate the movement parameters. Since the template performs rotational movement according to the book rotation parameter θ, the calculated movement parameters only constitute parallel movement (xd, yd).

Deformation of step 6f decoration image:

All images in the additional image forming section 5 are deformed by the movement parameters calculated in step 5f.

If the sixth embodiment is implemented as described above, in the case of setting marks, even if the books move, the inclination of the indicated area can still be detected and the area can be moved. The movement parameters of the decorative image can still be calculated, and the existing decorative image can be deformed and synthesized. In this way, even if the book moves during the demonstration using the setting mark, the part to be emphasized can still be correctly decorated.

Seventh embodiment

Next, a seventh embodiment of the image synthesizing apparatus of the present invention will be described.

In the fifth embodiment, the decoration that can be marked simply by setting or removing the mark for setting has been described. The idea of the seventh embodiment is to introduce decorations and effect sounds for setting marks, and to easily perform operation confirmation, thereby further improving the effect of the demonstration.

Fig. 27 is a structural diagram of the seventh embodiment. In this embodiment, the same structural parts as those in the fifth embodiment shown in Fig. 20 are given the same reference numerals, so their detailed descriptions are omitted here.

The main difference between the seventh embodiment and the fifth embodiment is that the increase/decrease detection unit 18 of the number of added markers selects predetermined sound data according to the increase or decrease of the number of installation marks detected at predetermined time intervals. In addition, reference numeral 19 denotes a sound data storage unit for storing sound data, and reference numeral 20 denotes a speaker for reproducing selected sound data. Here, the marker quantity increase/decrease detection unit 18 corresponds to the installation marker increase/decrease detection mechanism of the present invention.

Next, the operation of the image synthesizing apparatus of the seventh embodiment constructed as above will be described with reference to the flowchart of FIG. 28. FIG.

In the operation of the seventh embodiment, the following steps 1g to 4g are inserted between the steps 8e and 9e in the operation of the fifth embodiment.

Step 1g Comparison of the number of indicators used for setting between borders:

By the number of marker increase/decrease detection unit 18, the number of the detected setting markers is compared between the borders, if it is increased, step 2g is performed, if it is decreased, step 3g is performed, and if there is no change , go to step 9e.

Step 2g adds the option to use voice data:

Select voice data for addition, and proceed to step 4g.

Step 3g reduces the selection with sound data:

Select the audio data for reduction, and proceed to step 4g.

Output of step 4g effect sound:

The control unit 1000 outputs the audio data selected in step 2g or step 3g through the speaker 20 .

Through the above steps, when a new indicator is installed on a book or the above-mentioned indicator is removed, only the change in the number of indicators for installation is detected, and the effect sound can be output while decorating the instruction information.

According to the seventh embodiment, since the addition or removal of pointers to or from books can be recognized by the increase or decrease in the number of pointers, and an effect sound can be added according to the recognition situation, it is not only easy to carry out Confirmation of operation, but also efficient demonstration can be further carried out.

Eighth embodiment

Next, an eighth embodiment of the image synthesizing device of the present invention will be described.

Embodiments 1 to 7 illustrate the detection and decoration of the instruction portion with respect to relatively simple books consisting of item-by-item notes or photographs. The idea of the eighth embodiment is that by identifying the shapes and corresponding relationships of multiple marks, books with both pictures and texts can be decorated.

Fig. 29 is a structural diagram of the eighth embodiment. In this embodiment, the same structural parts as those in the fifth embodiment shown in Fig. 20 are given the same reference numerals, so their detailed descriptions are omitted here.

The main difference between the eighth embodiment and the fifth embodiment is that: a mark shape recognition unit 21 and a mark corresponding determination unit 22 are added, the mark shape recognition unit 21 is used to identify two or more mark shapes, and the marks are determined accordingly The unit 22 determines the corresponding relationship of the marks of the specified combination according to the shape of the mark recognized by the mark shape recognition unit 21 . Here, the mark shape recognizing unit 21 corresponds to the mark shape recognizing means of the present invention, and the mark correspondence determining unit 22 corresponds to the mark correspondence determining means of the present invention.

Next, the operation of the image synthesizing apparatus of the eighth embodiment constructed as above will be described with reference to the flow chart of FIG. 30. FIG. In order to describe the working process of this embodiment more clearly, the following examples illustrate that in books with a mixture of character strings and graphics as shown in Figure 31, two kinds of hook-shaped mark marks are used to decorate the area corresponding to the hook shape seen. . It is also possible if the mark is pre-marked in the place mentioned in the textbook such as after the school teacher conducts a lecture preview. Marking instructions can also be pasted on such an article in a layered form, and can also be drawn on books with a highlighter or the like.

Next, the operation procedure of the eighth embodiment will be described. The operation of the eighth embodiment is to insert the following steps 1h to 5h between the steps 1e and 2e of the operation of the fifth embodiment.

The detection of step 1h hook mark:

By a plurality of different mark position detection units 14, a region of a predetermined size from the image is detected as a mark, and the region has a predetermined color (RGB value of a predetermined range) representing a hook mark.

Step 2h identification of marked shape:

The type of the detected mark is recognized and recognized by the mark shape recognition unit 21 . Here, the start mark and end mark shown in FIG. 32 are recognized. Specifically, according to the method shown in Figure 32, the detected mark area is divided into four equal parts, and the number of pixels with the specified color (RGB value) representing the mark is calculated in areas 0-3. If there are more of them, it is judged as a start mark, otherwise, it is judged as an end mark.

Step 3h detects the recorded part according to the tag correspondence:

Which area to designate is determined by the marker corresponding determination unit 22 depending on whether it is a start marker or an end marker. First, as far as all start tags are concerned, their distances to all end tags located right below them are calculated. Afterwards, a corresponding relationship is established between the start marker and the marker whose distance is the smallest.

Step 4h indicates the detection of the feature quantity of information:

The indication information detecting unit 15 corresponding to a plurality of different marks calculates, as a feature quantity of the indication information, for example, the proportion occupied by white pixels in the indication area. Since the proportion of white pixels in the general indication area satisfies the following relationship, namely (image)<(character string)<(photograph), these feature quantities can be identified according to suitable limit values.

Step 5h is credited to the decoration of the section:

Corresponding additional image forming unit 16 by a plurality of different indication mechanisms, in the indication information area detected by step 3a, replace the RGB value of the specified color with the RGB value of the pixel other than black as an additional image image, and stored in the additional image graphic unit 5.

According to the above steps, the corresponding relationship between the hook marks can be established correctly, and the designated area shown in FIG. 33 can be decorated.

According to the eighth embodiment, it is also possible to decorate the following books, which refer to not only the character strings written one by one, but also more complex books with mixed graphics of characters and graphics.

9th embodiment

Next, a ninth embodiment of the image synthesizing apparatus of the present invention will be described.

Compared with the first to eighth embodiments, the idea of the ninth embodiment is to correctly display the emphasized part even when the camera is zoomed.

Fig. 34 is a structural diagram of the ninth embodiment. In this embodiment, the same structural parts as those of the first embodiment shown in Fig. 1 are given the same reference numerals, so their detailed descriptions are omitted here.

The main difference between the ninth embodiment and the first embodiment is that a zoom factor detection unit 23 for detecting the zoom factor of the camera 1 is added, and a decorative image enlargement device that enlarges or shrinks the decorative image according to the detected zoom factor is added. Shrink unit 24.

Next, the operation of the image synthesizing apparatus of the ninth embodiment constructed as above will be described with reference to the flow chart of FIG. 35. FIG.

In the operation of the ninth embodiment, the following steps 1i to 3i are inserted between the steps 1a and 2a in the operation of the first embodiment.

Step 1i Presence of existing indicated areas:

The additional image enlargement and reduction unit 24 proceeds to step 2i when the existing additional image is formed with reference to the additional image forming unit 5, and proceeds to step 2a when there is no additional image.

Step 2i detection of zoom factor:

The zoom factor detection unit 23 detects the zoom signal of the camera 1 by means of the control unit 1000 and transmits it to the additional image enlargement and reduction unit 24 .

Step 3j Enlargement and reduction of the decoration image:

All the images stored in the additional image forming section 5 are enlarged or reduced according to the zoom factor detected in step 21.

According to the above-mentioned ninth embodiment, since the additional image can be zoomed in or out according to the zoom factor even when the camera is zoomed, the part to be emphasized can be correctly decorated, thereby further expanding the scope of the presentation.

10th embodiment

Next, a tenth embodiment of the image synthesizing device of the present invention will be described.

Embodiments 1 to 9 show examples in which areas designated by a plurality of pointing mechanisms such as pens and setting marks are decorated with colors or underlines. The idea of the tenth embodiment is not to additionally display a specified area colored with a specified color.

FIG. 36 is a structural diagram of the tenth embodiment. In this embodiment, the same structural parts as those in the embodiment shown in FIG. 1 use the same reference numerals, so their detailed descriptions are omitted here.

The main difference between the tenth embodiment and the first embodiment lies in the addition of a coloring designated area deleting unit 25, which is used to detect the included area colored in a specified color, and delete the display of the information of this area.

The operation of the above-mentioned image synthesizing apparatus of the tenth embodiment will be described below.

The coloring designation area deleting unit 25 continuously scans the image, and sets the same flag on pixels that are similar (for example, near the number 8) having predetermined RGB values as the same designation area. Next, for a predetermined number or more of pixel groups provided with the same mark, all the colors of the pixels in the same group provided with the mark are replaced with a predetermined color (RGB value).

According to the tenth embodiment described above, since the information of the specified area colored with the specified color is not displayed on the display unit, when the teacher paints the answer to the question or the annotation part recorded on the textbook as a note with the specified color, the listener cannot See unnecessary information.

11th embodiment

Next, an eleventh embodiment of the image synthesizing apparatus of the present invention will be described.

The situation that the first to tenth embodiments are aimed at is that the presentation can be performed simply and efficiently by forming, synthesizing, and displaying an image that adorns or emphasizes the indicating means only by using a prescribed indicating means to indicate books, and the eleventh embodiment Embodiments also contemplate scaling up the education system with good efficiency.

Fig. 37 is a structural diagram of the eleventh embodiment. The same structural parts of this embodiment and the first embodiment shown in Fig. 1 use the same reference numerals, so their specific descriptions are omitted here.

The main difference of the 11th embodiment and the 1st embodiment is: also add following browsing corresponding file and form update unit 26, they can save the image synthesized by image synthesis unit 5 in the mode that can be shown by the browsing program in the Internet. elephant. In addition, reference numeral 27 denotes a server managed by a home page, which is connected to a plurality of terminals through a network or the like.

Next, the operation of the image synthesizing apparatus of the eleventh embodiment described above will be described.

The example described below illustrates the situation that the teacher of the school adopts the situation of the image synthesizing device of the present embodiment when giving lectures. The control unit 1000 fills the memory for storing images with information, such as saving the image synthesized by the image synthesis unit 5 by pressing a prescribed key of a keyboard connected to the control unit 1000 . Alternatively, in the same manner as in the third embodiment, for example, an image called "image storage" is superimposed on the image of the camera 1, and it is indicated with a pen or the like. Thus, the teacher can save in the system the images useful to the students among the images formed during the lecture. Next, the browser corresponding file forming/updating unit 26 converts the stored image into GIF or JPEG format, and creates or updates an HTML file that can be used as a reference with respect to the server 27 . As a result, students can view the above-mentioned lecture materials from outside the school.

If the present embodiment 11 is implemented according to the above steps, since the formed synthetic image is logged on the homepage of the network, it is possible to participate in lectures of schools such as universities from outside the school through the terminal, or reuse the lecture materials, and the teacher can provide reports ^** in addition An efficient educational support system for (report) subjects, etc.

12th embodiment

Next, a twelfth embodiment of the image synthesizing device of the present invention will be described.

The 1st to 11th embodiments refer to the fact that only the actual information recorded in books, etc. is indicated by means of a pen, and the additional image synthesized corresponding to the feature is displayed in a superimposed manner, so that an efficient demonstration system can be realized. The idea of the 12th embodiment is to simply acquire a plurality of actual objects as additional images for synthesis, process them, and send the synthesized images to equipment such as fax machines or televisions.

FIG. 38 is a structural diagram of the twelfth embodiment. In this embodiment, the same structural parts as those of the first embodiment shown in FIG. 1 use the same reference numerals, so their detailed descriptions are omitted here.

The main difference between the twelfth embodiment and the first embodiment is that the display unit 8 is replaced by an input and display integrated display 28. This embodiment includes an input position detection unit 29 for detecting the input position and the actual object in the above-mentioned display 28. Take out unit 30, it is provided with camera 1 according to the mode that can photograph at least a part in the display portion of display 28 at least, and can extract the contour of the actual object placed on the display portion in display 28, decorate image forming unit 31, Carry out by processing the image of the object extracted by the actual object extraction unit 30 to form an additional image for decoration or emphasis and displayed in the display. An additional image storage unit 32 for storing the formed additional image; decoration Function selection unit 33, which is used to select the display method of the additional image formed in the display; and image transmission and reception unit 34, which is used to receive the following image, or transmit this image to a facsimile machine or a television set In this case, the image is obtained by synthesizing the actual object on the display 28 and the displayed image on the display 28. In addition, the control unit 1000 is constituted by a memory such as a ROM and a CPU, and processing data corresponding to input through the display 28 described above is stored in the memory.

The working process of this embodiment will be described below.

Fig. 39 is a flow chart showing the operation of the image synthesizing device in the twelfth embodiment. Fig. 40 is an example of the range of shooting by means of the input display integrated display and the camera in this embodiment. In order to describe the actions of this embodiment more clearly, the example described here refers to the occasion of synthesizing the image of the message of thanking the gift on the person who received the watch at the celebration ceremony of joining the company, and sending it to the facsimile machine .

The action steps of this embodiment will be described below.

The setting of the actual object in step 1j:

In an area where photographing is performed by means of the camera 1 provided above the input-display-integrated display 28, actual objects intended to be used as information are set. As shown in Fig. 40, in the example described here, the watch that is received from the party that intends to send information is set on the display 28, and the name card received from the company after entering the company is determined to be assigned. Here, when the input surface of the display 28 is made of ultrasonic surface acoustic wave method or electromagnetic induction type, even when a relatively hard object such as a watch is placed, it will not be erroneously judged that there is input information in the display 28 . In addition, if the object is relatively light, if the material is pressure-sensitive or resistive film, there will be no misjudgment. Whether or not there is an input on the display 28 is controlled by the control unit 1000 .

Step 2j takes out the actual object:

As shown in FIG. 41, it is preferable that the function selected by the decorative function selection unit 33 is displayed on the display 28 as an icon. When the actual object extraction icon 401 is pressed, the actual object to be acquired becomes the selected mode. By means of pen input, the actual object to be taken out (here a watch and a name card) is surrounded by a closed curve. The control unit 1000 receives the final signal input by the pen, first obtains the image in the shooting area through the camera 1, and then activates the actual object extraction unit 30 to extract the actual object based on the acquired image. Here, the pen input starts and ends by pressing the pen input icon 411 . The actual object taking-out unit 30 automatically separates a closed curve (here, 2) according to the quantity corresponding to the quantity of objects according to the mode shown in FIG. 42, and takes out the watch and the name card separately. The above method will be described in detail later. In addition, if the pen input adopts the ultrasonic surface elastic wave method, soft materials such as fingers or rubber, in the case of pressure-sensitive or resistive film type, the material may be a hard material such as an intermediary of an ordinary ballpoint pen. Can be entered by finger.

Step 3j selects the decorator mode's:

The control unit 1000 activates the decorative image forming unit 31 to change the color or shape of the image of the actual object taken out to form a decorative image and store the image in the decorative image storage unit 32 . While there are multiple ways to achieve the above results, what is illustrated here is a narrowed-down instance. The control unit 1000 displays the decoration image formed by the decoration image forming unit 31 (represented by the extracted reduced image 422 of the actual object in FIG. 42 ) and the decoration mode button 421 shown in FIG. 42 on the display 28. . Here, the decoration image and decoration mode are selected in such a manner that an image displayed on the display 28 or an icon of a button is touched with a pen or the like. These input indications are processed by the input position detection unit 29 .

It is conceivable to have a plurality of decoration modes, but here, three modes of "background", "edge", and "free" will be described. The "background" mode is the entire display area of the display 28, the "edge" mode is along the inside of the above-mentioned display area, and the "free" mode is to display a corresponding decoration image anywhere in contact with the display 28 with a pen or the like. Fig. 43 shows the result, from which the image of the watch is selected as the decoration image from the reduced image 422 of the actual object, and the button 421-a corresponding to the "edge" mode is selected with a pen to obtain, that is, The control unit 1000 shrinks the watch image 422 image along the inner edge of the display area of the display 28 according to the "edge" mode.

Step 4j information input:

Enter the information to be sent to the other party by pen input. For example, input the information shown in Figure 44. This pen input is performed by converting the display display at the position in contact with the input surface to a predetermined color (for example, black) by the pen.

Step 5j composite image transmission:

As shown in FIG. 44, when the "transmit" button 402 displayed in the display 28 is pressed, the control unit 1000 first starts the image synthesis unit 7 to capture the image data of the actual object on the display 28 captured by the camera 1, The data of the decorative image displayed on the display 28 or the information data are subjected to composite image processing, and stored in a bitmap format or the like. Next, the above-mentioned stored image is converted into a data format corresponding to the data format of the device to which the data is transmitted by the image transmission and reception unit 34, and the composite image is transmitted. For example, when transmitting data to a fax machine, the image can be converted to a scan width format.

In addition, the above-mentioned image data of the actual object on the display 28 captured by the camera 1 is processed by the actual object extraction unit 30, and then sent to the image synthesis unit 7, so that in the image synthesis unit 7 and by the storage unit 32, the data and the like of the above-mentioned decorative image outputted are synthesized.

In addition, in addition to the above-mentioned synthesis methods, the following methods can also be employed.

This is a method in which the actual object on the display 28 and the decorative image or information displayed on the display 28 are photographed by the camera 1 as it is. In this case, the synthesized image from the camera 1 can also be directly transmitted to the image transmitting and receiving unit 34 . In addition, unlike the aforementioned synthesizing method, at this time, the display 28 displays images such as decorative images or messages of poor quality.

The method for extracting the actual object on the display 28 by the actual object extracting unit 30 in the above step 2j will be specifically described below.

FIG. 45 shows a detailed configuration diagram of the actual object take-out unit 30. As shown in FIG. Reference numeral 35 among Fig. 45 represents the image storage unit that stores the image that camera 1 shoots; Reference numeral 36 represents outline style storage unit, and it is stored object as the image in the image storage unit 35 to the closed curve of input display 28 The contour candidate points are discretely processed and stored; label 37 represents a contour sample deformation unit, which is used to move all contour candidate points in the contour pattern storage unit 36 according to certain rules; label 38 represents a contour pattern intersection judgment unit, which is used to Predetermined combination makes the closed curve formed by connecting the above-mentioned contour candidate points to judge whether there is a crossing phenomenon, and the reference numeral 39 represents a contour splitting unit, which is used to classify the contour candidate points into a plurality of groups when the crossing is detected by the above-mentioned crossing judging unit 38 Label 40 represents a contour candidate point formation/cancellation unit, which is used to add a new contour candidate point when judging that there is no intersection by the above-mentioned intersection judging unit 38; Completed extraction completion judging unit.

Next, the operation of the actual object taking-out unit 30 constructed in the above manner will be described with reference to the flowchart shown in FIG. 46 . The image storage unit 35 is composed of a common RAM, and digitally processes and stores the images captured by the camera 1 . Pixel data in the digitized image is stored in such a manner that its brightness is represented by, for example, 8 bits. The operation procedure of the actual object extracting unit 30 will be described below.

Initial configuration of candidate contour points in step 2j-1:

The n points constituting the closed curve (contour pattern) entered into the display 1 by pen input in the aforementioned step 2j are stored in the contour pattern storage unit 36 as initial contour candidate points vi(xi(0), yi(0) ) (i=1, 2, . . . , n). The symbol i indicates the connection order of the contour candidate points of the closed curve (contour pattern) formed by connecting the contour candidate points. The above-mentioned outline pattern storage unit 36 is constituted by a general RAM.

Deformation of step 2j-2 contour model:

The positions of all the candidate contour points are moved once by the contour pattern deformer 37 . As a result, the closed curve formed by connecting the contour candidate points is deformed. In addition, the closed curve is formed by connecting the contour candidate points in a predetermined connection order.

Deformation of the contour sample is achieved by moving each contour candidate point in a direction that minimizes a predefined energy -Esnake in the contour sample. This defined energy will be described later.

For example, if each contour candidate point moves within a specified range, the above-mentioned candidate point can be moved to the position where Esnake (vi) reaches the minimum, and each contour candidate point vi (xi, yi) (i=1, 2,...n) Calculate the steepest descent vector (-Esnake/xi, -Esnake/yi) in Esnake (vi), and move the above-mentioned candidate point according to the following mathematical formula.

xi(t+1)=xi(t)-Kx( Esnake/xi)

yi(t+1)＝yi(t)-Ky(Esnake/yi)        (1)

However, t(i=1, 2, . . . n) represents the number of times of movement of each point. In addition, Kx>0 and Ky>0 are constant numbers that control the amount of movement of the contour candidate points. The first movement of each contour candidate point can be performed sequentially along the contour, or independently and in parallel for each point. However, when the energy-Esnake(vi) increases, the above-mentioned candidate points do not move. At this time, the flag ^** (flag)F(vi)(i==1, for the contour candidate point that does not move, F(vi)=0. As the energy-Esnake(vi) in the contour model, the following formula can be used: The sum of the energy items represented by the above formula. In addition, it can be understood in various forms according to the characteristics of the object.

(1) Espline representing the smoothness of the profile sample: $\mathrm{Espline}\left(\mathrm{vi}\right)=(1/2)\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}[\mathrm{<figure-callout\; id="1"\; label="Wsp"\; filenames="A9712595700541.png,A9712595700561.png"\; state="\{\{state\}\}">Wsp</figure-callout>}1{|\mathrm{vi}-\mathrm{vi}-1|}^2+\mathrm{<figure-callout\; id="2"\; label="Wsp"\; filenames="A9712595700541.png,A9712595700561.png"\; state="\{\{state\}\}">Wsp</figure-callout>}2{|\mathrm{vi}+1-2\mathrm{vi}+\mathrm{vi}-1|}^2]----\left(2\right)$

The moving direction of the contour candidate point that minimizes the result of the expression (2) is the direction that shrinks the contour pattern. Here, i, i-1, and i-1 in vi, vi-1, and vi+1 are respectively appended with the letter v. The same applies to the following mathematical formulas as well.

(2) Earea equivalent to the area of the closed area around the contour pattern: $\mathrm{Earea}\left(\mathrm{vi}\right)=(1/2)\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}\mathrm{Warea}\left[\mathrm{xi}\right(\mathrm{yi}+1-\mathrm{yi})-(\mathrm{xi}+1-\mathrm{xi}\left)\mathrm{yi}\right]----\left(3\right)$

However, xn+1=x1, yn+1=y1. The movement direction of the contour candidate point which makes the expression (3) the minimum value corresponds to the direction perpendicular to the contour pattern. With Erea, the outline of a concave object can be extracted. Here, i, i+1 in xi, xi+1 are respectively appended with the letter x. In addition, i and i+1 in yi and yi+1 are respectively appended with the letter y. The same applies to the following mathematical formulas.

(3) Edist that averages the distance between contour candidate points: $\mathrm{Edist}\left(\mathrm{vi}\right)=(1/2)\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}\mathrm{Wdist}{\left\{\right|\mathrm{dav}-|\mathrm{vi}-\mathrm{vi}-1|\left|\right\}}^2----\left(4\right)$ However, dav represents the distance between outline candidate points. (4) Eedge with increasing gradient of image brightness l(vi): $\mathrm{Eedge}\left(\mathrm{vi}\right)=-(1/2)\underset{i-1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}\mathrm{Wedge}{|\&dtri;I(\mathrm{vi}\left)\right|}^2$ $=-(1/2)\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}\mathrm{Wedge}\left[\right(\&PartialD;I\left(\mathrm{vi}\right)/{(\&PartialD;\mathrm{xi})}^2+{(\&PartialD;I\left(\mathrm{vi}\right)/\mathrm{yi})}^2----\left(5\right)$

The moving direction of the contour candidate point in which the result of Equation (5) becomes the minimum value is the direction in which the brightness gradient value increases, that is, the direction in which the object faces the contour portion.

(5) Image brightness value Eintens: $\mathrm{Eintens}\left(\mathrm{vi}\right)=\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}\mathrm{Eintens\; I}\left(\mathrm{vi}\right)----\left(6\right)$

The moving direction of the contour candidate points in which the result of the formula (6) is the minimum value is the direction of decreasing (darkening) the brightness when Wntens>0, and the direction of increasing (brightening) the brightness when Wintens>0. In addition, Wsp1, Wsp2, Warea, Wdist, Wedge, Wintens>0 in formulas (2)-(6) are the main coefficients of each energy item respectively.

Deform the outline pattern so that the sum of the energy items in (2)-(6) reaches the minimum value, then the initial outline begins to shrink, and for the outline part of the object, the outline candidate points caused by each energy item generate a moving vector Stop moving in a balanced manner. A polygon formed by connecting the contour candidate points at this time in a predetermined order is taken out as the contour of the object.

Intersection judgment of step 2j-3 outline style:

The outline pattern intersection judging unit 38 judges whether or not two different combinations intersect in a closed curve (polygon) formed by connecting outline candidate points in a predetermined connection order. But the exception is the case of connecting two. Intersection determination can be performed, for example, as follows. If the line segment vivi+1 intersects the line segment vjvj+1, there are real numbers p(0≦p≦1), q(0≦q≦1) satisfying the following equation.

p(vi+1-vi)+vi=q(vj+1-vj)+vj (7)

Equation (7) is a simultaneous equation about p and q. It has a solution when the determinant det is 0 calculated according to the following formula. At this time, the line segment vivi+1 and the line segment vjvj+1 only intersect at one point. (But when det=0, can produce the situation that line segment vivi+1 is consistent with line segment vivj+1, judge the intersection in this case according to whether the contour candidate points are consistent before determinant calculation.)

det＝(xi-xi+1)(yj+1-yj)-(xj+1-xj)(yi-yi+1) (8)

Therefore, if det≠0, p and q calculated according to the following formula satisfy the relationship of 0≤p≤1, 0≤q≤1, then determine the polygon formed by connecting the contour candidate points according to the specified connection order (contour model) Intersection occurs, proceed to step 4a, except for the above cases, proceed to step 5a.

p=((yj+1-yj)(xi-xj)+(xj-xj+1)(yi-yj))/det

q＝((yi+1-yi)(xi-xj)+(xi-xi+1)(yi-yj))/det (9)

Step 2j-4 Splitting of contour styles:

The contour model is split into several by the contour model splitting unit 39, and it returns to step 2a. The outline pattern is split at the outline candidate points constituting the line segments with intersections. Fig. 47 shows an example of a typical intersection generated when the energy in the contour pattern adopts Earea. Earea takes a positive or negative value. In this example, the contour pattern is split into two in the following manner, that is, the contour candidate points vi and vj+1, and vj and vi+1 are respectively connected in such a way that their signs do not change. Therefore, the contour candidate point groups connected in the sequence {v1,...,vi,vj+1,...,vn} and {vi+1,...,vj} respectively form independent contour patterns in a new split. Although intersections are generated at multiple places at the same time, the step of splitting into two is repeated to split into a plurality of outline patterns.

Step 2j-5 Judgment of formation/deletion of contour candidate points:

A new contour candidate point is formed or canceled by the contour candidate point formation/cancellation unit 40 . For example, contour candidate points can be formed or deleted in the following cases. When the distance between the phase contour candidate points satisfies |vi+1-vi|>DTH, a new discrete point is generated between two points vi and vi+1. DTH is the maximum distance between predetermined contour candidate points. If there are formed discrete points, Gflag=1, and if there are no such discrete points, Gflag=0. In addition, the discrete point vi that satisfies the relationship of COsθ>θTH (θ is the angle between vivi-1 and vivi+1) is deleted. θTH is a threshold value for judging a sharp point of a predetermined contour pattern. If there are discrete points to be deleted, then Dflag=1. If there is no such discrete point, then Dflag=0. By additionally forming candidate contour points, the number of candidate contour points of each object reduced by splitting is supplemented, so that the contours of a plurality of objects can be correctly extracted. In addition, if the contour candidate points at positions other than the contour of the target object are deleted according to the deletion procedure, it is possible to avoid contour extraction by mistake.

Judgment that step 2j-6 has been taken out:

When all contour candidate points do not move (i; F(vi)=0), nor do new contour candidate points form or delete (Gflag=0 and Gflag=0), the judging unit 41 ends by taking out Outline removal. Otherwise, return to step 2a.

In the manner described above, the twelfth embodiment is implemented in the order of operations described in steps 1 to 5 and steps 2j-1 to 2j-6, and a plurality of fitted real objects are placed on the display. The closed curve of the object detects that the closed curve itself intersects with the deformation of the closed curve, splits it into multiples, and extracts these curves individually, so that a composite image of the actual object and the hypothetical object can be easily formed. In addition, even in the case of many synthetic images to be formed, it is possible to reduce the burden on the user to take out the actual object.

In addition, the above steps 1j to 4j do not need to follow the above order, for example, first input the information in step 4j, and then store the actual object in the computer, and the images can be synthesized by any combination of them, so as to form Decorative image. It goes without saying that the above steps 1j to 4j may be repeated in any order so that the images can be synthesized. However, after the decorative image or the information input by the pen is displayed on the display, when the actual object newly placed on the display is taken out, the image is captured by the camera 1 only while the closed curve is input with the pen. During this period, part of the display area of the display 28 including the closed curve input by the pen is converted to a predetermined color (for example, white or black). Thus, only actual objects can be retrieved without affecting hypothetical objects.

Furthermore, it goes without saying that the present embodiment is easy to realize the following extensions, that is, by connecting with a digital camera and synthesizing snapshot photos, and by connecting with a microphone to add sound information.

Thirteenth embodiment

Next, a thirteenth embodiment of the image synthesizing device of the present invention will be described.

The situation described in the twelfth embodiment means that in the case of inputting information with a pen, by pressing a button called "pen", the user inputs the following command into the system, which is used to distinguish whether information is input or Closed curves for selecting real objects. The idea of the thirteenth embodiment is to reduce the burden of input to the above-mentioned system, and without the user having to press a selection button, to provide an interface that uses a commonly used actual pen or eraser, or Miniatures such as facsimile machines are used as three-dimensional icons for intuitive understanding of their functions, making it easier to compose images. At this point, the idea of this embodiment is that even under the assumption that an object is displayed on the display 28 and has the same color as a plurality of real objects that can be discerned, it can be stabilized regardless of the actual object. distinguish.

Fig. 48 is a structural diagram of the thirteenth embodiment. In this embodiment, the same structural parts as those in the twelfth embodiment shown in Fig. 38 use the same reference numerals, so their detailed descriptions are omitted here.

The main difference between the thirteenth embodiment and the twelfth embodiment is that in the twelfth embodiment, a three-dimensional icon registration unit 42 and a three-dimensional icon identification unit 43 are added. The above-mentioned three-dimensional icon registration unit 42 is used for registering actual objects such as pens or three-dimensional icons for inputting into the system; the above-mentioned three-dimensional icon identification unit 43 is used for identifying the input method of the input information when the display has input information.

In the following, the working process of the image synthesizing device of the thirteenth embodiment constructed in the above-mentioned manner will be described only through the structural parts different from those of the twelfth embodiment.

First, the operation of the three-dimensional icon registration unit 42 will be described. The user activates the three-dimensional icon registration unit 42 by pressing the icon displayed on the display 28 by the decorative function selection unit 33 before synthesizing images. When the three-dimensional icon registration unit 42 is activated, a list of functions including "text input", "deletion" and other systems is displayed on the display 1 . The user uses the camera 1 to photograph actual objects such as ballpoint pens and erasing erasers that are commonly used, and registers their feature values in the above-mentioned list. The above-mentioned list is stored as a file in a text format, and its information can be displayed on the display 28 when the three-dimensional icon registration unit 42 is activated. As feature quantities, frequency distributions corresponding to RGB values are respectively stored. Also, when shooting with a camera, it is best to shoot the object in its actual state of use. In addition, when the product leaves the factory, special pens, erasing erasers, and three-dimensional icons of facsimile machines can be added, and the corresponding functions can be stored without being set by the user.

Next, the operation of the three-dimensional icon distinguishing unit 43 will be described. The three-dimensional icon distinguishing unit 43 is activated every time information is input to the display 28 . The control unit 1000 controls whether there is input information and its position index on the display 28 . This three-dimensional icon discrimination unit 43 first receives the input position index in the display 28 from the control unit 1000, and at this position, it is displayed corresponding to the size of the three-dimensional icon. The area of the square) is imaged by camera 1. Next, in the acquired image, the RGB value of each pixel is detected to form a frequency distribution only for the aforementioned region, and it is judged whether to use a three-dimensional icon of the object whose frequency distribution registered in the three-dimensional icon registration unit 42 is consistent with The most similarity is higher. The above similarity is the sum of the absolute values of the differences at all levels in the frequency distribution. As a result, if it is judged to be a pen, the movement locus of the pen as a character input is displayed on the display 28 . Also, if it is judged to be a small icon of a facsimile machine, a synthesized image is transmitted. However, in the case of using a facsimile machine icon, etc., if the angle of view of the camera is wider than the image synthesis area, the icon can be placed in this area without being affected by the synthesis image.

According to the above steps, by detecting only the characteristic amount of the actual object, any three-dimensional icon can be correctly identified without being affected by the displayed image, so that the registered three-dimensional icon can be used stably, and thus can be performed by more intuitive operation. Image synthesis.

According to the above-mentioned thirteenth embodiment, since an area of specified color and size is displayed at the input position of the display, and then the image is acquired, only the color detection is performed on this area, so that only the actual color can be detected without affecting the displayed image. The characteristic amount of an object, since any actual object can be correctly discriminated even if the hypothetical object displayed in the display includes the same color as a plurality of recognizable actual objects, so that the registered three-dimensional icon can be stably used, Image composition can be performed more intuitively.

14th embodiment

Next, a fourteenth embodiment of the image synthesizing apparatus of the present invention will be described.

The twelfth embodiment describes the case where the character color is black when inputting information with a pen. Generally, in the case of a pen input system, the user can select a desired color form from a menu. For example, when the synthesized image is transmitted to a monochrome facsimile machine, the color of the pen is black when inputting characters in a bright image, and a white pen can form a clear image in a darker image. The idea of the fourteenth embodiment is to automatically change the color of the pen according to the brightness of the image, thereby further reducing the burden of inputting style changes in the system and synthesizing a clearer image.

Fig. 49 shows a configuration diagram of the fourteenth embodiment. In this embodiment, the same structural parts as those in the twelfth embodiment shown in Fig. 38 are given the same reference numerals, so their detailed descriptions are omitted here.

The main difference between the fourteenth embodiment and the twelfth embodiment is that in the twelfth embodiment, a display image brightness calculation unit 44 and a pen input color conversion unit 45 are added. The display image brightness calculation unit 44 is used to calculate the brightness of the displayed image, and the pen input color conversion unit 45 is used to change the pen input color according to the brightness calculated by the display image brightness calculation unit 44.

The image synthesizing apparatus of the fourteenth embodiment constructed as above will be described only with respect to the displayed image brightness calculation unit 44 and the pen input color conversion unit 45 which are different from those of the twelfth embodiment.

When a pen input is made to the display 28, the displayed image brightness calculation unit 44 is activated to calculate the average brightness of the entire displayed image. Whether or not pen input is performed is judged by the method in the thirteenth embodiment. The pen input color conversion unit 45 refers to the average luminance calculated by the display image luminance calculation unit 44, and determines that the color of the pen input is black when the above-mentioned average luminance is 180 or more out of 256 levels, and determines the color of the pen input in other cases. for white. Thus, when inputting text in a brighter image, the color of the pen is black, and when in a darker image, the color of the pen is automatically changed to white, so that the user can choose the input color of the pen without realizing it. In the case of , synthesize an image in which the text is clearer.

According to the above fourteenth embodiment, since the color of the pen input is changed according to the brightness of the displayed image, it is possible to synthesize a clearer image of the input characters.

15th embodiment

Next, a fifteenth embodiment of the image synthesizing device of the present invention will be described.

The twelfth embodiment describes a decorative image that changes the size and color of the extracted image. Also in the fourteenth embodiment, the color of the pen input is changed according to the brightness in the displayed image, thereby synthesizing an image in which the character information is clearer. The idea of the fifteenth embodiment is that a clearer image can be synthesized by changing the color or brightness of a displayed image according to the color or brightness of an actual object placed on the display 1.

Fig. 50 is a structural diagram of the fifteenth embodiment. The structural parts in this embodiment that are the same as those in the twelfth embodiment shown in Fig. 38 are given the same reference numerals, so their detailed descriptions are omitted here.

The main difference between the 15th embodiment and the 12th embodiment is: in the 15th embodiment, additional display image correction units 46, 46 are used to change the brightness or color of the displayed image according to the brightness or color of the actual object.

Next, the operation of the image synthesizing apparatus of the fifteenth embodiment constructed as above will be described with regard to the display image correction unit 46 which is different from the twelfth embodiment.

The display image correction unit 46 is activated when the image synthesis is completed and the transfer button is pressed. Firstly, the display image correction unit 46 starts the actual object extraction unit 30 and uses a rectangle having the same size as the shooting area of the camera 1 as an initial outline pattern, thereby extracting all the actual objects on the display. After that, the above-mentioned display image correction unit 46 calculates the average brightness of all the extracted objects. Finally, the brightness of the displayed image on the display 28 is changed based on the average brightness. When the actual object is brighter (eg, above the average brightness), a clear composite image can be formed as a whole by also brightening the displayed image (eg, increasing the brightness of each pixel value by 20%).

According to the above fifteenth embodiment, the color or brightness of the displayed image is correspondingly changed according to the color or brightness of the actual object on the display, so that a clearer image can be synthesized.

However, the same operation as above can also be realized in the following manner, that is, the following magnetic recording medium or optical recording medium is produced, and the following program is recorded in the medium, and this program is used to realize the above-mentioned embodiments in a computer. All or part of the steps (or processing actions) of each step (or each mechanism) described in any of them.

In addition, although the separate extraction mechanism of the present invention is an actual object extraction unit in the above embodiment, and is described for the occasion of an integrated display for input and display, the present invention is not limited thereto, and the following diagrams can also be used: Like a synthesizer.

That is, the image synthesizing device at this time includes: a photographing mechanism such as a camera, which is used to photograph surfaces and/or objects such as books showing photos or illustrations; a closed curve detection mechanism, which is used to detect closed curve information, Said closed curve surrounds information in a plurality of photographs or illustrations taken by the above-mentioned photographing mechanism, or extraction target candidates of a plurality of candidate illustrations that can constitute an image extraction target in a plurality of objects; The measured closed curve information and the image information of the extraction target candidates such as the above-mentioned illustrations extract the images of these illustrations and the like from the images of the plurality of extraction target candidates such as the illustrations surrounding the above-mentioned curves; The images such as illustrations and the like extracted separately are image-combined with other images such as information. Thereby, information displayed on a predetermined plane and/or a plurality of objects can be imaged, closed curves that can constitute a plurality of extraction target candidates surrounding image extraction target candidates can be provided, and the provided closed curve information can be detected. According to the above-mentioned closed curve information measured and the image information of the above-mentioned extraction object candidates, the images of these extraction object candidates can be extracted respectively from the images of the plurality of extraction object candidates surrounded by the above-mentioned, and the above-mentioned extraction object candidates can be extracted respectively. images for image synthesis.

According to the above configuration, as in the above case, even when there are a plurality of image extraction objects, compared with the prior art, it is possible to further reduce the workload for instructing these image extraction objects. Therefore, designation of the indicated area can be performed more easily than in the prior art.

Alternatively, an image synthesizing device described below may also be used.

At this time, the image synthesizing device includes: a photographing mechanism, which is used to photograph the surface for displaying information such as photos, illustrations, character strings, etc.; a closed curve information detection mechanism, which is used to detect A closed curve surrounding a plurality of pointing object information that can constitute a pointing object; a separate extraction mechanism for extracting from the above-mentioned enclosed plurality of pointing object information based on the detected closed curve and the image information of the pointing object information The images indicating the object information are taken out from the image respectively; the indicating information detection unit is used to detect the above-mentioned respectively taken-out image regions and/or their image feature values; an additional image forming unit is used for detecting the above-mentioned detected The image area and/or image feature quantity form an image attached to the above-mentioned indicated object information; the additional image storage unit is used to store the additional image formed above; the image synthesis unit is used to obtain the above-mentioned shooting mechanism The image pointing to the object information is synthesized with the additional image formed above. The above structure can also be used in an alternate embodiment of the image synthesizing apparatus described in the eighth embodiment. That is, as shown in Figure 31, in the case that the extracted objects are a collection of polygonal figures and two photos, according to the method described in the eighth embodiment, each object is indicated separately, and a closure surrounding the above two objects is provided. curve. The structure of the separate extraction mechanism is the same as above.

According to the above configuration, as in the above case, even when there are a plurality of image extraction objects, compared with the prior art, the effect of further reducing the workload for instructing these image extraction objects can be exhibited.

In addition, in the above-mentioned embodiment, the image synthesizing device of the present invention is described for the case of being used for an input display integrated display having an actual object extracting unit, but the image synthesizing device of the present invention is not limited to this case , which can also use the following image synthesis means.

This image synthesis device includes an input and display integrated display; a photographing mechanism for photographing at least a part of the display unit of the above-mentioned display; a pointing position detection unit for detecting pointing position information pointing to an actual object placed on the above-mentioned display. , the actual object take-out mechanism, used to take out the image of the above-mentioned actual object from the image taken by the above-mentioned shooting mechanism; the additional image forming unit, used to form an additional image displayed on the above-mentioned display from the above-mentioned taken-out image The additional image storage unit is used to store the additional image formed above; the display method selection unit is used to select the display method of the additional image formed above in the display; the image synthesis unit; The display method synthesizes the above-mentioned additional image and the image of the above-mentioned actual object. Thereby, the effect of simply forming a composite image of an actual object and a hypothetical object can be exerted in a manner that is not available in the prior art.

Implement the first embodiment according to the above-mentioned method, because the person who performs the demonstration only indicates the desired emphasized part in the written information itself with the indicator rod, automatically extracts the area or feature of this part, and superimposes an appropriate decorative image, so that there is no need to replace it with ways to use each color separately. In addition, the decoration method can be easily changed by the pointing position of a pen, etc., and if it is near the end of the book, since the frame, title, date, etc. can be given, or the decoration can be deleted in the decorated information part by its pointing position , to brighten the above decorations, so that efficient presentations can be obtained even without pre-formed presentation sheets.

Therefore, there is an advantage that processing such as embellishment or emphasis can be performed more easily, such as forming feature quantities based on image regions.

In addition, according to the second embodiment, since the decoration method can be changed by recognizing various marks given by the pen etc., it is possible to replace the pen with various colors, save the time for recording, etc., and can smoothly perform the presentation.

According to the third embodiment, since the existing decorative images can be displayed in a list form by superimposing them, they can be reproduced in such a way that these images can be indicated by the indicator rod. Even if the same string is written to the book class multiple times, the presentation can still be performed smoothly.

According to the fourth embodiment, since the template comparison is performed between the continuous borders using part of the decoration image as a template, the movement parameters of the book can be calculated to deform the decoration image, so that even if the book moves during the presentation Even in the case of the book, it is possible to correctly decorate the part to be emphasized, and the user can realize the presentation without noticing the movement of the book.

According to the fifth embodiment, because of the intuitive and easy-to-understand operation of placing a mark to decorate and removing a mark to delete the decoration, anyone can make an efficient presentation.

According to the sixth embodiment, since the inclination of the inscribed part or the book is detected by setting the mark, the number of movable parameters can be estimated, and since the additional image corresponding to the movement of the book is synthesized more quickly, the number can be further reduced. A sense of incongruity caused by late prompts for compositing images.

According to the seventh embodiment, since the increase or decrease of the number of marks is detected, it is possible to distinguish between the case of setting a mark and the case of removing a mark, and since an effect sound corresponding to the case can be formed, a particularly efficient presentation can be performed .

According to the eighth embodiment, by confirming the shapes of a plurality of marks, it is possible to decorate complex books with rich illustrations and texts according to the confirmed conditions, thereby further improving the presentation effect.

According to the ninth embodiment, since the decorative image can be zoomed in or out according to the zoom ratio of the camera, even when the camera is zoomed, the highlighted part can be decorated correctly, and the scope of presentation can be further expanded.

According to the tenth embodiment, by coloring only the part which is not desired to be displayed by a highlighter or the like, the colored part can be deleted from the image, so that the part not to be seen by the viewer can be hidden.

According to the eleventh embodiment, since the composite image formed can be registered on the homepage in the Internet, it is possible to participate in lectures in schools such as universities outside the school through the terminal, and reuse the lecture notes. In addition, the teacher can provide report topics, thereby realizing efficient educational support system.

According to the twelfth embodiment, since a plurality of actual objects desired to be synthesized can be placed on the display, a closed curve including only these actual objects can be provided, self-intersections generated with the deformation of the closed curve can be detected, and it can be split into In this way, these curves can be taken out separately, so that a composite image of the actual object and the hypothetical object can be easily formed.

According to the thirteenth embodiment, since the distinguishing color of the three-dimensional icon is only detected at a predetermined position corresponding to the input position of the display, any three-dimensional icon can be correctly distinguished without affecting the displayed image, so that all three-dimensional icons can be used stably. Registered 3D icons enable image synthesis through more intuitive operations.

According to the fourteenth embodiment, since the color of the pen input can be changed according to the brightness of the displayed image, it is possible to synthesize an image whose input characters are clearer.

According to the fifteenth embodiment, since the color or brightness of a displayed image can be changed according to the color or brightness of an actual object on the display, a clearer image can be synthesized.

Claims (28)

1. An image synthesis device, characterized in that it comprises: a given filming agency; An indication position detection mechanism, which is used to detect the position of the indication information given on the surface of the display information from the image captured by the shooting mechanism; The pointing object information detection mechanism is used to detect the area of the pointing object information displayed on the surface corresponding to the pointing position detected by the above pointing position detection mechanism and/or the feature value of the pointing object information; additional image forming means for forming an additional image to be added to the pointing object information image based on the above-mentioned area and/or the above-mentioned feature value; and an image synthesizing means for synthesizing the image of the pointing object information obtained by the imaging means with the additional image formed above. 2. The image synthesis device according to claim 1, characterized in that it comprises: The indication position other information addition method determination unit is used to determine the synthesis method and/or display method of the above additional image according to the above detected indication position; The above-mentioned image synthesizing means uses the determination result obtained by the above-mentioned information adding method determining unit when performing the above-mentioned synthesizing. 3. The image synthesizing device according to claim 1, wherein said additional image is an image for decorating and/or emphasizing said pointing object information image. 4. The image synthesis device according to claim 1, characterized in that it comprises: A pointing mechanism, used to point out pointing objects with one or more of the above pointing information; The indicated position detection mechanism also judges the type of the indicated information through the above-mentioned indicated information obtained from the image captured by the above-mentioned photographing mechanism; The above-mentioned additional image forming means also forms the above-mentioned additional image using the above-mentioned judged instruction information type. 5. The image synthesis device according to claim 1 or 4, characterized in that it comprises: an additional image storage unit for storing the above formed additional image; In the existing additional image reproduction unit, the above-mentioned additional image stored in the additional image storage unit is combined with the image obtained by the above-mentioned shooting mechanism, and displayed in a list form, and the composite is displayed at the above-mentioned indicated position. The above-mentioned additional images selected for display in a list form. 6. The image synthesis device according to claim 1, 4 or 5, characterized in that it comprises: A movement parameter estimation unit, which is used to estimate the position of the surface displaying the above information after movement; An additional image warping unit deforms said additional image based on said estimated motion parameter. 7. The image synthesis device according to claim 1, 4, 5 or 6, characterized in that: The indicating mechanism includes a plurality of indicating parts; The indicated position detection mechanism detects the indicated position for each of the plurality of indicated components; The pointing object information detecting unit performs the above-mentioned detection for each of the above-mentioned plurality of pointing components; The above-mentioned additional image forming means forms the above-mentioned additional image for each of the above-mentioned plurality of indicating members. 8. The image synthesizing device according to claim 7, characterized in that at least one indicating member in the plurality of indicating members is an indicating member for setting as follows, which is used to pass through The method indicates the above-mentioned referent. 9. The image synthesis device according to claim 8, characterized in that: The pointing object information detection unit estimates an inclination angle of the pointing object information area relative to the coordinate system of the imaging mechanism based on the installation position of the setting pointing member, and calculates the pointing object information area and/or Indicates the characteristic quantity of object information; The image synthesizing device is provided with a movement parameter calculation unit for estimating a parallel movement parameter of the surface displaying the above-mentioned information based on the above-mentioned estimated inclination angle. 10. The image synthesis device according to claim 7 or 9, characterized in that it comprises: The increase/decrease detection mechanism of the setting indicating parts is used to detect the increase or decrease of the number of the setting indicating parts arranged on the surface displaying the above information; A sound data storage unit for storing sound data corresponding to the increase or decrease in the number of the above-mentioned setting indicating parts; A speaker for outputting the sound data in the sound data storage unit according to the increase or decrease of the number of the setting indicating parts. 11. The image synthesis device according to claim 7, 9 or 10, characterized in that: The indicating mechanism includes indicating components of different shapes; The image synthesis device includes: The corresponding determination mechanism of the indicating components is used to establish correspondence between the above-mentioned different indicating components according to a given combination criterion. 12. The image synthesis device according to any one of claims 1-11, characterized in that: The shooting mechanism is a camera; The image synthesis device includes: The zoom factor detection unit is used to detect the zoom factor of the camera; The additional image enlargement and reduction unit is configured to enlarge and reduce the above additional image according to the detected zoom factor. 13. The image synthesizing device according to any one of claims 1 to 12, characterized in that it includes a coloring designation area deletion unit for exchanging the given color on the surface displaying the above-mentioned information with a color different from the above-mentioned given color area. 14. The image synthesis device according to any one of claims 1 to 13, characterized in that it includes a browsing program corresponding file forming/updating single unit, which saves the above-mentioned image in a format that can be displayed by a browsing program in the Internet The image synthesized by the compositing mechanism. 15. An image synthesis device, characterized in that it comprises: a given filming agency; Closed curve information detection means for detecting closed curve information that surrounds a plurality of extracted object candidates that can constitute information and displayed on a given surface photographed by the above-mentioned photographing means / or multiple object candidates for image extraction inside the object; The respective extracting mechanism is used for extracting the images of the corresponding extracting target candidates respectively from the plurality of extracting target candidate images surrounded by the above-mentioned detected closed curve information and the image information of the above-mentioned extracting target information candidates; The image synthesis mechanism is used to perform image synthesis through the above-mentioned respectively extracted images. 16. An image synthesis device, characterized in that it comprises: a given filming agency; A closed curve information detection mechanism is used to detect closed curve information, and the closed curve surrounds a plurality of pointing object information that can constitute a pointing object in the information captured by the above-mentioned shooting mechanism; A separate extraction mechanism is used to extract corresponding images of the indicated object information respectively from the plurality of indicated object information images surrounded by the above-mentioned detected closed curve information and the image information of the indicated object; An indication information detection unit, configured to detect the aforementioned image regions and/or image feature quantities extracted separately; An additional image forming unit, configured to form an additional image added to the image of the above-mentioned pointing object information according to the above-mentioned detected image area and/or image feature value; The image synthesis unit is used to synthesize the image indicating the object information obtained by the above-mentioned shooting unit and the above-mentioned additional image formed. 17. An image synthesis device, characterized in that it comprises: Input display integral display; a photographing mechanism, the photographing mechanism is used to photograph at least a part of the display portion of the display; a closed curve information detection mechanism for detecting closed curve information that encloses a plurality of objects constituting image extraction target candidates within a plurality of actual objects placed on the display; The separate extraction mechanism is used to extract the images of the corresponding objects from the images of the enclosed multiple objects according to the detected closed curve information and the image information of the multiple objects surrounded by the above-mentioned photographing mechanism. image; an additional image forming unit for forming an additional image displayed on the above-mentioned display from the above-mentioned separately taken-out images; A display method selection unit for selecting the display method of the additional image formed above in the display; The image synthesis mechanism is used for synthesizing the above-mentioned additional image and the above-mentioned image of the actual object according to the above-mentioned selected display method. 18. An image synthesis device, characterized in that it comprises: Input display integral display; a camera for photographing at least a part of the display portion of the display; A closed curve information detection mechanism for detecting closed curve information, the closed curve enclosing a plurality of objects constituting image extraction target candidates among the plurality of actual objects placed on the display; Separate extraction mechanism, used to extract images of corresponding objects from the images of these enclosed multiple objects according to the above-mentioned detected closed curve information and the image information of the above-mentioned multiple objects surrounded by the above-mentioned camera ; an additional image forming unit for forming an additional image displayed on the above-mentioned display from the above-mentioned separately taken-out images; A display method selection unit for selecting the display method of the additional image formed above in the display; The control unit is configured to use the camera to capture the above-mentioned additional image displayed on the above-mentioned display and the actual object placed on the above-mentioned display according to the above-mentioned selected display method. 19. An image synthesis device, characterized in that it comprises: Input display integral display; A photographing mechanism for photographing at least a part of the display portion of the display; a pointing position detection unit for detecting pointing position information pointing to an actual object placed on the above-mentioned display; The actual object taking-out mechanism is used to take out the image of the above-mentioned actual object from the image captured by the above-mentioned shooting mechanism according to the above-mentioned detected indication position information; an additional image forming unit for forming an additional image displayed on the above display from the image taken out above; A display method selection unit for selecting the display method of the above-mentioned formed additional image in the display; The image synthesis mechanism is used for synthesizing the above-mentioned additional image and the above-mentioned image of the actual object according to the above-mentioned selected display method. 20. The image synthesizing device according to any one of claims 15-17, characterized in that said separate extraction mechanism comprises; The image storage unit is used to store the images taken by the above-mentioned shooting mechanism; A contour model storage unit, configured to store the closed curve information as a candidate contour; a contour model deformation unit, configured to deform the candidate contours stored in the contour model storage unit according to certain rules; A contour model intersection detection unit, configured to detect whether there is an intersection of the above-mentioned candidate contours; A contour model splitting unit, configured to split the candidate contours stored in the contour model storage unit into multiple when the intersection detection unit determines that there is an intersection; The extraction completion judging unit is used for judging whether the extraction of the outline is completed according to the above splitting results; After the outline is extracted, the image surrounded by the outline formed by the division is extracted. 21. The image synthesis device according to claim 17 or 19, characterized in that it comprises: A three-dimensional icon registration unit for registering the feature quantity and its function of the actual object by inputting to the above-mentioned display; A three-dimensional icon identifying unit configured to identify an actual object used for the input based on the feature quantity when there is an input on the display. 22. The image synthesis device according to claim 17 or 19, characterized in that it comprises: A display image brightness calculation unit, used to calculate the brightness of the image displayed by the display; A pen input color conversion unit for changing the color of the pen input based on the luminance calculated by the display image luminance calculation unit. 23. The image synthesis device according to claim 17 or 19, characterized in that it also adds the following display image correction unit, which changes the brightness or color of the displayed image according to the brightness or color of the actual object placed on the above-mentioned display. color. 24. A medium characterized by recording a program for realizing all or part of the mechanism functions of the mechanism according to any one of claims 1 to 23 in a computer. 25. A method for image synthesis, characterized in that it comprises the steps of: The specified position in the display information surface is indicated by the instruction information; Take pictures of the information displayed on the above-mentioned face and the above-mentioned instruction information; According to the shooting result, detecting the indicated position indicated by the above indicated information; Detecting the area and/or feature quantity of the information indicating the object displayed on the above-mentioned surface according to the above-mentioned indicated position; Forming an additional image added to the image of the above-mentioned pointing object information according to the region and/or feature value of the above-mentioned pointing object information; The above-mentioned image pointing to the object information and the above-formed additional image are synthesized. 26. The method according to claim 25, wherein when synthesizing the above-mentioned images, the synthesizing method and/or display method of the above-mentioned additional image are determined according to the above-mentioned indicated position, and the determined result is to synthesize the above-mentioned images. 27. A method for image synthesis, characterized in that it comprises the steps of: Capture information and/or multiple objects displayed on a prescribed surface; providing a closed curve of a plurality of candidate extraction objects that can constitute candidates for image extraction objects within the photographed information and/or a plurality of actual objects; Detect the closed curve information provided above; According to the above-mentioned detected closed curve and the image information of the extraction object candidate, from the plurality of extraction object candidate images surrounded by the above-mentioned, respectively extract the corresponding image of the extraction object candidate; An image is synthesized from the above separately extracted images. 28. A medium, characterized in that the medium records the following program for implementing all or part of the steps in any one of claims 25-27 in a computer.

Images