JP2016212291A - Image projection system, image processing apparatus, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to detect each lighting point when pinpointing a picture projected on a projection plane part using visible light and invisible light, and discriminate each lighting point between the lighting point of the visible light and the lighting point of the invisible light.SOLUTION: An image projection system (1) comprises: irradiation means (11 and 12) that radiates visible light and invisible light; an imaging unit (10a) that projects pictures onto a projection plane; an imaging unit (10b) that photographs lighting points of the visible light and invisible light formed on the picture of the projection plane at prescribed timing; a discrimination unit (25) that discriminates a mark corresponding to each lighting point of the visible light and invisible light from photographing image data obtained from the photographing between the mark of the visible light and the mark of the invisible light to detect the mark of the visible light and the mark of the invisible light; and a synthesis unit (26 and 21) that synthesizes unique synthesis information on the visible light and invisible light with respect to the picture of the projection unit on the basis of a position in an image of the photographing image data of each mark of the visible light and invisible light detected by the discrimination unit.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an image projection system, an image processing apparatus, and a program.

  Projectors are widely used for presentations to a large number of people because they can enlarge and project still images and motion images on a screen. During this presentation, the presenter often points the image projected on the screen by irradiating a colored laser pointer or the like to help understand the explanation. When an image is pointed directly with a laser pointer or the like, it is often difficult to visually recognize the irradiation point (light emitting point) of the laser pointer due to the color used in the image. As an improvement measure, a technique of detecting a spot irradiated by a presenter with a laser pointer with a CCD (Charge Coupled Device) camera and projecting a pointer image or the like at the irradiated spot has been studied.

  However, when a pointer image or the like is projected at an irradiation point, in addition to irradiating visible light with a colored laser pointer or the like, irradiating invisible light such as infrared light is also considered. There has never been a technology for detecting each light emitting point when the image projected on the screen is pointed out using visible light and invisible light, and it is an urgent issue to be able to detect each light emitting point together. ing.

  The present invention has been made in view of the above, and detects each light emitting point when an image projected on a projection surface portion such as a screen is indicated using visible light and invisible light, and makes each light emitting point visible. An object of the present invention is to provide an image projection system, an image processing apparatus, and a program that can discriminate between light and invisible light.

  In order to solve the above-described problems and achieve the object, the present invention provides an image projecting system, an irradiating unit that irradiates visible light and invisible light, a projecting unit that projects an image on a projection surface, and a projection surface. An imaging unit that captures a light emission point of visible light and invisible light formed by irradiation of the irradiation unit on the projected image at a predetermined timing, and the visible light from captured image data obtained by imaging of the imaging unit And a discrimination unit for discriminating and detecting whether the mark corresponding to each light emitting point of the invisible light is the visible light mark or the invisible light mark, and each of the visible light and the invisible light detected by the discrimination unit A combining unit configured to combine combined information unique to each of the visible light and the invisible light on an image projected by the projection unit based on a position of the mark in the image of the captured image data; Characterized in that it comprises a.

  According to the present invention, each light emitting point is detected when the image projected on the projection surface portion is indicated using visible light and invisible light, and each light emitting point is discriminated from visible light and invisible light. Therefore, it is possible to synthesize composite information unique to each light emitting point.

FIG. 1 is a diagram for explaining the overall configuration of the image projection system of the first embodiment. FIG. 2 is a diagram illustrating an example of the internal configuration of the hardware of the projection unit. FIG. 3 is a diagram illustrating an example of an internal configuration of hardware of the camera unit. FIG. 4 is a diagram illustrating an example of spectral sensitivity characteristics of a color CCD image sensor. FIG. 5 is a block diagram illustrating an example of a functional configuration of the integrated image projection apparatus. FIG. 6 is a diagram illustrating an example of a pattern image. FIG. 7 is a diagram illustrating an example of a color arrangement of the color wheel. FIG. 8 is a diagram illustrating a correlation between the color filter and the RGB light. FIG. 9 is a diagram illustrating a projection period of the projected light color of light passing through one rotation of the color wheel. FIG. 10 is a diagram showing how the light emitting points appear on the image of the projection plane. FIG. 11 is a diagram illustrating image data when a light emitting point on the image of the projection plane is captured in a corresponding blank period. FIG. 12 is a diagram illustrating an example (part 1) of the relationship between the rotation period of the color wheel and the shutter timing of the camera unit. FIG. 13 is a diagram illustrating an example (part 2) of the relationship between the rotation period of the color wheel and the shutter timing of the camera unit. FIG. 14 is a diagram illustrating an image of a captured image of each light emitting point of visible light and infrared light on the image of the projection surface. FIG. 15 is a diagram illustrating an example of an operation flow of the drive control unit of the integrated image projection apparatus. FIG. 16 is a flowchart showing an example of the operation of various processing units related to the light emission point detection process of the integrated image projection apparatus. FIG. 17 is a diagram illustrating a state when a pointer image as an example is combined with an infrared light emitting point on the projection surface. FIG. 18 is a diagram illustrating another example of the pattern image. FIG. 19 is a diagram illustrating an example of a configuration of a projection surface unit using a stress luminescent material. FIG. 20 is a diagram illustrating a state of light emission on the projection surface. FIG. 21 is a diagram for explaining the overall configuration of the image projection system of the second embodiment. FIG. 22 is a diagram for explaining a passing position on the projection surface of the infrared light irradiated by the infrared light irradiation device. FIG. 23 is a block diagram illustrating an example of a functional configuration of the integrated image projection apparatus. FIG. 24 is a diagram illustrating a state of the projection surface when a point on the image of the projection surface is pressed with an LED light and another point is touched with a finger. FIG. 25 is a diagram illustrating an image image of a captured image when the state of FIG. 24 is captured by the camera unit. FIG. 26 is a diagram illustrating an example of the luminance distribution of the instruction mark after the RRC process is performed on the image. FIG. 27 is a diagram illustrating an example of an operation flow of the drive control unit according to the second embodiment. FIG. 28 is a diagram illustrating an example of a composite video of a video projected on the projection plane unit and a pointer image. FIG. 29 is a diagram illustrating an example of functional blocks of the image processing unit according to the first modification of the second embodiment. FIG. 30 is a transition diagram of frame images stored in the partial image buffer unit. FIG. 31 is an external view of an integrated image projection device and an infrared light irradiation device provided as a separate body. FIG. 32 is an external view of a projection unit, a camera unit, and an infrared light irradiation device. FIG. 33 is a diagram illustrating an arrangement example of a projection unit, a camera unit, and an infrared light irradiation device. FIG. 34 is a diagram illustrating an example of the configuration of the camera unit. FIG. 35 is a diagram illustrating an example of hardware blocks of a computer. FIG. 36 is a diagram illustrating an example of functional blocks of the projection unit and the camera unit (computer). FIG. 37 is a diagram illustrating an example of an operation flow of the camera unit. FIG. 38 is a diagram illustrating an example of a configuration of a projection unit, a camera unit, and an infrared light irradiation device as a modification of the third embodiment.

(First embodiment)
Hereinafter, embodiments of an image projection system, an image processing apparatus, and a program will be described in detail with reference to the accompanying drawings.
FIG. 1 is a diagram for explaining the overall configuration of the image projection system of the first embodiment. An image projection system 1 according to the first embodiment shown in FIG. 1 includes an integrated image projection apparatus (image processing apparatus) 10, a first laser pointer 11, a second laser pointer 12, and a projection surface unit 13. Including.

  The integrated image projection apparatus 10 includes a projection unit 10a that projects an image such as a still image or a moving image on the projection surface unit 13, and a camera unit (imaging unit) 10b that captures an image on the projection surface unit 13 projected by the projection unit 10a. It has. Video data of a video projected by the projection unit 10a is received via communication from an external PC (Personal Computer) or the like. In addition, it may be reproduced from a recording medium such as a flash memory for storing video data. The camera unit 10b includes an area image sensor that can image a visible light region and an invisible light region as an image sensor. The area image sensor is configured by a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like. In the present embodiment, the invisible light region is described as an infrared region (particularly, a near infrared region) as an example.

  Although not particularly illustrated, the integrated image projector 10 includes display input means such as operation buttons and a liquid crystal display screen, or a touch-input type liquid crystal display screen. The display input means designates a pattern image for calibration (see FIG. 6) described later, sets various modes (see FIG. 15), and wavelength regions (red, green, blue, red) of the various laser pointers 11 and 12. This is to allow the user to specify "External etc."

  The first laser pointer 11 and the second laser pointer 12 are a combination example of the two irradiation devices of the first embodiment. The first laser pointer 11 is an example of an irradiation device for visible light, and is an irradiation device that oscillates light of primary colors red, green, blue, or colors of those systems. The second laser pointer 12 is an example of an irradiation device for invisible light, and is an irradiation device that oscillates light in an infrared region (infrared light). These are battery powered and used by the user U.

  The projection surface unit 13 is various screens such as a mat screen, a bar screen, a silver screen, and a bead screen. In the present embodiment, an example in which any one of various screens is provided as the projection surface unit 13 is shown. Instead of the projection surface unit 13, a screen other than a dedicated screen such as a white wall or floor may be used as the projection destination.

  FIG. 2 is a diagram illustrating an internal configuration of hardware of the projection unit 10a. As shown in FIG. 2, the projection unit 10a includes an optical system 3a and a projection system 3b. The optical system 3 a includes a color wheel 5, a light tunnel 6, a relay lens 7, a plane mirror 8, and a concave mirror 9. Further, a DMD (Digital Micromirror Device) 2 is provided in the projection unit 10a.

  The disc-shaped color wheel 5 has a plurality of color filters. By rotating, the color wheel 5 converts white light from a light source 4 such as a high-pressure mercury lamp into time-division light in which each color of RGB is repeated and emits the light. The light emitted from the color wheel 5 is emitted toward the light tunnel 6. When the color filter of the color wheel 5 is set by appropriately combining RGB colors in addition to the RGB colors, the color wheel 5 is a time-division method in which white light is repeated in a color corresponding to each filter. Convert to light. The details of the color wheel 5 will be described later using an example in which a color filter is configured by appropriately combining RGB colors.

  The light tunnel 6 is formed in a cylindrical shape by laminating plate glasses, and guides the light emitted from the color wheel 5 to the relay lens 7. The relay lens 7 is configured by combining two lenses, and collects light while correcting axial chromatic aberration of light emitted from the light tunnel 6. The plane mirror 8 and the concave mirror 9 reflect the light emitted from the relay lens 7 and guide it to the DMD 2 to collect it.

  The DMD 2 has a large number of pixel-specific micromirrors, and each micromirror is individually time-division driven based on video data to reflect light necessary for video toward the projection system 3b. Specifically, an OFF light plate is provided in the optical system 3a to receive light that is not used for video out of light incident on the DMD 2. The DMD 2 drives each micromirror on / off individually in a time-sharing manner based on the video data, reflects light not used for video among incident light to the OFF light plate side, and uses light used for video in the projection system 3b. Reflects toward the projection lens side. As a result, light indicating a color plate of each RGB color or a color plate of each color formed by combining RGB colors is sequentially generated at a time interval of a minute time (for example, microsecond order), and each light is projected in a time-sharing manner. Incident on 3b.

  The projection system 3b enlarges and projects incident light through a plurality of projection lenses. As a result, the images of the respective color plates are sequentially projected on the projection surface unit 13, and these color plates are visually synthesized as a video.

  FIG. 3 is a diagram illustrating an example of an internal configuration of hardware of the camera unit 10b. As shown in FIG. 3, the camera unit 10b includes an optical system unit 31, a color CCD image sensor 32, an A / D conversion unit 33, a shutter 34, and the like.

  The optical system unit 31 is an optical lens group that collects the light of the projection surface unit 13 and forms an image of the collected light on the color CCD image sensor 32.

  The color CCD image sensor 32 is an area sensor in which light sensors (photodiodes and the like) for each color of RGB are arranged in a matrix. Specifically, the color CCD image sensor 32 includes an imaging area in which optical sensors (such as photodiodes) that sense light in the visible region and infrared region are arranged in a matrix. Further, on the imaging area, a color filter is provided in which color filters for each color of RGB are provided in a Bayer arrangement or the like so as to correspond to each optical sensor. In this embodiment, in order to allow infrared light to pass, an infrared filter that blocks the passage of infrared light is not provided in the color filter.

  In the color CCD image sensor 32, the light of the projection surface 13 formed in the imaging area is split into RGB colors via the color filter, and the split RGB light and infrared light passing through the color filter are transmitted. Is absorbed by the optical sensor. The optical sensor stores the electric charge by converting the absorbed light into an electric signal. Then, the electric charge stored in each photosensor is output at a predetermined timing, and digitally converted by the A / D converter 33. Thereby, color image data (color digital image data) is generated.

FIG. 4 is a diagram illustrating an example of spectral sensitivity characteristics of a color CCD image sensor.
A graph curve g1 shown in FIG. 4 indicates the sensitivity characteristic of the photosensor corresponding to the blue color filter. A graph curve g2 indicates the sensitivity characteristic of the photosensor corresponding to the green color filter. A graph curve g3 indicates the sensitivity characteristic of the photosensor corresponding to the red color filter. Each graph curve g1, g2, g3 shows high sensitivity at the wavelength indicated by the corresponding color in the visible light region, and extremely low sensitivity at other wavelengths. Further, in the invisible light region, the sensitivity is high at a wavelength near 870 nm in the near infrared.

  In addition, in FIG. 3, although the camera part 10b was shown as a thing of a single plate type structure, it is not this limitation. For example, it is good also as a thing of the 3 plate type structure which provided each area sensor of each color of RGB. In this case, for example, a dichroic prism is provided in the optical system unit 31, for example, so that incident light is split into RGB colors and each is incident on a corresponding area sensor. Further, in addition to the RGB color area sensors, a monochrome area sensor for invisible light may be further provided. In this case, in the dichroic prism, visible light and infrared light are dispersed with a predetermined wavelength as a boundary, and light such as infrared light is incident on a monochrome area sensor. In addition, the configuration may be changed as appropriate.

  The shutter 34 in FIG. 3 is a mechanical shutter for adjusting the exposure time and exposure timing for each optical sensor in the imaging area. The shutter 34 opens and closes under the control of a drive control unit 22 (see FIG. 5), which will be described later, and controls the passage time (exposure time) and passage timing (exposure timing) of light into the imaging area.

  FIG. 5 is a block diagram showing a functional configuration of the integrated image projector 10. As shown in FIG. 5, the integrated image projection apparatus 10 includes a video processing unit (including a synthesis unit) 21, a drive control unit 22, a video signal input unit 23, a conversion coefficient calculation unit 24, and an instruction mark detection unit (discrimination unit). 25, an image processing unit 26, and the like.

  The video processing unit 21 receives a projection video signal (first video signal), and performs arbitrary digital image processing on the first video signal for each frame. Then, the signal after the digital image processing is output to the drive control unit 22. Specifically, the video processing unit 21 performs, as digital image processing, contrast, brightness, saturation, hue, RGB gain, sharpness, enlargement / reduction, or correction processing according to the characteristics of the drive control unit 22. .

  Further, when the second video signal is input from the image processing unit 26, the video processing unit 21 superimposes the second video signal on the first video signal. For example, it is assumed that a frame image including a predetermined partial image is input as the second video signal. In that case, the partial image included in the frame image of the second video signal is superimposed on the input frame of the first video signal by combining it with the corresponding position of the input frame. For example, the RGB information of the corresponding position of the input frame is replaced with the RGB information of the partial image, or the RGB information of the corresponding position transparently shows the partial image so that the image information of the corresponding position of the input frame remains. Implement it by replacing it with information.

  Then, the video processing unit 21 performs arbitrary image processing for each frame on the superimposed video signal (third video signal), and outputs the digital image processed signal to the drive control unit 22. The video processing unit 21 holds the frame image of the predetermined partial image input from the image processing unit 26 for a predetermined period until the frame image of the next partial image is input, and sequentially receives the first video signal. A partial image of the held frame image is synthesized with the input frame.

  The drive control unit 22 determines the drive conditions of the color wheel 5, each micromirror of the DMD 2, the lamp power source 17 that controls the drive current of the light source 4, the camera unit 10 b, and the like according to the input signal from the video processing unit 21. decide. For example, the rotational speed of the color wheel 5, the on / off timing of each micromirror of the DMD 2, and the current value of the driving current of the lamp power source 17 are determined.

  The drive control unit 22 further determines the timing of the shutter 34 of the camera unit 10b. As for the shutter timing, the timing at which the emission points of the irradiated light of the first laser pointer 11 and the second laser pointer 12 can be imaged is determined from the rotational position information of the color wheel 5 and the like. This shutter timing will be described in detail later.

  Then, the drive control unit 22 drives and controls the color wheel 5, DMD 2, lamp power source 17, camera unit 10b, and the like according to the determined drive conditions. By this control, an image based on the first image signal, the third image signal, and the like is emitted from the optical system 3a and enlarged and projected on the projection surface unit 13 through the projection system 3b. Further, the image projected on the projection surface unit 13 is captured at a predetermined timing by the camera unit 10b.

  Further, in the present embodiment, the drive control unit 22 positions each position in the image between an output video output for projection and an input image obtained by inputting the video back from the projection surface unit 13 by imaging of the camera unit 10b. The execution of the calibration process for adjusting the positional deviation is controlled. Specifically, the drive control unit 22 instructs the conversion coefficient calculation unit 24 to execute calibration, and a pattern image (see FIG. 6) read from the built-in memory or the like by the conversion coefficient calculation unit 24 is used as a fourth video signal. Output to the video processing unit 21.

  FIG. 6 is an example of a pattern image. FIG. 6 shows a black and white rectangular pattern image P1. In addition to this color scheme, the color scheme may be changed to, for example, red white, green white, or blue white.

  When the frame of the pattern image P1 is input from the video processing unit 21 as the fourth video signal, the drive control unit 22 determines the driving condition of the pattern image P1 and controls the driving of each unit. As a result, an image of light indicating the pattern image P1 is emitted from the optical system 3a and enlarged and projected onto the projection surface unit 13 via the projection system 3b. The camera unit 10b captures an image projected on the projection surface unit 13 of the pattern image P1. In the case of the calibration operation, the drive control unit 22 performs control to continue projecting the pattern image P1 for a period necessary for imaging by the camera unit 10b. In addition, the opening / closing of the shutter 34 of the camera unit 10b is controlled so that the exposure is completed within the projection time.

  The timing for performing the calibration process should be determined as appropriate, such as when projecting images, manually starting the calibration mode, starting each time, starting the device for the first time, or before shipping the device. good.

  The video signal input unit 23 performs color conversion or the like on the color image data captured by the camera unit 10b and converts it into RGB.

  As described above, the conversion coefficient calculation unit 24 reads the pattern image P1 from the built-in memory or the like during calibration, and outputs the pattern image P1 to the video processing unit 21 as a fourth video signal. Further, the conversion coefficient calculation unit 24 coordinates (x, y) of each lattice point of the pattern image on the fourth video signal, and coordinates (x ′) of each lattice point on the image data input back from the camera unit 10b. , Y ′) and a projective transformation coefficient that links the mutual positional relationship of the respective grid points is calculated. Then, the conversion coefficient calculation unit 24 sets the calculated projective conversion coefficient in the parameter H of the image processing unit 26. The parameter H is a parameter for adjusting the position of each grid point of the second video signal (that is, the partial image) to the position of each grid point of the first video signal.

  The indication mark detection unit (discrimination unit) 25 emits each of the visible and invisible laser pointers 11 and 12 recorded as predetermined indication marks in the RGB image data generated by the video signal input unit 23. Part. That is, it is detected by color information or the like. Then, the respective position coordinates {(x1 ′, y1 ′), (x2 ′, y2 ′)} are acquired.

  The image processing unit 26 converts the position coordinates {(x1 ′, y1 ′), (x2 ′, y2 ′)} of each indication mark to {(x1, y1), ( x2, y2)}. Further, the image processing unit 26 forms a second video signal (partial image) formed by forming different unique instruction data (composite information) at each corrected position {(x1, y1), (x2, y2)}. ) And output to the video processing unit 21. In the present embodiment, a pointer image indicating an irradiation point for highlighting the visible light emitting part and the infrared light emitting part of the projection surface 13 by the laser pointers 11 and 12 is formed as the instruction data. For example, circular data for a radius z pixel centered on each coordinate {(x1, y1), (x2, y2)} is formed as the instruction data. In addition, the second video signal may be generated by arranging a pointer image registered in advance at the corrected position.

Next, a flow of processing for detecting visible light and invisible light emitting units by the instruction mark detection unit 25 will be described.
FIG. 7 is a diagram illustrating an example of a color arrangement when the color wheel 5 is configured with seven color filters. As an example, one having seven color filters of Cy-W-R-M-Y-G-B is shown. In addition to the RGB colors, a color filter formed by appropriately combining RGB colors is provided. In the figure, the color wheel 5 rotates counterclockwise, and the white light from the light source 4 passes through the color filter in the order of the Red segment, the Magenta segment, and the Yellow segment.

  FIG. 8 is a diagram showing the correlation between the seven color filters and the RGB light passing through each color filter. A solid line arrow in FIG. 8 indicates a passing color that passes through the color filter. For example, the Red segment of the seven color filters transmits Red light and restricts transmission of Green light and Blue light. Further, the secondary color Cyan segment transmits Green light and Blue light and restricts transmission of Red light. In the white of the tertiary color, all of the Red light, the Green light, and the Blue light are transmitted. The description of the color filters for the other colors is omitted, but is assumed to be as illustrated.

  FIG. 9 is a diagram illustrating a projection period of light colors (projected light colors) projected through each segment in one rotation (one cycle) of the color wheel 5. A period (corresponding period) indicated by a solid arrow in the drawing indicates a projection period of the projection light color projected through the segment of the color wheel 5 corresponding to the period. For example, pay attention to the projected light color Green. The segments that allow the light of the projected light color Green to pass through are the yellow segment, the green segment, the cyan segment, and the white segment as shown in FIG. Therefore, as shown in FIG. 9, Green is projected as a projected light color in a continuous period of the Yellow segment and the Green segment, and in a continuous period of the Cyan segment and the White segment after a certain period.

  On the other hand, a period indicated by a broken-line arrow in the figure indicates a blank period in which the corresponding color light is not included in the projected light color. The blank period corresponding to the Green light is indicated by a dashed arrow of TimingB. As shown in FIG. 8, the Red segment, the Magenta segment, and the Blue segment do not pass the light of the projection light color Green. For this reason, the period corresponding to the Red segment, the Magenta segment, and the Blue segment shown in FIG. 9 is a green blank period in which green light is not included in the projected light color. During the green blank period, an image that does not use the green plate is formed on the projection plane 13. That is, when the first laser pointer 11 is irradiated to the projection surface portion 13 to form, for example, a green light emission point, if the green light is detected during the blanking period of Timing B, the detected light is transmitted to the first laser pointer 11. It can be specified as a light emitting point.

The specification of the light emitting point will be described in more detail below with reference to FIGS.
FIG. 10 shows how a light emitting point on the projection surface 13 is seen by human eyes when the image of the first laser pointer 11 is irradiated with red, green, and blue light on a certain image projected on the projection surface 13. FIG. FIG. 10 shows how the light emitting points of each color appear when the image near the position indicated by the light emitting points is Black, Red, Green, Blue, and White as an example. Black is black expressed by a luminance difference from the surroundings without irradiating light. As shown in the figure, when the color of the image is black, the light emission point of each color of the first laser pointer 11 can be identified by human eyes, but when the color of the image is other colors, the light emission point is identified. It becomes difficult.

  FIG. 11 shows a case in which red, green, and blue light of the first laser pointer 11 is irradiated on a certain image projected on the projection plane unit 13, and the projection plane unit 13 is camerad during a blank period (Timing A, B, C). It is image data when it images in the part 10b. The first color plane image data G1 shown in “TimingA Red Shot Image” in FIG. 11 is Red image data obtained by imaging the projection plane portion 13 during the blank period of TimingA (see FIG. 9). Black, Red, Green, Blue, and White shown below the first color plane image data (Red image data) G <b> 1 indicate video colors of the projection surface unit 13.

  As described above, the Red plane image is not projected onto the projection plane 13 during the blanking period of Timing A. For this reason, in the captured first color plane image data (Red image data) G1, the indication mark indicating the red light emission point of the first laser pointer 11 shown as a dot in the drawing depends on the projected image at that time. It emerges from the background without As for the position where the video color is Red, the Red plate is not actually projected, but for comparison with the other color plate image data G2 and G3, the light emission point is shown as rising. In the present embodiment, an instruction mark emerging from the background is detected and specified as a red light emission point of the first laser pointer 11.

  The first color plane image data (Red image data) G1 is expressed in grayscale in monochrome in the same figure, but is expressed in grayscale in red. The luminance of the light emitting point and the background is high at the light emitting point and low at the background.

  Next, the second color plane image data G2 shown in “TimingB Green Shot Image” in FIG. 11 is Green image data obtained by imaging in the blank period of TimingB (see FIG. 9). The green image data can be described in the same manner as the red image data. In other words, the green image is not projected on the projection surface 13 during the blanking period of Timing B. Therefore, in the captured second color image data (Green image data) G2, an indication mark indicating the green light emission point of the first laser pointer 11 shown as a dot in the drawing depends on the projected image at that time. It emerges from the background without As for the position where the video color is Green, the Green plate is not actually projected, but the light emission point is shown as being raised for comparison with the other color plate image data G1 and G3. In the present embodiment, an instruction mark emerging from the background is detected and specified as a green light emitting point of the first laser pointer 11.

Next, the third color plane image data G3 shown in “Timing C Blue Shot Image” in FIG. 11 is Blue image data obtained by imaging in the blank period of Timing C (see FIG. 9). Blue image data can also be described in the same manner as Red image data and Green image data. In order to repeat the description, a part of the description is omitted, but in the present embodiment, the instruction mark floating in the third color image data (Blue image data) G3 is detected, and the first laser pointer 11 It is specified as a blue light emitting point.
As described above, in the present embodiment, the color obtained by imaging the light emitting point formed by the irradiation of the first laser pointer 11 on the image of the projection surface unit 13 in the blank period of the projection light color corresponding to the light emission color. Detect from the image data of the plate.

  On the other hand, regarding the infrared light emitting points formed by irradiating the projection surface 13 with the second laser pointer 12, all of the red, green, and blue photosensors of the camera unit 10b are in addition to the corresponding colors. Infrared light in the invisible light region can be sensed. Therefore, the emission point of the infrared light can be detected from all the image data of the R, G, and B colors captured in an arbitrary blank period of R, G, and B colors.

  Next, the relationship between the rotation period of the color wheel 5 and the shutter timing of the camera unit 10b will be described. 12 and 13 are diagrams showing an example of the relationship between the rotation period of the color wheel 5 and the shutter timing of the camera unit 10b. Here, the color wheel 5 rotates at a cycle of 120 Hz, and the camera unit 10b generates captured image data at a cycle of 30 Hz.

  FIG. 12 is an example (example 1) showing the shutter timing of the camera unit 10b when the laser pointer forms a light emission point of green (or green color) on the projection surface. The period B shown in the figure corresponds to the green blank period (TimingB corresponding to the period of the Red segment and the Magenta segment) shown in FIG. 9. The C period corresponds to the blue blank period (Timing C corresponding to the period of the Yellow segment and the Green segment) illustrated in FIG. 9. The A period corresponds to the Red blank period (Timing A corresponding to the period of the Blue segment and the Cyan segment) illustrated in FIG. 9. The period D corresponds to the period of the White segment not including the blank period.

  The drive control unit 22 outputs a control signal for opening and closing the shutter 34 of the camera unit 10b to the camera unit 10b at the timing of the period B shown in FIG. Specifically, the drive control unit 22 outputs a control signal for opening the shutter 34 after the start of the B period to the camera unit 10b, and transmits a control signal for closing the shutter 34 to the camera unit 10b before the end of the B period. In this example, every time the color wheel 5 rotates four times, the camera unit 10b can output one captured image data. Therefore, the shutter 34 is opened and closed during the second B period of the color wheel 5 in each cycle of the camera unit 10b. A control signal is transmitted.

  Thereby, as shown in the cycle of the camera unit in FIG. 12, the camera unit 10b outputs one piece of color image data every four rotations of the color wheel 5 with the B period as an exposure time. The green light emission point of the first laser pointer 11 can be detected by observing the green image data. Further, when infrared light is formed on the projection surface portion 13 by irradiation of the second laser pointer 12, a light emission point by infrared light can be detected from the Green image data.

FIG. 13 is an example (example 2) showing the shutter timing of the camera unit 10b when the laser pointer forms a light emitting point of green (or green color) on the projection surface.
The drive control unit 22 outputs a control signal for opening and closing the shutter 34 of the camera unit 10b to the camera unit 10b at the timing of each period shown in FIG. Specifically, in the first cycle of the camera operation cycle, a control signal for performing an opening / closing operation within the period B is output to the camera unit 10b. As a result, the camera unit 10b outputs one color image data in the first period with the B period as the exposure time. From the output color image data, the green light emission point of the first laser pointer 11 can be detected by observing the Green image data. Further, when infrared light is formed on the projection surface portion 13 by irradiation of the second laser pointer 12, a light emission point by infrared light can be detected from the Green image data.

  In the subsequent second period, a control signal for performing an opening / closing operation within the C period is output to the camera unit 10b. As a result, the camera unit 10b outputs another piece of color image data in the second period with the C period as the exposure time. From the output color image data, a light emitting point by infrared light can be detected by observing the blue image data therein.

  In the subsequent third cycle, a control signal for performing an opening / closing operation within the A period is output to the camera unit 10b. As a result, the camera unit 10b outputs another piece of color image data in the third period with the period A as the exposure time. From the output color image data, a light emission point by infrared light can be detected by observing the Red image data.

  Similarly, after the fourth period, the camera unit 10b repeats the output of the color image data by repeating the opening / closing operation of the shutter 34 in the order of the B period, the C period, and the A period.

  When the color image data obtained in this way is observed, the green light emission point of the first laser pointer 11 can be detected from the green image data in the first cycle. In addition, when the second laser pointer 12 irradiates the projection surface unit 13 with infrared light, the emission point by infrared light can be detected from the image data of the first to third periods. .

  Note that, here, as an example, the green light emission point of the first laser pointer 11 is detected. However, other colors can be detected in the same manner. For example, in the case of a red light emitting point, it can be detected by observing Red image data of color image data output with the A period of the third cycle as the exposure time. In the case of a blue light emitting point, it can be detected by observing the blue image data of the color image data output with the C period of the second cycle as the exposure time. Similarly, infrared light can be detected from the color image data output with each period as an exposure time.

  The rotation of the color wheel 5 and the shutter timing are synchronized, for example, a black seal serving as an index for detecting rotation in the color wheel 5 itself, and a sensor for detecting the black seal in the holder of the color wheel 5 are provided. And realized. In this case, the drive control unit 22 is caused to acquire the timing at which the black seal is detected from the sensor, and output a signal for controlling opening and closing of the shutter.

Next, the detection process of the light emission point of visible light and the light emission point of invisible light (infrared light as an example) in the instruction mark detection unit 25 will be described.
FIG. 14 is a diagram illustrating an image image of an image captured by imaging when both a visible light emission point and an infrared light emission point are formed on the image of the projection surface unit 13. Here, as an example, the case where the emission point of visible light is green is shown.

  An image G11 shown in FIG. 14 shows green version image data of color image data captured using the B period (see FIG. 12) of the first operation cycle of the camera as the exposure time. An image G12 shows blue version image data of color image data picked up with the C period of the second cycle as the exposure time. An image G10 shows red image data of color image data captured with the A period of the third cycle as the exposure time. In the figure, it is assumed that a bright part represents a part with high luminance and a dark part represents a part with low luminance.

  Each of the images G10 to G12 in FIG. 14 includes an instruction mark M1 having substantially the same shape at a substantially same position as an area with high luminance. The instruction mark detection unit 25 detects this instruction mark M1. For example, the instruction mark detection unit 25 extracts areas where the luminance exceeds a predetermined threshold from the image data of each color plane shown as the images G10 to G12, and compares the positions of the areas (such as the center of gravity position) with each other. Then, when the degree of coincidence of the positions of the areas exceeds a predetermined level, the instruction mark detection unit 25 specifies the extracted area as an instruction mark area indicating an infrared light emission point.

  In addition, the green image data shown as the image G11 in FIG. 14 includes an instruction mark M2 in addition to the instruction mark M1 as a high-luminance area. The instruction mark detection unit 25 further detects the instruction mark M2. For example, the instruction mark detection unit 25 extracts an area where the luminance exceeds a predetermined threshold value from Green image data shown as the image G11. Then, an area other than the instruction mark M1 in the extracted one is specified as an area of the instruction mark M2 indicating a green light emitting point. In this way, the color image is used, that is, the color information is used to identify the infrared and green emission points from the image data.

  In the example shown in FIG. 14, an area with high luminance is detected as the area of the instruction mark, and the instruction mark included in each of the RGB color image data is emitted as invisible light (infrared light in this example). It was discriminated as an instruction mark M1 indicating a point. In addition, the indication mark included in one version (the Green version in this example) was discriminated as an indication mark M2 indicating a light emission point of visible light (green light in this example). However, the method for discriminating between visible light and invisible light on the image data formed by irradiation of the irradiation device is not limited to this.

  For example, after the indication mark is detected by extracting an area with high luminance, the shape of each indication mark may be compared to distinguish between the visible light emission point and the invisible light emission point. In this case, it is possible to specify whether the light emission point is visible light or invisible light with one version. The shape of the visible light emission point and the shape of the invisible light emission point are registered in advance. In addition, the shapes of the light emitting points are different from each other so that they can be distinguished. For example, by making the shape of one light emitting point a small circle, the shape of the other light emitting point being a large circle, or by making the shape of one light emitting point a circle and the shape of the other light emitting point a star shape, etc. , Make each shape different.

  Thus, the instruction mark detection unit 25 detects the light emitting unit recorded as the instruction mark in the RGB image data generated by the video signal input unit 23. Then, the respective position coordinates {(x1 ′, y1 ′), (x2 ′, y2 ′)} are acquired. The position coordinates are, for example, the coordinates of the barycentric position of the area indicating the instruction mark.

  The image processing unit 26 converts the position coordinates {(x1 ′, y1 ′), (x2 ′, y2 ′)} of each indication mark {(x1, y1), (x2, x2) according to the projective transformation coefficient set in the parameter H. y2)}. Further, the image processing unit 26 generates a frame image formed by forming different instruction data (pointer image or the like) at the corrected position {(x1, y1), (x2, y2)}. The video signal (partial image) is output to the video processing unit 21.

  As described above, if it is possible to identify the visible light emission point or the invisible light emission point in the first version, the partial image can be supplied to the video processing unit 21 in a shorter cycle than that in the third version. .

Next, the operation of the integrated image projector 10 will be described.
Here, as an example, the entire operation of the integrated image projection apparatus 10 will be described on the assumption that a calibration operation is performed during video reproduction after activation. In addition, an operation example in the case where the emission points of visible light and invisible light are specified in one version (see FIG. 12) is shown.

FIG. 15 is an operation flowchart of the drive control unit 22 of the integrated image projector 10.
In the processing shown in FIG. 15, the video signal is processed in units of one frame. First, the drive control unit 22 projects a video signal (first video signal) output from the video processing unit 21 (step S101). Specifically, the drive control unit 22 receives the first video signal for one frame input from the input I / F and output from the video processing unit 21 by controlling the rotation of the color wheel 5, the micromirror of the DMD 2, etc. 13 to project. In the case of a moving image, the first few frames of the first video signal are not included in the main video, for example, by empty frames.

  Next, the drive control unit 22 determines whether or not the light emission point detection mode is set (step S102). The light emission point detection mode is a light emission point of visible light or invisible light when the projection surface unit 13 is irradiated with light using an irradiation device such as the first laser pointer 11 (or the second laser pointer 12). This is a mode for detection. The light emission point detection mode is set by, for example, the user U setting in advance a light emission color of the irradiation device, a flag for designating execution of the mode, or the like on an operation screen (not shown) of the main body device. The drive control unit 22 reads the flag setting and activates the light emission point detection mode. When it is determined that the mode is not the light emission point detection mode (step S102: No), the process returns to step S101, and the first video signal is sequentially projected from the second frame.

  On the other hand, when it is determined that the light emission point detection mode is set (step S102: Yes), the drive control unit 22 determines whether or not the mode is the initial setting mode (step S103). The initial setting mode is a process for calculating a projective transformation coefficient when the parameter H is not set. A flag or the like is set in advance to be executed when the light emission point detection mode is first activated. Note that the determination as to whether or not to execute the initial setting mode may be performed by other methods. For example, it may be determined whether or not a fixed time has elapsed since the projective transformation coefficient was set from timekeeping information such as a timer (not shown), and may be executed when the fixed time has elapsed. Further, whether or not the installation environment has changed may be read from the output value of the sensor or the like, and the initial setting mode may be executed when the installation environment has changed.

  Here, when it is determined that the initial setting mode is set (step S103: Yes), the drive control unit 22 drives the DMD 2 and the like, and projects a pattern image P1 for projective transformation (see FIG. 6) (step). S104). Specifically, in this example, the drive control unit 22 stops the transfer of the first video signal of the second frame from the video processing unit 21, and then instructs the conversion coefficient calculation unit 24 to read the pattern image P1. . The conversion coefficient calculation unit 24 reads the designated pattern image P1 from the built-in memory, and outputs it to the video processing unit 21 as a fourth video signal. The video processing unit 21 outputs the fourth video signal to the drive control unit 22 as a frame. Then, the drive control unit 22 projects the pattern image P1 on the projection surface unit 13 for a certain period (time necessary for imaging) by controlling the rotation of the color wheel 5, the micromirror of the DMD 2, and the like.

  Next, the drive control unit 22 drives and controls the camera unit 10b to capture an image of the pattern image P1 on the projection surface unit 13 (step S105). As a result, color image data is output from the camera unit 10 b and converted into RGB by the video signal input unit 23.

  Next, the conversion coefficient calculation unit 24 calculates a projective conversion coefficient from the comparison between the pattern image P1 and the RGB image data, and sets the calculated value in the parameter H of the image processing unit 26 (step S106). In this example, the conversion coefficient calculation unit 24 sets a value for the parameter H, and then outputs a restart signal to the video processing unit 21 to restart the transfer of the stopped first video signal.

  When the calibration operation is completed as described above, the following projection operation is performed in the subsequent processing after the second frame.

  First, the drive control unit 22 projects the second frame of the first video signal on the projection surface unit 13 (step S101).

  Next, following step S102 (Yes determination) and step S103 (No determination), the drive control unit 22 determines whether the rotation speed K (K is a positive integer) of the color wheel 5 is a multiple of a predetermined value. (Step S107). The rotational speed K of the color wheel 5 is acquired by, for example, a counter that counts up every time the color wheel 5 rotates once. The default value for the counter rotation speed K is “4”. Further, the rotation order of the color wheel 5 is Red, Magenta, Yellow,... (See FIG. 9), and the position of Red is detected from the output of the sensor.

  In step S107, since the value of the first rotation speed K is the default value “4”, a Yes determination is made. Then, the drive control unit 22 outputs a control signal for opening and closing the shutter at a predetermined timing to the camera unit 10b (step S108). The drive control unit 22 adjusts the shutter timing based on the emission color (red, green, blue, infrared, etc.) of the irradiation device set by the user U on the operation screen of the main device.

  For example, it is assumed that the user U sets green and infrared as the emission colors of the irradiation apparatus on the operation screen of the main body apparatus. In that case, in the example shown in FIG. 12, a control signal for opening and closing the shutter is output to the camera unit 10b in Timing B of the second period of the color wheel 5 (second frame of the video). As a result, the camera unit 10b exposes the light of the projection surface unit 13 within the timing B in the imaging area.

  After the process of step S108, the drive control unit 22 increments the value of the rotation speed K by one (step S109). That is, at this stage, the value of the rotation speed K is changed from “4” to “5”. After the process of step S109, the process proceeds to step S101, and the drive control unit 22 projects the third frame of the first video signal on the projection surface unit 13.

  In the cycle of the third frame, the rotational speed K is “5” in step S107 and does not become a multiple of 4, so a No determination is made. If the determination is No, the value of the rotation speed K is incremented by 1 in step S109, and the process proceeds to step S101. In step S <b> 101, the drive control unit 22 projects the fourth frame of the first video signal on the projection surface unit 13.

  In the cycle of the fourth frame and the subsequent cycle of the fifth frame surface, the values of the rotation speed K are “6” and “7”, respectively, and are not a multiple of 4. For this reason, as in the cycle of the third frame, No determination is made in step S107.

  In the cycle of the sixth frame, the value of the rotation speed K is “8”, which is a multiple of 4. For this reason, it becomes Yes determination again in step S107. Then, the drive control unit 22 outputs a control signal for opening and closing the shutter with Timing B to the camera unit 10b (step S108). In the example shown in FIG. 12, this corresponds to Timing B of the sixth period (sixth frame of the video) of the color wheel 5.

  Similarly thereafter, step S101 to step S103 and step S107 to step S109 are repeated, and the drive control unit 22 outputs a control signal for Timing B to the camera unit 10b every four frames.

  The processing described above ends when, for example, the video signal to be projected ends in step S101.

  In addition, after the output of the control signal in Timing B, the third video signal in which the second video signal (partial image) is superimposed on the first video signal is appropriately input by the video processing unit 21. Become. Therefore, in step S <b> 101, the drive control unit 22 projects the third video signal in which the partial image is superimposed on the first video signal on the projection surface unit 13. This process will be described in the next operation flow.

FIG. 16 is an operation flowchart of various processing units related to the light emission point detection process of the integrated image projection apparatus 10.
First, the camera unit 10b outputs the color image data generated using the B period as the exposure time once every four cycles of the color wheel 5 (step S110). The color image data output from the camera unit 10 b is converted into RGB image data by the video signal input unit 23 and input to the instruction mark detection unit 25.

  Next, the instruction mark detection unit 25 detects the green light emission point and the infrared light emission point for the green image data of the RGB image data (step S111). In this detection, the instruction mark detection unit 25 first extracts areas with high luminance to identify two instruction mark areas. Further, the registered shape pattern is compared with the shape pattern of the indication mark. For example, it is assumed that the registered indication mark for the green emission point is a small circle and the indication mark for the infrared emission point is a large circle. In that case, the radii of the two specified indicator marks are compared, and the smaller radius is discriminated as an indicator mark indicating a green light emitting point, and the other is discriminated as an indicator mark indicating an infrared light emitting point. Then, the instruction mark detection unit 25 outputs, for example, data indicating each barycentric coordinate to the image processing unit 26 as the position of each identified instruction mark.

  Next, the image processing unit 26 corrects the barycentric coordinates to the corresponding position on the first video signal by the parameter H (step S112). Then, the image processing unit 26 generates a frame image in which a pointer image or the like is set as predetermined data (partial image) at each correction position, and outputs the frame image to the video processing unit 21 as a second video signal (step). S113).

  Then, the video processing unit 21 synthesizes a partial image of the second video signal with a predetermined number of input frames subsequent to the first video signal, and drives and controls the combined frame image as a third video signal. It outputs to the part 22 (step S114). Here, the description has been made assuming that the partial image is synthesized with a predetermined number of the subsequent input frames, but this is because the frame in which the partial image is not generated due to the difference between the frame rate on the projection side and the frame rate on the imaging side. This is because it occurs. In this example, in order to interpolate those frames, partial images are synthesized with respect to the subsequent four input frames. Accordingly, the third video signal is continuously projected by the drive control unit 22, and a pointer image or the like is smoothly displayed on each light emitting point on the projection surface unit 13. This is not the case when there is no difference between the frame rate on the projection side and the frame rate on the imaging side. In this case, there is no need for interpolation because no frame is generated in which no partial image is generated.

  FIG. 17 is an example showing a composite image of a video projected on the projection plane unit 13 and a partial image. FIG. 17A is a diagram illustrating a state when a pointer image is combined with an infrared light emitting point on the projection surface unit 13. In the example of FIG. 17A, an image Q1 indicating an arrow is synthesized as a pointer image. FIG. 17B is a diagram illustrating a state in which a pointer image is combined with a green light emitting point when the first laser pointer 11 is further irradiated with green light in the state illustrated in FIG. . In the example of FIG. 17B, the pointer image is obtained by combining the enlarged image Q2 of the pointer with the green light emitting point.

  As described above, when the projection surface where the image is projected is irradiated with a laser pointer or the like for visible light and invisible light, each light emitting part is discriminated as a light emitting part for visible light or a light emitting part for invisible light, and each light emission Separate partial images can be superimposed on the part. For this reason, visible light and invisible light laser pointers can be used separately. In this embodiment, an example in which a pointer image such as an arrow image or a magnified image of a pointer is combined as instruction data is shown, but this is not restrictive. For example, an image obtained by enlarging the video at the position may be synthesized.

  In the above, an example in which visible light and invisible light are simultaneously irradiated onto the projection surface portion has been shown, but this is an example for explanation. Normally, visible light and invisible light are often used separately. In this system, it goes without saying that even if they are used separately, the type of light being used is identified and the part corresponding to that light is used. The image can be synthesized at the position of the light emitting point.

  In the present embodiment, a black and white rectangular pattern image P1 (see FIG. 6) is used as an example of the pattern image, but the pattern image pattern and color may be appropriately modified.

  FIG. 18 is another example of a pattern image. FIG. 18A and FIG. 18B show examples of two pattern images. When the pattern image shown in FIG. 18A is used, more accurate coordinates can be obtained without disturbance due to ambient light. Further, when the pattern image shown in FIG. 18B is used, even if there is a projective deformation, the center of gravity of the circle in the projection image formed by projecting the pattern image is used as the coordinate position. , You will be able to get some accurate coordinates.

(Modification of the first embodiment)
In the modification of the first embodiment, an example in which one of visible light and invisible light emission points is formed by a stress light emitter provided on the projection surface portion is shown.
FIG. 19 is an example of a configuration diagram of a projection surface portion using a stress light emitter (light emitting portion due to stress). A projection surface portion 130 shown in FIG. 19 is formed of a three-layer structure including a base member 131, a stress light emitter 132, and a protection member 133. The projection surface portion 130 is provided with a stress light emitter 132 that emits light when receiving a force on the base member 131, and the exposed surface of the stress light emitter 132 is protected by covering it with a transparent protective member 133. The projection surface unit 130 uses the protection member 133 side as a projection surface. The light emission wavelength of the stress-stimulated illuminant 132 is selected to be the same type as the wavelength of the light that is detected as the instruction mark by the imaging of the camera unit 10b. For example, those in various predetermined wavelength regions such as red, green, blue, and infrared are selected.

  FIG. 20 is a diagram illustrating a state of light emission of the projection plane unit 130. As shown in FIG. 20, when the user U presses the surface of the projection surface unit 130 with a finger (or an indicator stick or the like), the stress light emitter 132 (see FIG. 19) receives the force and emits light. The light emission point is imaged by the camera unit 10b (see FIG. 1) and detected through image processing or the like, in the same manner as the detection of the light emission point by the laser pointer.

  In addition, the structure of the projection surface part using a stress light-emitting body is not limited to this. Since the projection surface portion only needs to be composed of a stress illuminant that emits light when receiving a force, the structure thereof may be appropriately modified. For example, a structure in which the stress-stimulated light emitter 132 is sandwiched between a pair of substrates may be employed.

  As described above, in the first embodiment and the modification, an example in which two laser pointers are used as the irradiation device and an example in which one of the laser pointers is changed to a stress light emitter are shown. However, other forms may be used as long as the light emitting points of visible light and invisible light can be formed on the projection surface.

  For example, an LED light emitting device that does not perform laser oscillation, such as an LED light, may be used as the irradiation device. In this case, since the LED has a light straightness lower than that of the laser and the light easily diffuses, the LED is used by directly contacting the light emitting portion with the projection surface portion. Thereby, since a light emission part is formed on a projection surface part, it can be detected as a light emission point with a camera part.

  Alternatively, one light emitting point may be formed by a laser pointer or an LED light, and the other light emitting point may be formed by reflected or scattered light. For example, you may combine the irradiation apparatus which irradiates light to the whole range on a projection surface part. In this case, when the user designates the projection surface with a finger or an indicator stick, the light from the irradiation device hits the finger or the indicator stick and is reflected or scattered. The reflected or scattered light is imaged by the camera unit and detected as a light emitting point.

  In addition, various means for forming the light emitting points described above may be appropriately combined.

  Further, the relationship between various means and the light used (visible light or invisible light) may be interchanged. For example, if the configuration is such that the light of the irradiation device that irradiates the entire range on the projection surface is invisible light, and the other laser pointer or the like is visible light, the entire range of the irradiation device is made visible, the laser pointer, etc. The relationship of the light to be used may be exchanged, such as making invisible light.

  In the present embodiment, an integrated image projector is shown, but the present invention is not limited to this. The functions may be appropriately divided and separated into separate devices. For example, a video projection unit and a processing unit that outputs a partial image (such as a pointer image) to the projection unit by image processing may be configured separately.

  In addition, the size of the integral type and the separated type image projection apparatus may be appropriately changed according to the intended use. For example, it may be changed as appropriate according to the intended use, from a large one for in-house use to a small one for individual use. Further, the present invention may be applied to a pico projector having a size that fits in one hand such as a smartphone.

  In the first embodiment, the color projection frame is configured to be projected in a time division manner by using a color wheel in the image projection apparatus. However, the color plate frames are projected in a time division manner. The configuration is not limited to this. For example, instead of the color wheel, RGB color LEDs may be provided, and the RGB color LEDs may be irradiated onto the DMD in a time-division manner so that each color frame is time-divisionally projected.

(Second Embodiment)
In the first embodiment, a mode in which two laser pointers are used as an irradiation apparatus that forms light-emitting points of visible light and invisible light on a projection surface portion has been described. In the second embodiment, an LED light is used as an irradiating device that forms a light emitting point of visible light, and an irradiating device that irradiates invisible light to the entire range of the projection surface portion as an irradiating device that forms an invisible light emitting point. The aspect of the used image projection system is shown. Note that in the image projection system of the second embodiment, portions that overlap with the image projection system of the first embodiment are omitted from the illustration and description, and are mainly different from the image projection system of the first embodiment. Will be described.

  FIG. 21 is a diagram for explaining the overall configuration of the image projection system of the second embodiment. An image projection system 100 according to the second embodiment illustrated in FIG. 21 includes an integrated image projection apparatus (image processing apparatus) 50, an LED light 51, and a projection plane unit 13.

  The integrated image projector 50 is a hand-held image projector of a size that can be carried by a person with one hand. The integrated image projection apparatus 50 includes a small projection unit 50a and a camera unit 50b having the same functional configuration as the projection unit 10a and the camera unit 10b shown in the first embodiment. In addition, an infrared light irradiation device (invisible light irradiation unit) 50c that irradiates the entire range of the projection surface unit 13 with invisible light (in this example, infrared light) is provided. Since the specific configurations of the projection unit 50a and the camera unit 50b are substantially the same as the configurations shown in the first embodiment, illustration and description thereof will be omitted as appropriate, and mainly the configurations of the infrared light irradiation device 50c and the LED light 51 will be described. Etc. will be described.

  As shown in FIG. 21, the infrared light irradiation device 50 c is provided below the integrated image projection device 50. The integrated image projection apparatus 50 is used so that the infrared light irradiation apparatus 50c is grounded to the horizontal plane of the projection surface portion 13 as shown in FIG. The camera unit 50 b is tilted in such a state that the projection surface unit 13 can be imaged, and captures the image so as to include the projection surface unit 13. In this example, it is assumed that the angle of the camera unit 50b is fixed. However, a movable unit is provided on the upper surface of the projection unit 50a so that the angle of the camera unit 50b can be freely changed. The part 50b may be fixed. The movable part can be configured, for example, for rotating the camera part 50b.

  The infrared light irradiation device 50c irradiates infrared light from the irradiation window 500 to the projection surface unit 13 side. For example, laser light in the infrared region is irradiated as the infrared light in parallel to the projection surface unit 13 so as to pass through a certain height above the projection surface unit 13, and the red shown in FIG. 21 above the projection surface unit 13. The outside light irradiation range E is line-scanned with the laser beam.

  The LED light 51 is a battery-driven visible light bullet-type LED, and includes an LED in the vicinity of the tip portion 510. When the user pushes the tip portion 510 of the LED light 51 into the projection surface portion 13, the power switch is turned on, and the LED in the vicinity of the tip portion 510 is lit. The tip portion 510 is provided with a transmission portion that allows infrared light to pass therethrough. Here, a transparent material such as transparent glass or transparent plastic is used as an example of the material. When the LED is turned on, the light from the LED is emitted from the transmissive portion, and the transmissive portion functions as a light emitting portion.

  FIG. 22 is a diagram for explaining the passage position of the infrared light irradiated by the infrared light irradiation device 50 c when the LED light 51 is used on the projection surface unit 13. FIG. 22A shows the projection surface portion 13 from the side, and shows a state when the LED is caused to emit light by pushing the LED light 51 into the projection surface portion 13 in the arrow Y1 direction.

  As shown in the figure, the passing position on the projection surface part 13 of the infrared light (dashed arrow Y2 in the figure) irradiated by the infrared light irradiation device 50c is the range of the transparent body 510a of the tip part 510 of the LED light 51. Set within H (however, H> 0). That is, in the irradiation range of infrared light irradiated by the infrared light irradiation device 50c (infrared light irradiation range E in FIG. 21), the irradiation position J in the vertical direction of the projection surface portion 13 is within the range H (preferably the irradiation position). Limited to J << range H). Thereby, even when the user applies the LED light 51 to the projection surface portion 13 and indicates the position by light emission of the LED, the infrared light from the infrared light irradiation device 50 c passes through the LED light 51. For this reason, infrared light (broken arrow Y2 in the figure) irradiated by the infrared light irradiation device 50c is reflected or scattered by the LED light 51, thereby forming a light emitting point having a luminance value of a predetermined level or more at that position. Can be suppressed.

  FIG. 22B shows a state when the projection surface unit 13 is touched with a finger as an example of an instruction unit instead of the LED light 51 shown in FIG. As shown in the figure, infrared light (broken line arrow Y2 in the figure) is reflected, absorbed, and scattered at the position of the finger U1, and an infrared light emitting point is formed at the position of the finger U1 that blocks the infrared light. Is done. The infrared light irradiation position is preferably close to the tip of the finger U1 when the user touches the projection surface unit 13 with the finger U1. For example, it is set within the first joint from the tip of the finger U1 or within the position of the nail U2 of the finger U1. In the example shown in FIG. 22B, the irradiation position is set to the position of the nail U2 of the finger U1.

  FIG. 23 is a block diagram showing a functional configuration of the integrated image projector 50. As shown in FIG. 23, the integrated image projector 50 further includes an infrared light scanning unit 60 in the block configuration shown in the first embodiment (see FIG. 5). Further, a drive control unit 61 and an instruction mark detection unit 62 are provided.

  The infrared light scanning unit 60 activates the infrared light irradiation device 50 c according to the irradiation command of the drive control unit 61 and causes the projection surface unit 13 to emit infrared light. In this example, the irradiation light is further controlled to scan within the infrared light irradiation range E.

Here, a method of discriminating between the light emission point of visible light and the light emission point of invisible light (infrared light in this example) in the instruction mark detection unit 62 of the second embodiment will be described.
FIG. 24 is a diagram showing a state of the projection plane unit 13 when an instruction is given by pressing one point on the image of the projection plane unit 13 with the LED light 51 and the user touches another point with a finger. . That is, the example shown in FIG. 24 shows a pattern in the case where visible light and invisible infrared light are emitted simultaneously on the projection plane 13. Hereinafter, the detection process of the light emission point of the LED light 51 and the light emission point of the user's finger when the light emission point of the LED light 51 is green will be described.

  FIG. 25 is a diagram illustrating an image image of a captured image from the camera unit 50b when the state of FIG. 24 is captured by the camera unit 50b. Images G20, G21, and G22 in FIG. 25 are obtained in the same manner as the images G10, G11, and G12 (see FIG. 14) of the first embodiment. That is, each of the images G20, G21, and G22 is obtained by exposure at the timing shown in the color camera cycle of FIG. That is, the image G21 shows the Green version image data that is exposed and output in the period B by the camera unit 50b. An image G22 indicates the Blue plane image data that is exposed and output in the period C. An image G20 shows Red image data that is exposed and output in the period A. In the figure, a bright portion represents a portion with high luminance, and a dark portion represents a portion with low luminance.

  Each of the images G20 to G22 in FIG. 25 includes a crescent-shaped instruction mark M20 indicating the characteristics of a finger (or nail) at a substantially same position as an area with high luminance. Further, the green image data shown as the image G21 in FIG. 25 includes a circular instruction mark M21 of the LED light 51 in addition to the instruction mark M20 as a high luminance area. The instruction mark detection unit 62 detects these instruction marks M20 and M21 based on a luminance difference, and further discriminates each of them from a visible light instruction mark and an invisible light instruction mark. The method for discriminating each indication mark M20, M21 is also described in the first embodiment, but here it depends on the luminance pattern of the pixels that can detect and discriminate each indication mark M20, M21 in one version. The method is shown. As an example, a discrimination method using RRC (Radial Reach Correlation) processing for recognizing a shape by examining luminance differences in eight directions from a pixel of interest in image data will be described.

  FIG. 26 is a luminance distribution diagram of the instruction mark after RRC (Radial Reach Correlation) processing is performed on the Green image G21. FIG. 26A is a luminance distribution diagram of the light emission point indication mark M21 formed by the LED light 51, and FIG. 26B is a luminance distribution diagram of the light emission point indication mark M20 formed on the user's finger. . 26A and 26B, it can be seen that a difference appears in the luminance distribution between the circular instruction mark M21 and the crescent moon-shaped instruction mark M20.

  The instruction mark detection unit 62 performs RRC processing on the Green image data (image G21 in FIG. 25) input via the video signal input unit 23. Then, the instruction mark M21 of the LED light 51 and the instruction mark M20 of the user's finger are discriminated from the luminance distribution shown in FIGS. 26 (a) and 26 (b).

  Specifically, the instruction mark detection unit 62 first sets the pixel of interest (m210, m200) from the Green version of the image data, and examines the luminance values of the pixels in eight directions radially from the pixel of interest, and starts from the vicinity of the pixel of interest. In order, the luminance difference between the target pixel and each pixel is obtained. For the pixels (m211 and m201) whose luminance difference is less than the predetermined threshold, the luminance difference of the next pixel is obtained, and the pixels (m212 and m202) whose luminance difference exceeds the predetermined threshold, the radial pixel group up to that point The array pattern is a candidate for the indication mark. The instruction mark detection unit 62 performs the process on each pixel of the Green image data to generate a plurality of instruction mark candidates. The instruction mark detection unit 62 further obtains a standard deviation of the array pattern (that is, radial luminance distribution) of the candidate pixel group.

  For example, in the case of the LED light 51, since the instruction mark is often circular or elliptical, the luminance distribution has substantially the same length in eight directions as in the example shown in FIG. On the other hand, in the case of a finger, since the indication mark has a crescent shape, there is a unique variation in the length in eight directions as in the example shown in FIG. These features are obtained as standard deviation values.

  Thereafter, the indication mark detection unit 62 obtains the similarity between the indication mark M21 of the LED light 51 and the indication mark M20 of the user's finger based on the standard deviation value, and the indication mark M21 of the LED light 51 and the indication of the user's finger. The instruction mark M20 is discriminated. For the calculation of the similarity, for example, a plurality of patterns of image data (Green version) of the instruction mark M21 of the LED light 51 and the instruction mark M20 of the user's finger are stored in the internal memory in advance. The indication mark detection unit 62 obtains the similarity based on the standard deviation value calculated from the data. As a result, the instruction mark detection unit 62 uses the position coordinates {(x1 ′, y1 ′), (x2 ′, y2 ′)} of the target pixel of each pixel group specified as the instruction marks M20 and M21 as the image processing unit 26. Output to.

  As shown in the first embodiment, the image processing unit 26 sets the position coordinates {(x1 ′, y1 ′), (x2 ′, y2 ′)} of each instruction mark by the projective transformation coefficient set in the parameter H { It is corrected to (x1, y1), (x2, y2)}. Further, the image processing unit 26 generates a frame image formed by forming different instruction data at the corrected position {(x1, y1), (x2, y2)}, and uses this as a second video signal for video processing. To the unit 21.

  FIG. 27 is an operation flowchart of the drive control unit 61 according to the second embodiment. This operation flow diagram partially overlaps the operation flow diagram (see FIG. 15) of the drive control unit 22 of the first embodiment. Therefore, the description of overlapping parts is omitted here, and only different parts are described.

  In the operation flow of the drive control unit 61 of the second embodiment, the drive control unit 61 determines the activation state of the infrared light irradiation device 50c after No determination in step S103 (step S200). When the infrared light irradiation device 50c is stopped (step S200: No determination), the drive control unit 61 outputs an irradiation command to the infrared light scanning unit 60 (step S201). Thereby, the infrared light irradiation device 50c is activated and scanning of the infrared light on the projection surface unit 13 is started.

  After the process of step S201, the process proceeds to step S200, and in the subsequent cycle, the infrared light irradiation device 50c is activated (step S200: Yes determination), so the process proceeds to step S107, and the camera unit 50b Imaging on the projection plane unit 13 is started. That is, when imaging on the projection surface unit 13 by the camera unit 50b is started, infrared light has already been scanned from the infrared light irradiation device 50c to the projection surface unit 13. If the infrared light is blocked when the user points on the projection surface unit 13 with the finger by the scanning of the infrared light, the indication mark detection unit 62 detects the light emitting point on the finger. Then, a pointer image or the like is synthesized at the position of the instruction mark on the video.

  The infrared light irradiation device 50c is stopped by, for example, outputting a stop command to the infrared light scanning unit 60 when the video signal is ended in step S101.

  FIG. 28 is an example showing a composite image of a video projected on the projection plane unit 13 and a pointer image. In the example shown in FIG. 28, an image Q1 showing an arrow as a composite image is formed at the light emitting point of the finger. In addition, an enlarged image Q2 of a pointer is formed as a composite image on a green light emitting point. When the user moves the finger U1 or the LED light 51 on the projection plane 13, the image Q1 indicating an arrow or the enlarged image Q2 of the pointer moves following the movement.

  In the second embodiment, an example in which an image such as an arrow or a pointer is combined as instruction data to each of the visible light emission point and the invisible light emission point and is followed as the light emission point moves is shown. . For example, the instruction data synthesized at the position of the light emitting point may be left in the subsequent frames, and the instruction data having different positions due to the movement of the light emitting points may be sequentially added to the subsequent frames. In this case, if the instruction data is a dot, a line is formed when the light emitting point is moved on the projection surface portion 13.

(Modification 1 of 2nd Embodiment)
In the first modification of the second embodiment, a line is drawn on the projection surface unit 13 by the light emitting point formed by the LED light 51, and the line drawn on the projection surface unit 13 is erased by the light emitting point formed by the user's finger. The structure of the image process part 26 for functioning as an eraser for this is shown.

  FIG. 29 is a functional block diagram of an image processing unit according to Modification 1 of the second embodiment. As shown in FIG. 29, the image processing unit 70 includes a partial image buffer unit 700 for temporarily storing a copy of the frame image (partial image) output to the previous video processing unit.

  The image processing unit 70 is similar to the image processing unit 26 of the second embodiment (that is, as described in the image processing unit 26 of the first embodiment), and the visible light detected by the instruction mark detection unit 62. The position of the invisible light emission point is corrected by the projective transformation coefficient set in the parameter H. Further, a frame image formed by forming different instruction data (the partial image) at the corrected position is generated and output to the video processing unit 21. The partial image buffer unit 700 temporarily stores the output frame image. Thereafter, when the next position coordinates of the visible light and the invisible light (the same position coordinates as before or the position coordinates after movement) are input from the instruction mark detection unit 62, the image processing unit 70 similarly corrects the position. Furthermore, the partial image is arranged on the previous frame image temporarily stored in the partial image buffer unit 700, and the partial image buffer unit 700 is updated. Then, the image processing unit 70 outputs the updated frame image to the video processing unit 21.

  FIG. 30 is a transition diagram of frame images (partial images) stored in the partial image buffer unit 700. FIG. 30A shows the transition of the partial image when the light emitting point of the LED light 51 is moved from one point on the projection plane unit 13 to another point. As shown in FIG. 30A, the image processing unit 70 first forms a dot image D1 in order to form a line at a position corresponding to a light emitting point formed at one point on the projection surface unit 13, and the portion. The image is stored in the partial image buffer unit 700. Further, when the light emitting point of the LED light 51 is moved from one point on the projection surface unit 13 to another point, the image processing unit 700 sequentially forms dot images D2 and D3 newly formed on the dot image D1 formed previously. ... DN (N is an integer) is added. Then, the partial image in the partial image buffer unit 700 is updated with the additional image. Thereby, a line is formed in the locus of the light emitting point.

  FIG. 30B shows the transition of the partial image when the user traces the line on the projection plane 13 corresponding to the line shown in FIG. The image processing unit 70 first forms a dot image F1 for erasing the line at the corresponding position of the light emitting point formed by the light emission of the finger on the projection surface unit 13, and stores the line image stored in the partial image buffer unit 700. A dot image F1 is added to the partial image including. The pixel information of the dot image F1 is a value that cancels the pixel information of the dot image D1. Thus, when the dot images F1... FN (N is an integer) are moved over the line, the dot images D1, D2,.・ The pixel information is canceled and the line disappears.

(Modification 2 of the second embodiment)
In the second embodiment, the infrared light irradiation device 50c is provided as an integrated image projection device, but the infrared light irradiation device 50c may be provided separately.
FIG. 31 is an external view of a separate infrared light irradiation device 50c-1 and an integrated image projection device 50-1. The separate infrared light irradiation device 50c-1 is provided with a power switch (not shown) for turning on the power. The infrared light irradiation device 50c-1 is activated when the user turns on its power switch, and irradiates infrared light. In this case, it is not necessary to control the infrared light irradiation device by the drive control unit of the integrated image projection device (image processing device) 50-1.

  In the first and second embodiments described above, the image projection apparatus including the projection unit and the camera unit in one apparatus has been described. However, these are not integrated, and a part of the functions may be provided separately to constitute a separate type.

  As described above, in the second embodiment and the modified example, when the pointing means such as a finger or a pointing stick is used to directly touch and point on the projection surface, the composite information such as the pointer image is displayed at the indicated position. be able to. In the second embodiment, scanning is performed by irradiating infrared light that is not visually recognized on the projection surface portion, and a light emitting point is formed on the pointing means by the infrared light. For this reason, infrared light does not interfere with the image, and the light emitting point does not interfere with the indicated position.

(Third embodiment)
In the third embodiment, a separation type aspect of the integrated image projector is shown. Here, as an example, an example in which the projection unit 80a, the camera unit 80b, and the infrared light irradiation device 80c are configured as separate bodies is shown.
FIG. 32 is an external view of the projection unit 80a, the camera unit 80b, and the infrared light irradiation device 80c.
The projection unit 80a has the internal configuration of the projection unit 10a (see FIG. 2) shown in the first embodiment. The infrared light irradiation device 80c corresponds to the infrared light irradiation device 50c-1 (see FIG. 31) shown in the second embodiment, and includes a power switch. The infrared light irradiation device 50c-1 starts irradiation and scanning of infrared light when the user turns on the power switch. The configuration of the camera unit 80b will be described later.

  FIG. 33 is an arrangement example of each unit (projection unit 80a, camera unit 80b, and infrared light irradiation device 80c). FIG. 33 shows a state in which the projection unit 80a is placed on the infrared light irradiation device 80c and the camera unit 80b is connected to the projection unit 80a using a long bendable cable I. In this connection example, since the orientation and position of the camera unit 80b can be freely changed, the imaging by the camera unit 80b can be performed from the front of the projection surface unit. In addition, the projection surface part is provided in the projection range of the projection part 80a similarly to the image projection system 100 (refer FIG. 21) of 2nd Embodiment. A laser pointer, an LED light, or the like may be appropriately selected as the irradiation device that emits visible light.

  FIG. 34 is an example of a configuration diagram of the camera unit 80b. The separation-type camera unit 80b according to the present embodiment includes an imaging camera 800 and a computer (image processing apparatus) 801. The computer 801 is an information processing apparatus such as a personal computer.

  FIG. 35 is a hardware block diagram of the computer 801. The computer 801 includes a CPU (Central Processing Unit) 810, a ROM (Read Only Memory) 811, a RAM (Random Access Memory) 812, a memory unit 813, an operation input unit 814, a display output unit 815, a communication interface 816, and the like. Each part is connected to each other by a bus 818.

  The CPU 810 is a central processing unit that executes various programs and performs arithmetic processing and control of each unit. The ROM 811 is a non-volatile memory that stores fixed programs and data. The RAM 812 is a volatile memory used as a work area for the CPU 810. The memory unit 813 is an external storage device such as an HDD or a portable flash memory that stores various programs and data. The operation input unit 814 receives user input through connection of a keyboard, a mouse, or the like. The display output unit 815 displays setting information and the like by connecting a display display (for example, a liquid crystal display). The communication interface 816 is an interface that communicates with the projection unit 80a.

Under the above configuration, the CPU 810 loads various programs from the memory unit 813 or the like to the RAM 812 and executes them, thereby causing the computer 801 to function as the following units.
FIG. 36 is an example of a functional block diagram of the projection unit 80a and the camera unit 80b (computer 801).

  As shown in FIG. 36, the projection unit 80a includes a video processing unit 21, a drive control unit 22, a synchronization signal output unit 820, a video signal input processing unit 823, and the like. The camera unit 80b includes an imaging control unit 824, a video signal input unit 23, a conversion coefficient calculation unit 24, an instruction mark detection unit 25, an image processing unit 26, a synchronous video signal transmission unit 825, and the like. The image processing unit 26 and the synchronous video signal transmission unit 825 function as a composite information output unit. Note that functional units that are substantially the same as those in the functional block diagram of the first embodiment (see FIG. 5) will be briefly described by attaching the same reference numerals, and different parts from the first embodiment will be described in detail. .

First, the projection unit 80a will be described.
The video signal input processing unit 823 receives a digital signal such as HDMI (registered trademark) or an analog signal such as a VGA or a component signal. The video signal input processing unit 823 performs processing for processing video into RGB, YPbPr signals, and the like according to the input signals. Note that when the input video signal is a digital signal, the video signal input processing unit 823 converts the input video signal into a bit format defined by the video processing unit 21 according to the number of bits of the input signal. Further, when the input video signal is an analog signal input, the video signal input processing unit 823 performs a DAC process for digitally sampling the analog signal and inputs an RGB or YPbPr format signal to the video processing unit 21.

The video processing unit 21 outputs the video signal to the drive control unit 22. When receiving a partial image generated by the image processing unit 26 from the video signal input processing unit 823, the video processing unit 21 outputs a video obtained by synthesizing the partial images to the drive control unit 22.
The drive control unit 22 controls the DMD 2, the color wheel 5, the lamp power source 17, and the like. In addition, a signal for controlling opening and closing of the shutter in synchronization with the rotation of the color wheel 5 is output to the synchronization signal output unit 820.

  The synchronization signal output unit 820 processes a signal for controlling the opening / closing of the shutter of the drive control unit 22 according to the specifications of the external device, and outputs a timing signal corresponding to the external device to the external device.

Next, the camera unit 80b will be described.
The imaging control unit 824 detects the imaging timing from the timing signal of the synchronization signal output unit 820 input via the communication interface 816 (see FIG. 35), and controls the opening / closing operation of the shutter of the imaging camera 800. The timing of shutter opening / closing control is the same as in the first embodiment. That is, the timing of Timing A (see FIG. 9) is used when the emission light of the irradiation device is red, and Timing B (see FIG. 9) is green, and Timing C (see FIG. 9) is blue. To do. In addition, when using infrared light together as irradiation light of an irradiation apparatus, any of Timing A, B, and C may be sufficient.

  The video signal input unit 23 receives color image data output from the imaging camera 800 and converts it into RGB.

  The conversion coefficient calculation unit 24 calculates a projective conversion coefficient during the calibration operation and sets it as a parameter of the image processing unit 26. An instruction to start calibration is received from the drive control unit 22 of the projection unit 80a via the communication interface 816 (see FIG. 35). Further, after setting the projective transformation coefficient as a parameter, the transformation coefficient computing unit 24 outputs a signal indicating the end to the drive control unit 22 or the video processing unit 21 via the communication interface 816 (see FIG. 35).

  The instruction mark detection unit 25 detects an instruction mark from the RGB image.

  The image processing unit 26 corrects the position of the instruction mark with the parameter H, and forms a partial image in which a pointer image or the like is set at the corrected position.

  The synchronous video signal transmission unit 825 converts the partial image generated by the image processing unit 26 into a transmission format that conforms to the connection method with the projection unit 80a, and the like, and the projection unit 80a via the communication interface 816 (see FIG. 35). The signal is transmitted to the video signal input processing unit 823. For example, RGB signals are transmitted after being converted into YPbPr signals. When the projection unit 80a accepts only analog input, the synchronous video signal transmission unit 825 converts the partial image from a digital signal to an analog signal and transmits the partial image.

  Next, operations of the projection unit 80a and the camera unit 80b (mainly the computer 801) will be described. The operation of the drive control unit 22 of the projection unit 80a is substantially the same as the operation (see FIG. 15) of the drive control unit 22 of the first embodiment. The difference is that in step S105, the drive control unit 22 outputs a timing signal to the camera unit 80b via the synchronization signal output unit 820.

FIG. 37 is a flowchart of the camera unit 80b.
In the following description, it is assumed that the emission colors of visible light and invisible light of the irradiation device are stored as used colors in the memory unit 813 by, for example, keyboard input in advance. In this example, the respective emission colors are green and infrared. Further, the calibration operation is substantially the same as that described as the operation of the conversion coefficient calculation unit 24 in the operation flow diagram (see FIG. 15) of the first embodiment, and thus the description thereof is omitted here.

  First, the imaging control unit 824 detects an imaging timing from a timing signal received via the communication interface 816 (see FIG. 35) (step S200), and outputs an imaging instruction signal to the imaging camera 800 at the start and end timings. (Step S201).

  Next, the video signal input unit 23 acquires color image data output from the imaging camera 800 and converts it into RGB (step S202).

  Next, the instruction mark detection unit 25 detects each light emission point for one version of the RGB image data (the Green version in this example) based on the set color set in the memory unit 813, and Are distinguished into visible light and invisible light (step S203). Since the discrimination method has already been described in the first embodiment and the second embodiment, description thereof is omitted here.

  Next, the image processing unit 26 corrects the coordinates to a corresponding position on the first video signal by the parameter H (step S204). Then, the image processing unit 26 generates a frame image in which a pointer image or the like is set as predetermined data (partial image) at each correction position (step S205).

  Next, the video signal transmission unit 825 converts the frame image into a predetermined transmission signal between the projection units 80a and transmits the frame image to the video signal input processing unit 823 of the projection unit 80a via the communication interface 816 (see FIG. 35) (step). S206).

  Thereafter, in the projection unit 80a, the video processing unit 21 combines the partial image of the second video signal with a predetermined number of input frames subsequent to the first video signal, and the combined frame image is converted into the third frame image. This is output to the drive control unit 22 as a video signal.

  Note that the computer 801 that is an information processing apparatus may perform a part of processing by an external information processing apparatus connected through a network. For example, an initial projective transformation matrix calculation may be performed using a high-performance calculation processing server placed on a network, or superimposition synthesis information may be downloaded and used during image processing.

(Modification of the third embodiment)
In the third embodiment, the separated type devices are shown as being connected by wire, but the devices may be wirelessly connected.

  FIG. 38 is an example of a configuration diagram of a projection unit 90a, a camera unit 90b, and an infrared light irradiation device 90c as a modification of the third embodiment. Here, only the wireless communication unit is shown, and the other components are not shown.

  As shown in FIG. 38, wireless communication units 900, 901, and 902 are provided in each device of the projection unit 90a, the camera unit 90b, and the infrared light irradiation device 90c. The wireless communication unit is a short-range wireless communication unit such as Bluetooth (registered trademark) or Wi-Fi, for example. Each device communicates via a respective wireless communication unit. The wireless communication unit 902 of the infrared light irradiation device 90c communicates with the wireless communication unit 900 of the projection unit 90a, and receives signals such as an infrared light irradiation command and an irradiation stop command from the projection unit 90a. The wireless communication unit 900 of the projection unit 90a and the wireless communication unit 901 of the camera unit 90b transmit and receive video signals, timing signals, and the like.

By using wireless communication, the following modes can be taken. For example, an information device such as a smartphone incorporating a camera can be used as the camera unit 90b.
In this case, an information device such as a smartphone is previously downloaded with a program for controlling the imaging camera and generating a partial image, as shown in FIG. Then, the information device is wirelessly connected to the projection unit 90a, and the information device is used for calculation processing such as partial image processing.

  As described above, in each embodiment and modification, each light emitting point is detected when the image projected on the projection surface unit is indicated using visible light and invisible light, and each light emitting point is a visible light one. It can be distinguished from invisible light. Furthermore, it is possible to synthesize unique synthesis information for each light emitting point.

  The program executed in the present embodiment is provided by being incorporated in advance in a ROM, a memory unit, or the like, but is not limited thereto. The program executed in this embodiment may be recorded on a computer-readable recording medium and provided as a computer program product. For example, a file in an installable format or an executable format may be provided by being recorded on a recording medium such as a flexible disk, a CD-R, a DVD (Digital Versatile Disk), a Blu-ray Disc (registered trademark), or a semiconductor memory. Good.

  In addition, the program executed in the present embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program executed in this embodiment may be configured to be provided or distributed via a network such as the Internet.

  As mentioned above, although each embodiment and modification of this invention were demonstrated, each embodiment and modification are shown as an example and are not intending limiting the range of invention. These novel embodiments and modifications can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 1 Image projection system 10 Integrated image projector 10a Projection part 10b Camera part 11 1st laser pointer 12 2nd laser pointer 13 Projection surface part
DESCRIPTION OF SYMBOLS 21 Video processing part 22 Drive control part 23 Video signal input part 24 Conversion coefficient calculating part 25 Instruction mark detection part 26 Image processing part

Japanese Patent No. 3733915

Claims (10)

Irradiation means for irradiating visible light and invisible light;
A projection unit that projects an image on a projection surface;
An imaging unit that captures a light emission point of visible light and invisible light formed by irradiation of the irradiation unit on an image projected on the projection surface at a predetermined timing;
A discriminator for discriminating and detecting whether the visible light mark or the invisible light mark is a mark corresponding to each light emitting point of the visible light and the invisible light from captured image data obtained by imaging of the imaging unit; ,
Based on the positions of the visible light and invisible light marks detected by the discriminating unit in the image of the captured image data, each of the visible light and the invisible light is unique to the image projected by the projection unit. A synthesis unit for synthesizing the synthesis information of
An image projection system comprising: An invisible light irradiating unit that irradiates the invisible light on the projection surface instead of the irradiating means of the invisible light and forms a light emitting point of the invisible light on an instruction unit that touches and indicates the projection surface. Prepare
The image projection system according to claim 1. As the irradiating means of the visible light, a light source having a transmissive part that transmits the invisible light in an end light emitting part,
The image projection system according to claim 2. In place of the irradiating means for forming either one of the visible light or the invisible light, a light emitting part that emits light by stress is provided on the projection surface.
The image projection system according to claim 1. The discriminating unit discriminates and detects the visible light mark or the invisible light mark based on color information.
The image projection system according to claim 1, wherein the image projection system is an image projection system. The discriminating unit discriminates and detects the visible light mark or the invisible light mark based on a luminance pattern in the captured image data.
The image projection system according to claim 1, wherein the image projection system is an image projection system. The combining unit forms a line in the locus of the light emission point with the movement of the light emission point of the visible light or the invisible light on the projection plane.
The image projection system according to claim 1, wherein the image projection system is an image projection system. A projection unit that projects an image on a projection surface;
An imaging unit that captures light emission points of visible light and invisible light formed by irradiation of an irradiation unit on an image projected on the projection surface at a predetermined timing;
A discriminator for discriminating and detecting whether the visible light mark or the invisible light mark is a mark corresponding to each light emitting point of the visible light and the invisible light from captured image data obtained by imaging of the imaging unit; ,
Based on the positions of the visible light and invisible light marks detected by the discriminating unit in the image of the captured image data, each of the visible light and the invisible light is unique to the image projected by the projection unit. A synthesis unit for synthesizing the synthesis information of
An image processing apparatus comprising: The size that people can carry with one hand,
The image processing apparatus according to claim 8. Computer
An imaging control unit that controls the timing at which imaging points of visible light and invisible light formed by the irradiation unit on the image projected on the projection surface are imaged by the imaging camera;
A discriminator for discriminating and detecting whether the visible light mark or the invisible light mark is a mark corresponding to each light emitting point of the visible light and the invisible light from captured image data obtained by imaging by the imaging camera; ,
The visible light combined with the corresponding position of the image projected on the projection plane based on the position in the image of the captured image data of each mark of the visible light and the invisible light detected by the discrimination unit and the A synthesis information output unit that outputs synthesis information unique to each invisible light;
Program to function as.