JP2008152622A - Pointing device - Google Patents

Abstract

A pointing device capable of accurately and easily detecting a designated position in a projected image is obtained.
A projection image is projected by a projector and is photographed by a camera. The hand region extraction unit 106 extracts a hand region included in the photographed image based on the difference image between the photographed image photographed by the camera 300 and the projection image. The fingertip detection means 107 detects the fingertip position from the hand area extracted by the hand area extraction means 106, and outputs this as the designated position in the projection image. The pointing means 108 notifies the system of the indicated position detected by the fingertip detection means 107.
[Selection] Figure 1

Description

  The present invention relates to a pointing device that detects a position instructed by a finger or the like in a projected image.

  A conventional pointing device extracts a hand region by taking a difference between an image obtained by photographing a fixed background in advance and an image when a hand is placed on the background (see, for example, Patent Document 1). . A method for extracting a hand region by extracting skin color pixels has also been proposed (see, for example, Patent Document 2). Further, a method of extracting a hand region using a near-infrared camera (for example, see Patent Document 3), a device called a virtual keyboard that tracks finger movements using an infrared laser (for example, see Non-Patent Document 1), etc. There is.

Japanese Patent Laid-Open No. 11-25260 JP 09-35066 A JP 2001-282456 A http://www.tanomi.com/limited/html/00034.html

  In a conventional pointing device, in the method of extracting a hand region based on a difference from a fixed background, when directly manipulating an image projected by a projector or the like, the projected image is projected onto the background, so the image is projected. Every time it is performed, the background is different from that stored in advance, and the hand area cannot be extracted correctly. Further, in the method of extracting skin color pixels, when the projection image is directly manipulated, the image is projected on the hand. For example, when a blue image is projected on the hand, There was a problem that could not be extracted correctly. Furthermore, the conventional pointing device using a special device has a problem that the cost for the device becomes high.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a pointing device that can accurately and easily detect a designated position in a projection image.

  The pointing device according to the present invention includes a projecting unit that projects an arbitrary projected image, an image capturing unit that captures an image projected by the projecting unit and including an instruction unit, and a captured image and a projected image captured by the image capturing unit. Based on the difference image, the instruction unit region extracting means for extracting the region of the instruction unit included in the photographed image, and the instruction unit of the projection image based on the region of the instruction unit extracted by the instruction unit region extraction unit And pointing position detecting means for detecting which position is indicated.

  The pointing device according to the present invention extracts a region of the instruction unit included in the photographed image based on the difference image between the photographed image and the projection image, and based on the region of the instruction unit, the pointing unit determines which position of the projection image Therefore, it is possible to accurately and easily detect the designated position in the projection image.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a pointing device according to Embodiment 1 of the present invention.
In the figure, the pointing device includes a pointing control device 100, a projector 200, and a camera 300. In the first embodiment, it is assumed that the instruction unit is a human hand, and hereinafter, the instruction unit is a human hand. The pointing control device 100 is a functional unit that controls the projector 200 and the camera 300 and extracts an instruction position from a projected image and a captured image, and includes a video output unit 101, a video memory 102, an image input unit 103, a frame memory 104, a calibration. , A hand region extracting means 106, a fingertip detecting means 107, and a pointing means 108.

  The projector 200 is a device that projects an image onto various objects based on the image signal output from the video output unit 101, and constitutes a projection unit. The camera 300 is an imaging unit that captures an image projected by the projector 200 including an instruction unit and outputs the captured image as a captured image to the pointing control apparatus 100. Further, as shown in FIG. 3 to be described later, the pointing control device 100, the projector 200, and the camera 300 are integrally provided as a terminal such as a mobile phone, for example, and the camera 300 captures an image projected by the projector 200. It is arranged at a possible position. In the terminal shown in FIG. 3, the projector 200 and the camera 300 are disposed adjacent to each other. However, there is no particular limitation on the positional relationship between them, and the projector 200 and the camera 300 are separated from the main body of the terminal. May be provided.

  The video output unit 101 is a functional unit that outputs an image signal stored in the video memory 102 to the projector 200, and constitutes a projection unit together with the projector 200. The video memory 102 is a memory for storing projection image data. The image input unit 103 is an input interface for inputting a captured image including an instruction unit output from the camera 300, and the frame memory 104 is a memory for storing the captured image input by the image input unit 103. The calibration means 105 is a means for obtaining parameters for correcting the color for hand region extraction and the distortion of the captured image, and includes a distortion correction calibration means 109 and a color calibration means 110. The distortion correction calibration means 109 is a means for obtaining a parameter for correcting distortion in order to compare the captured image with the projected image, and the color calibration means 110 indicates the tendency of color change between the captured image and the projected image. It is a means to seek.

  The hand region extraction unit 106 is an instruction unit region extraction unit that extracts a hand region from a captured image stored in the frame memory 104. The hand region extraction unit 106 is a shake correction unit 111, a distortion correction unit 112, a color correction unit 113, and a difference image extraction unit 114. It has. The blur correction unit 111 is a unit that corrects the blur of the captured image, and the distortion correction unit 112 is a unit that corrects the distortion of the captured image based on the distortion correction parameter of the distortion correction calibration unit 109. The color correction unit 113 is a unit that performs color correction based on the color correction parameter obtained by the color calibration unit 110. Further, the difference image extraction means 114 is a means for obtaining a difference image between the photographed image and the projection image. Note that specific processing of the shake correction unit 111, the distortion correction unit 112, and the color correction unit 113 will be described later.

  The fingertip detection unit 107 is an instruction position detection unit that detects the position coordinates of the fingertip that is the instruction position from the hand region extracted by the hand region extraction unit 106. The pointing unit 108 is a functional unit that notifies the coordinate of the fingertip detected by the fingertip detection unit 107 to a GUI (Graphical User Interface) system such as a window system.

  The pointing control device 100 is realized by a computer, and the calibration means 105, the hand region extraction means 106, the fingertip detection means 107, and the pointing means 108 described above are software corresponding to each function and software for executing the software. It consists of hardware such as CPU and memory.

Next, the operation of the pointing device according to the first embodiment will be described.
FIG. 2 is a flowchart showing the operation of the pointing device according to the first embodiment.
FIG. 3 is an explanatory diagram illustrating an operation of the pointing device according to the first embodiment.
As shown in FIG. 3, the pointing device according to the present embodiment is intended for a case where a user directly operates an image 201 projected by a terminal equipped with a camera 300 and a projector 200 with a hand 202. Here, the terminal may be any method of use, such as a method in which the user operates with the other hand or a method in which the terminal is placed on a desk.

  In the pointing device, first, an image stored in the video memory 102 is projected by the projector 200 via the video output means 101 (step ST1). Next, a projected image is captured by the camera 300, the captured image is input via the image input means 103, and the captured image is stored in the frame memory 104 (step ST2). The image projected from the video output means 101 has markers 301a, 301b, 301c, and 301d at the four corners as shown in FIG.

The hand area extraction unit 106 acquires the captured image stored in the frame memory 102 and the projection image stored in the video memory 102, and first corrects the shake of the captured image in the shake correction unit 111 (step ST3). The specific operation of this blur correction process is as follows.
FIG. 5 is a flowchart showing the operation of the shake correction unit 111.
First, the blur correction unit 111 detects the markers 301a to 301d included in the captured image shown in FIG. 4 (step ST31). The marker to be detected is composed of pixels of a special color for easy detection, and each of the markers 301a to 301d has a different color. In the detection process, pixels of the above color are extracted, an area where the extracted pixels are connected is obtained, and an area within a certain range is obtained. And the center coordinate is calculated | required. Next, a transformation matrix is obtained using the coordinates of the obtained four points and the coordinates (original coordinates) of the four corners of the projection image obtained in advance by the calibration means 105 (step ST32). Note that how to obtain the coordinates of the four corners will be described later. This conversion matrix is used to correct the current captured image to the original position. When the coordinate value of each pixel of the captured image is (xc, yc) and the original coordinate value is (xo, yo). The following equation holds.

式（３）から、ｈｓはＢ^TＢの最小固有値に対応する固有ベクトルとして求めることができる。

From Expression (3), hs can be obtained as an eigenvector corresponding to the minimum eigenvalue of B ^T B.

By converting the captured image using the conversion matrix Hs obtained above (step ST33), it is possible to correct the blur.
FIG. 6 is an explanatory diagram of the correction operation of the shake correction unit 111.
For example, when the projection image 402 exists in the captured image 401, the projection image 402 can be returned to the position of the projection image 404 when calibrated by the calibration means 105 as shown in the correction image 403. .
Next, an image for comparison with the projected image is obtained from the photographed image subjected to the blur correction.

  In order to obtain an image that can be compared with the projected image, first, the distortion correction unit 112 converts the image using the transformation matrix obtained by the distortion correction calibration unit 109 of the calibration unit 105 (step ST4). Next, the color correction unit 113 converts each pixel value of the image obtained by the distortion correction unit 112 using the conversion parameter obtained by the color calibration unit 110 (step ST5). Note that the color calibration means 110 obtains a table storing hue and brightness errors as shown in FIG. The color correction unit 113 performs color correction using this table. First, each pixel value of the photographed image is converted from RGB to HSV. Then, the pixel value of the pixel whose saturation is lower than the threshold is divided into the pixel whose saturation is lower than the threshold, and the pixel value of the pixel whose saturation is lower than the threshold is corrected by referring to the brightness table (the error stored in the table is added). To do). When the saturation is higher than the threshold value, the hue is corrected using the hue table (the error stored in the table is added).

Then, the difference image extraction means 114 obtains the difference between the projection image and the image obtained above (step ST6). For the difference image, the captured image and the projection image are converted into HSV, respectively. First, a hue difference is obtained for pixels having a saturation greater than or equal to a threshold, and if the difference is within a certain threshold, it is regarded as the same pixel. The corresponding pixel value of the difference image is set to 0. If the difference is greater than or equal to the threshold, the corresponding pixel value of the difference image is set to 1. When the saturation is less than or equal to the threshold value, the brightness difference is obtained. If the difference falls within a certain threshold range, the pixels are regarded as the same pixel, and the corresponding pixel value of the difference image is set to 0. If the difference is greater than or equal to the threshold, the corresponding pixel value of the difference image is set to 1.
In addition to hue, saturation and brightness may be combined.

  By performing the above processing, for example, as shown in FIG. 7, if a difference image 503 between the projection image 501 and the converted captured image 502 is obtained, a hand region can be extracted. However, in consideration of the influence of noise, the largest region among the regions where pixels are connected from the difference image is set as a hand region.

Next, the fingertip detection unit 107 detects the fingertip from the hand area obtained as described above (step ST7).
FIG. 8 is a flowchart showing the operation of the fingertip detection unit 107.
First, the obtained hand region is converted into a distance image by distance conversion processing (step ST71). Here, the distance conversion process is a conversion in which each pixel value of the object in the image is replaced with the shortest distance from the background to the background. As the concept of distance, the simplest urban distance (4 connecting distances) and chessboard distance (8 connecting distances) are often used.
Here, an algorithm using city distance is described.

〈ステップ２〉初期化した画像を左上から右下に向かって走査し、次式（５）に示す規則で逐次Ｄ’_i,jを更新する。

〈ステップ３〉ステップ２で得られたＤ”_i,jに対して、右下から左上に向かって走査し、次式（６）に示す規則で逐次Ｄ”_i,jを更新する。

得られたＤが距離画像となる。 <Step 1> The image f _{i, j} obtained by binarizing the input image is set to D _{i, j} and further converted as in the following equation (4).

<Step 2> The initialized image is scanned from the upper left to the lower right, and D ′ _{i, j} is sequentially updated according to the rule shown in the following equation (5).

<Step 3> D ″ _{i, j} obtained in Step 2 is scanned from the lower right to the upper left, and D ″ _{i, j} is sequentially updated according to the rule shown in the following equation (6).

The obtained D becomes a distance image.

  The coordinate at which the pixel value (distance) is maximum from the distance image obtained above is set as the center of gravity (xg, yg) of the hand region (step ST72). If the hand region extracted by the hand region extraction means 106 is assumed to be in a state where the pointing finger is extended as indicated by 601 in FIG. 9, it can be approximated by an ellipse as indicated by 602 in FIG. At this time, an angle θ (603) between the long side of the ellipse and the y-axis is obtained, and the pixel having the maximum y-coordinate when the ellipse is rotated so that the long side and the y-axis match is the coordinate of the fingertip. Become.

よって式（７）においてＭ₁₁、Ｍ₂₀、Ｍ₀₂を求め、式（８）においてθを求める（ステップＳＴ７４）。 In order to obtain the angle between the long side and the y-axis, a moment feature represented by the following equation (7) is obtained (step ST73).

Therefore, M ₁₁ , M ₂₀ , and M ₀₂ are obtained in equation (7), and θ is obtained in equation (8) (step ST74).

  Next, rotation is performed using θ obtained so that the long side of the ellipse and the y axis coincide with each other (step ST75). Further, a point where the y coordinate is maximum is obtained, and the coordinate value before the rotation is used as the coordinate of the fingertip. (Step ST76). That is, the coordinate value is determined as the position designated by the user. The coordinate values obtained as described above are transferred to the pointing means 108 and subjected to the pointing process, whereby the user can directly operate the projection image (step ST8).

Next, the operation of the calibration unit 105 will be described with reference to FIGS.
FIG. 10 is a flowchart for explaining the operation of the calibration means 105.
FIG. 11 is an explanatory diagram showing the operation of the distortion correction calibration means 109.
FIG. 12 is an explanatory diagram showing the operation of the color calibration unit 110.
FIG. 13 is an explanatory diagram of a color conversion table obtained by the color calibration unit 110.

The pointing device of the present invention needs to perform distortion correction and color correction of the captured image so that the captured image can be compared with the projected image as described above. Each calibration is performed in order to obtain the distortion correction parameter and the color correction parameter at this time.
First, the video output unit 101 projects a grid as indicated by reference numeral 701 in FIG. 11 through the projector 200 (step ST101), and the projection image is acquired by the image input unit 103 through the camera 300 (step ST102). Next, the photographed image is input to the distortion correction calibration means 109, and grid points of the grid are obtained from the photographed image in order to obtain the distortion correction parameter (step ST103). The lattice points are described in, for example, Tsubasa Saburo, Takeshi, “3D Vision”, pp. 36-37, Kyoritsu Publishing, using a corner detection technique such as a SUSAN operator as in 1999.
The lattice point coordinates in the captured image obtained as described above are associated with each lattice point of the projection image (grid image) stored in the video memory 102. The coordinates of each grid point of the grid image are known.

式（１１）から、ｈはＢ^TＢの最小固有値に対応する固有ベクトルとして求めることができる。 When the lattice point coordinates in the captured image are (xc, yc) and the lattice point coordinates on the projection image are (xo, yo), the following equation is established.

From Expression (11), h can be obtained as an eigenvector corresponding to the minimum eigenvalue of B ^T B.

  Next, the video output means 101 projects a color pattern as indicated by reference numeral 801 in FIG. 12 through the projector 200 (step ST105), and the projection image is acquired by the image input means 103 via the camera 300 (step ST106). . Then, the image is input to the color calibration unit 110, and a change in color tone is measured (step ST107). Here, it is assumed that the color pattern 801 includes, for example, representative colors obtained when the hue is divided into nine and white, gray, and black. That is, nine colors represent colors in a certain range of hues.

In the color calibration unit 110, first, distortion correction of the captured image is performed in the same manner as the distortion correction unit 112 using the conversion matrix obtained by the distortion correction calibration unit 109, and the color pattern 801 stored in the video memory 102 is corrected. The average of the pixel values (RGB values) of the captured image region corresponding to the region of each color is obtained. Next, the obtained average value (RGB value) is converted into HSV, and a difference from the original nine hues stored in the video memory 102 is obtained and stored in the hue table 901 of FIG.
Similarly, the average of the pixel values (RGB values) of the photographed image areas corresponding to the white, gray, and black areas of the color pattern 801 is obtained, and the brightness of the obtained value is obtained and stored in the video memory 102. Differences from the original white, gray, and black brightness are obtained and stored in the brightness table 902 of FIG.
As described above, by providing the calibration unit 105, it is possible to appropriately obtain the difference image between the projected image and the captured image, and it is possible to accurately extract the hand region.

  As described above, according to the pointing device of the first embodiment, the projection unit that projects an arbitrary projection image, the imaging unit that projects an image that is projected by the projection unit and includes an instruction unit, and the imaging unit Based on a difference image between the photographed captured image and the projected image, an instruction unit region extracting unit that extracts an instruction unit region included in the photographed image, and an instruction unit region extracted by the instruction unit region extracting unit On the basis of this, it is provided with the indicated position detecting means for detecting which position of the projected image the pointing unit is pointing to, so that the indicated position in the projected image can be detected accurately and easily.

  In addition, according to the pointing device of the first embodiment, the instruction area extraction unit includes a shake correction unit that corrects image blur for a captured image, and a distortion that corrects image distortion for the captured image. Since the correction means and the color correction means for correcting the color of the image with respect to the captured image are provided, and the difference image between the captured image and the projection image corrected by each of the correction means is acquired, the instruction unit Can be accurately extracted, and as a result, the indicated position in the projection image can be accurately obtained.

  Further, according to the pointing device of the first embodiment, a distortion image is used as a projection image, and the distortion correction parameter is calculated based on the relationship between the lattice point of the captured image of the lattice image and the lattice point of the projection image. Since the correction calibration means is provided and the distortion correction means corrects the distortion of the photographed image based on the distortion correction parameter, the distortion of the image can be corrected favorably. Can be obtained accurately.

  Further, according to the pointing device of the first embodiment, a color pattern is used as a projection image, and the color correction parameter is calculated based on the relationship between the color information of the captured image of the color pattern and the color information of the projection image. Since the calibration unit is provided and the color correction unit performs color correction of the captured image based on the color correction parameter, it is possible to perform color correction on the captured image satisfactorily. Can be obtained accurately.

  Further, according to the pointing device of the first embodiment, an image including a marker indicating a position in the image is used as the projection image, and the blur correction unit is configured to detect the position of the marker in the captured image of the image including the marker and the projection image. Corrected the image blurring based on the marker position coordinates, so that even if it is handheld in places where there is periodic vibration, such as in a train, the image blurring can be corrected well. Thus, the pointing position can be reliably detected even in such a use state.

  In addition, according to the pointing device of the first embodiment, the pointing position detecting unit performs distance image conversion on the pointing area, approximates the pointing area with an ellipse centered on the obtained barycentric coordinates, and calculates the length of the ellipse. Since the designated position is detected by obtaining the angle of the side, the designated position can be reliably detected.

Embodiment 2. FIG.
The second embodiment is an embodiment in which a monochrome background for facilitating extraction of hand regions at a predetermined interval is displayed and hand region extraction is possible at that time.
FIG. 14 is a configuration diagram of the pointing device shown in the second embodiment.
In the figure, the pointing device includes a pointing control device 100a, a projector 200, and a camera 300. The projector 200 and the camera 300 are the same as those in the first embodiment. The pointing control device 100a includes a video output unit 101, a video memory 102, an image input unit 103, a frame memory 104, a calibration unit 105a, a hand region extraction unit 106, a fingertip detection unit 107, a pointing unit 108, and a synchronization unit 115. The second embodiment is the same as the first embodiment except that the background color determination unit 116 is added to the calibration unit 105a and the synchronization unit 115 is provided.
Synchronizing means 115 is means for matching the timing of camera shooting with the projection of the projector. Further, the background color determining means 116 in the calibration means 105a is a means for determining the background color used for hand region extraction.

Next, the operation of the pointing device according to the second embodiment will be described.
In the present embodiment, as shown in FIG. 15, a monochrome image display frame is inserted between normal image display frames at a predetermined frame interval, and a hand region is detected at the timing of monochrome image display.
FIG. 16 is a flowchart for explaining the operation of the pointing process.
The synchronization unit 115 synchronizes the drawing timing of the projector 200 and the shooting timing of the camera 300 using the frame count of the camera 300. The video output means 101 checks the frame count of the camera 300 (step ST201), and outputs a background image when the count number satisfies, for example, the following conditions (steps ST202 and 203).
If count mod N = 0 (12)
The above equation (12) means that a background image is output at intervals of N frames.

If the condition of Expression (12) is satisfied in step ST202, a background image for obtaining a hand area is projected from the projector 200, and at the same time, an image is input via the image input unit 103 (step ST203).
In the subsequent processing, as in the first embodiment, blur correction of the input image is performed (step ST204), distortion correction is performed (step ST205), color correction is performed (step ST206), and a difference image is extracted (step ST207). ), Fingertip detection is performed (step ST208), and pointing processing is performed (step ST209).
On the other hand, if Expression (12) is not satisfied in step ST202, a normal projection image is projected by the projector 200 (step ST210).

Next, the operation of the background color determination unit 116 of the calibration unit 105a will be described.
FIG. 17 is a flowchart for explaining the operation of the background color determining means 116.
Here, it is assumed that the background color determination means 116 has skin color information in advance.
First, an image is shot without projecting anything from the projector 200 and with the hand serving as an instruction unit not in the field of view of the camera 300, and the average hue, average saturation, and average brightness at that time are obtained (step ST301). To Step ST303). That is, the actual background color value projected by the projector 200 is obtained. Then, it is checked whether or not the saturation is equal to or less than the threshold value (step ST304). If the saturation is a value larger than the threshold value, the hue obtained in step ST301 within the hue range opposite to the skin color is determined. A hue having a large difference is obtained (step ST305). Specifically, since the hue is expressed in the range of 0 to 2π, the hue CSH at the counter electrode can be obtained by the following equation when the hue of the skin color is SH.
CSH = SH + π (13)
If the hue obtained in step ST301 is BH, the obtained hue TH is obtained by the following equation.

  If the saturation is not more than the threshold value in step ST304, the obtained average brightness is checked (step ST306). If the brightness is greater than the threshold value, step ST305 is executed. If the saturation is not more than the threshold value, the background color is selected. The flag is determined to be black, and the difference image extraction unit 114 sets a flag for instructing the difference to be a difference in brightness (step ST307).

  As described above, a simple background image is displayed at intervals of several frames, and a hand region is extracted by comparing the background image with a captured image, thereby reducing the amount of processing required for hand region extraction.

  In the first and second embodiments, the pointing unit is a human hand. However, any object other than the hand can be applied as long as the object is different from the projected image. However, an object with a sharp tip is desirable in order to uniquely detect the indicated position.

  As described above, the pointing device according to the second embodiment includes the synchronization unit that synchronizes the projection image and the captured image, and the background color determination unit that determines the background color based on the color information of the instruction unit. The projecting unit projects the specific image of the background color determined by the background color determining unit at a predetermined interval, and the pointing unit region extracting unit is configured to project the specific image projected at the predetermined interval and a captured image synchronized with the specific image. Therefore, it is possible to reduce the amount of processing required for the extraction of the instruction area, and as a result, it is possible to reduce the processing as the pointing device.

  Further, according to the pointing device of the second embodiment, the background color determination unit has skin color information, and the specific image is determined based on the background color and the skin color projected by the projection unit. Since it is possible to accurately extract the pointing unit region when the pointing unit is a human hand, it is possible to reliably detect the pointing position.

  Further, according to the pointing device of the first and second embodiments, since the pointing unit is a human hand, a special instrument or the like is not required as means for designating the position in the image, and a highly convenient pointing device is provided. Obtainable.

It is a block diagram which shows the pointing device by Embodiment 1 of this invention. It is a flowchart which shows operation | movement of the pointing device by Embodiment 1 of this invention. It is explanatory drawing which shows the mode of operation | movement of the pointing device by Embodiment 1 of this invention. It is explanatory drawing which shows the marker of the pointing device by Embodiment 1 of this invention. It is a flowchart which shows operation | movement of the shake correction means of the pointing device by Embodiment 1 of this invention. It is explanatory drawing of the correction | amendment operation | movement of the shake correction means of the pointing device by Embodiment 1 of this invention. It is explanatory drawing which shows the projection image in the pointing device by Embodiment 1 of this invention, a picked-up image, and a difference image. It is a flowchart which shows operation | movement of the fingertip detection means of the pointing device by Embodiment 1 of this invention. It is explanatory drawing which shows the extraction operation | movement of the hand area | region of the pointing device by Embodiment 1 of this invention. It is a flowchart which shows operation | movement of the calibration means of the pointing device by Embodiment 1 of this invention. It is explanatory drawing which shows operation | movement of the distortion correction calibration means of the pointing device by Embodiment 1 of this invention. It is a flowchart which shows operation | movement of the color calibration means of the pointing device by Embodiment 1 of this invention. It is explanatory drawing of the conversion table of the color calculated | required with the color calibration means of the pointing device by Embodiment 1 of this invention. It is a block diagram of the pointing device by Embodiment 2 of this invention. It is explanatory drawing which shows the timing of the image display of the pointing device by Embodiment 2 of this invention. It is a flowchart which shows operation | movement of the pointing device by Embodiment 2 of this invention. It is a flowchart which shows operation | movement of the background color determination means of the pointing device by Embodiment 2 of this invention.

Explanation of symbols

  100, 100a pointing control device, 101 video output means, 103 image input means, 105, 105a calibration means, 106 hand area extraction means, 107 fingertip detection means, 108 pointing means, 109 distortion correction calibration means, 110 color calibration Means, 111 blur correction means, 112 distortion correction means, 113 color correction means, 114 difference image extraction means, 115 synchronization means, 116 background color determination means, 200 projector, 300 camera, 301a to 301d marker, 601 hand area, 602 ellipse , 701 grid, 801 color pattern, 901 hue table, 902 brightness table.

Claims (9)

Projection means for projecting an arbitrary projection image;
Photographing means for photographing an image projected by the projecting means and including an instruction unit;
An instruction area extraction unit that extracts an area of the instruction unit included in the captured image based on a difference image between the captured image captured by the imaging unit and the projection image;
A pointing device comprising: pointing position detecting means for detecting which position of the projection image the pointing section points to based on the area of the pointing section extracted by the pointing section area extracting means. The instruction area extraction means
Blur correction means for correcting image blur for a captured image;
Distortion correction means for correcting image distortion for the captured image;
Color correction means for correcting the color of the image with respect to the captured image;
The pointing device according to claim 1, wherein a difference image between the captured image and the projection image corrected by each of the correction units is acquired. Using a lattice image as a projection image, comprising distortion correction calibration means for calculating a distortion correction parameter based on the relationship between the lattice point of the captured image of the lattice image and the lattice point of the projection image,
The pointing device according to claim 2, wherein the distortion correction unit corrects distortion of the captured image based on the distortion correction parameter. Using a color pattern as a projection image, and comprising color calibration means for calculating a color correction parameter based on the relationship between the color information of the captured image of the color pattern and the color information of the projection image,
The pointing device according to claim 2, wherein the color correction unit performs color correction of the captured image based on the color correction parameter.   An image including a marker indicating a position in the image is used as the projected image, and the blur correction unit is configured to detect the position of the captured image based on the marker position in the captured image of the image including the marker and the position coordinate of the marker in the projected image. 3. The pointing device according to claim 2, wherein projection blur is corrected.   The pointing position detecting means converts the pointing area into a distance image, approximates the pointing area with an ellipse centered on the obtained barycentric coordinates, and obtains the angle of the long side of the ellipse to obtain the pointing position. 6. The pointing device according to claim 1, wherein the pointing device is detected. Synchronization means for synchronizing the projected image and the captured image;
A background color determining means for determining a background color based on the color information of the instruction unit;
The projecting unit projects the specific image of the background color determined by the background color determining unit at a predetermined interval, and the instruction area extraction unit synchronizes with the specific image projected at the predetermined interval and the specific image. The pointing device according to claim 1, wherein an instruction area is extracted based on a photographed image.   8. The pointing device according to claim 7, wherein the background color determination means has skin color information and determines the specific image based on the background color and the skin color projected by the projection means.   The pointing device according to claim 1, wherein the instruction unit is a human hand.

Images