CN101005565A - Electronic equipment - Google Patents

Abstract

Provided is an electronic device capable of more accurately detecting a simple operation for recognition without being affected by noise or the like when the electronic device is controlled by image recognition. And a detection unit (19) having a plurality of detectors provided in correspondence with a plurality of detection regions obtained by dividing the screen of the image output from the camera (2) into N in the horizontal direction and M in the vertical direction, and generating a first detection signal based on the movement performed by the operator. The generator generates a second detection signal based on the first detection signal. A timing generator (12) selectively supplies a timing pulse to a plurality of detectors including at least a detector which generates a flag when the value of the accumulated second detection signal exceeds a threshold value.

Description

Electronic equipment

technical field

The present invention relates to electronic equipment such as a television receiver equipped with a video camera, and to electronic equipment for recognizing motion images of human hands and the like and performing remote operations on the electronic equipment.

Background technique

In the 1980s, infrared remote controllers (commonly called remote controllers) were widely used to attach home appliances such as TV receivers, and user interfaces that can be controlled at hand greatly changed the way home appliances are used. . Now this kind of operation method is still the mainstream, but the basic structure of the remote control is to realize a function by pressing a button. It is a very convenient remote operation method for previous TV receivers.

However, when the remote controller is not at hand or cannot be found, it will be extremely inconvenient. In response to this, a method of recognizing the motion or shape of an image, and performing switching operations such as switching on and off a power source has been studied. For example, Japanese Unexamined Patent Application Publication No. 11-338614 (Patent Document 1) discloses a technique for recognizing the motion or shape of a hand and applying it to device operation. In this technique, a dedicated infrared sensor or an image sensor is used as a detection device for detecting the motion or shape of a hand.

On the other hand, in the recently started data broadcasting, in order to select a desired menu screen, it is necessary to press the "Up", "Down", "Left", "Right" and "Enter" keys on the remote control multiple times, and the The operation is cumbersome and inconvenient to use. Also, in the EPG (Electronic Program Guide), a desired position is selected from the introduction screens arranged in a matrix and a key is pressed, so there is the same problem as in data playback. In addition, for such a very fine selection operation, a system that can respond to various operations by utilizing the motion and shape of the image in the same manner is desired.

In Japanese Unexamined Patent Publication No. 2003-283866 (Patent Document 2), in order to solve such a problem, a control device is proposed in which position specifying information obtained by using a mouse or a similar position specifying operation device is encoded as a key. The timing pattern of the pressing signal is the key pressing timing code, and the control device that sends the key pressing timing code to the television receiver.

Patent Document 1: Japanese Unexamined Patent Application Publication No. H11-338614

Patent Document 2: JP-A-2003-283866

In general consumer AV equipment (audio equipment or video equipment) such as television receivers, long-distance operation is realized by making use of conventional remote controllers. Therefore, when the remote controller is not at hand, for example, when the power is turned on, it is necessary to confirm the location of the remote controller, obtain the remote controller, and select and operate the corresponding key, which will be inconvenient for the user. In addition, when the location of the remote controller is unknown, it is necessary to turn on the main power switch of the television receiver body. These are the problem points about remote control operation that I have often experienced in the past.

On the other hand, for the operation of turning off the power, it is convenient to use the remote control to turn off the power of the television receiver when the remote control is already at hand. However, the same problem exists when trying to turn off the power when the remote controller, which is slightly away from the seat or the like, is not at hand.

The motions used in the control method described in Patent Document 1 are simple motions such as circular motion, up and down motion, and side to side motion. If the operation by image recognition can be realized, it will become a very convenient operation method. However, while the operation is easy, there is a problem of tolerance for misrecognition, and there are also difficult problems in realizing the device on an appropriate scale and sharing it with other image recognition processing devices.

The control device disclosed in Patent Document 2 is a device for remotely operating a television receiver by operating a pointing ball similar to the operation of a personal computer. Therefore, it is inconvenient to use for people who do not use a personal computer, and it is inappropriate to directly introduce convenient operations on a personal computer into electronic equipment from the viewpoint of information usability. Therefore, a new operating mechanism is required in the current usage of television receivers requiring remote operation.

In image recognition that requires a variety of selection operations, from power on/off to image recognition of two-step selection operations and menu screen selection, a new operation mechanism can be realized with an appropriate device size using the same mechanism, providing low-cost The aspect of civilian equipment is a very important subject. In addition, a simple image recognition operation is prone to erroneous recognition, including the possibility of causing fatal erroneous operations such as turning off the power when performing an operation similar to the recognition operation while watching TV.

Contents of the invention

The present invention has been made in view of such problems, and an object of the present invention is to provide an electronic device capable of more accurately detecting a simple movement for recognition without being affected by noise or the like when controlling the electronic device by image recognition.

In order to solve the above-mentioned problems, the present invention provides the following electronic devices (a) to (f).

(a) An electronic device comprising: a display device 23; a camera 2 that takes pictures of the operator 3 positioned in front of the above-mentioned display device; A plurality of detectors respectively arranged correspondingly to a plurality of detection areas of N division and M division in the vertical direction, utilizes the plurality of detectors to generate a first detection signal based on the actions of the operator photographed by the above-mentioned camera; a timing pulse The generator 12 supplies timing pulses for making the plurality of detectors work corresponding to the plurality of detection areas; the generators 20-1 to 20-5 generate a second detection signal based on the first detection signal; the flag generator 20 , when the summed value of the above-mentioned second detection signals accumulated within a predetermined period exceeds a predetermined threshold value, a flag is generated; The second detection signal of the first detection signal output by the detector corresponding to a part of the detection area in the detection area is valid, and the second detection signal based on the first detection signal output from other detectors is invalid; the above-mentioned timing pulse generator is in the above-mentioned The flag generator selectively supplies the above-mentioned detector to a specific detector that generates the above-mentioned flag and a detector corresponding to a detection area in the vicinity of the above-mentioned specific detection area among the plurality of detectors within a predetermined period after the flag generator generates the above-mentioned flag. Timing pulse, wherein the detection area near the specific detection area includes at least a detection area adjacent to the specific detection area corresponding to the specific detector; wherein, M and N are integers of 2 or more.

(b) The electronic device described in (a) above has N first detectors 317-325 and M second detectors 301-316, and the N first detectors correspond to the image output from the above-mentioned camera. The detection area where the screen is divided into N in the horizontal direction, and the M second detectors correspond to the detection area in which the image output from the camera is divided into M in the vertical direction; the timing pulse generator is in the above-mentioned mark generator A width of a timing pulse supplied to the N first detectors or the Mth second detector based on the control of the controller is narrowed in accordance with an operation of the operator during a predetermined period after the flag is generated.

(c) The electronic device as described in (a) above, having an N×M detection area corresponding to N×M detection areas provided by dividing the image screen output by the above-mentioned camera into N in the horizontal direction and M in the vertical direction. detectors; the controller performs the following control: within a predetermined period after the flag generator generates the flag, the first detection based on the output from a specific detector that generates the flag in the N×M detector The second detection signal of the signal and the second detection signal based on the first detection signal output from the detector corresponding to the detection area in the vicinity of the above-mentioned specific detection area are valid, and the second detection signal based on the first detection signal output from other detectors is enabled. The second detection signal is invalid.

(d) The electronic equipment described in any one of the above (a) to (c), the above-mentioned electronic equipment includes: a mirror converter 14, which performs mirror conversion of the image taken by the above-mentioned camera; an operation image generator 16, which generates at least an operation image; and a mixer 17 that mixes the image signal for mirror transformation output from the above-mentioned mirror converter and the image signal for operation output from the above-mentioned image generator for operation; when the image mixed by the above-mentioned mixer is displayed on the above-mentioned display device In a state, the detection unit generates the first detection signal corresponding to a predetermined movement of the operator operating the operation screen displayed on the display device.

(e) The electronic device described in any one of (a) to (d) above, wherein the detecting unit includes: a digital filter kn that performs a tap coefficient corresponding to the first reference signal waveform generated when detecting the first operation and the first Two detection signals are multiplied, and the above-mentioned first action causes the object photographed by the above-mentioned camera to move in the longitudinal direction; The action performed by the person is not the first action mentioned above.

(f) The electronic device described in any one of (a) to (d) above, wherein the detection unit includes: a digital filter kn that compares the tap coefficient corresponding to the second reference signal waveform generated when detecting the second operation and the first Two detection signals are multiplied, and the above-mentioned second action causes the object photographed by the above-mentioned camera to move in the lateral direction; The action performed by the person is not the second action mentioned above.

According to the present invention, when electronic equipment is controlled using image recognition, simple motions for recognition can be more reliably detected without being affected by noise or the like.

Description of drawings

FIG. 1 is a diagram for explaining the outline of an electronic device identification operation method according to the present invention.

FIG. 2 is a block diagram showing a configuration of main parts of a television receiver according to an embodiment of the present invention.

FIG. 3 is a diagram illustrating an example of controlling a television receiver by recognizing an operator's motion.

FIG. 4 is a diagram showing a state of a user captured by a camera.

FIG. 5 is a diagram for explaining the relationship between a y-axis detector, a detection area within a screen, and a timing pulse for controlling it.

FIG. 6 is a diagram for explaining the relationship between an x-axis detector, a detection area within a screen, and a timing pulse for controlling it.

Fig. 7 is a block diagram showing the configuration of the detector shown in Fig. 2 .

Fig. 8 is a block diagram showing the structure of the object extractor shown in Fig. 7 .

FIG. 9 is a diagram for explaining the hue and saturation of an object extracted by the specific color filter shown in FIG. 8 .

FIG. 10 is a flowchart of processing for calculating hue from color difference signals.

Fig. 11 is a diagram showing the luminance signal level of an object extracted by the gradation limiter shown in Fig. 8 .

FIG. 12 is a block diagram showing the configuration of the motion detection filter shown in FIG. 8 .

FIG. 13 is a graph showing characteristics of a motion detection filter.

Fig. 14 is a diagram depicting how the output of the object extractor is displayed on the screen.

Fig. 15 is a block diagram showing the configuration of a control information determiner (CPU).

FIG. 16 is a schematic diagram showing output signals of a histogram detector and an average luminance detector in an object characteristic data detection unit.

FIG. 17 is a diagram for explaining the relationship between a vertically moving hand and coordinates indicating a detection area displayed on the screen.

18 is a table showing the detection data of the x-axis detector and the y-axis detector and the value of the center of gravity calculated from the detection data (when the movement of the hand is vertical swing).

Fig. 19 is a time chart showing changes in the center-of-gravity coordinates of the hand position (when the hand motion is vertical swing).

FIG. 20 is a block diagram showing the configuration of a high-pass filter.

FIG. 21 is a diagram depicting a screen and timing pulses when a detection area is limited by an active flag (Flg_x).

FIG. 22 is a diagram for explaining a method of generating an x-axis timing pulse of a y-axis detector.

FIG. 23 is a diagram for explaining the content of control performed by two timing pulses of the x-axis and y-axis of the y-axis detector.

Fig. 24 is a diagram showing the detection data of the x-axis detector and the y-axis detector after removing the data of unnecessary detectors by activating the flag (Flg_x), and the value of the center of gravity calculated from the detection data (hand movement is when swinging vertically).

Fig. 25 is a diagram for explaining the content of the cross-correlation digital filter (when the hand motion is vertical swing).

Fig. 26 is a timing chart showing fluctuations in the output of the cross-correlation digital filter (when the hand motion is vertical swing).

FIG. 27 is a diagram for explaining the relationship between a laterally moving hand displayed on the screen and coordinates indicating a detection area.

Fig. 28 is a table showing detection data of the x-axis detector and y-axis detector, and the value of the center of gravity calculated from the detection data (when the hand motion is lateral swing).

Fig. 29 is a time chart showing changes in the center-of-gravity coordinates of the hand position (when the hand movement is lateral swing).

FIG. 30 is a diagram depicting a screen and timing pulses when a detection area is limited by an active flag (Flg_y).

FIG. 31 is a diagram for explaining a method of generating a y-axis timing pulse of an x-axis detector.

FIG. 32 is a diagram for explaining the content of control performed by two timing pulses of the x-axis and y-axis of the x-axis detector.

33 is a table showing the detection data of the x-axis detector and the y-axis detector after removing the data of unnecessary detectors by activating the flag (Flg_y), and the value of the center of gravity calculated from the detection data (hand movement is during lateral swing).

Fig. 34 is a diagram for explaining the contents of the cross-correlation digital filter (when the hand movement is lateral swing).

Fig. 35 is a timing chart showing fluctuations in the output of the cross-correlation digital filter (when the hand movement is lateral swing).

Fig. 36 is a flowchart showing the processing procedure of the motion detection method.

Fig. 37 is a diagram showing detection areas and corresponding detectors in the second embodiment.

FIG. 38 is a diagram showing a hand performing a vertical swing motion on the detection area of the second embodiment.

FIG. 39 is a block diagram of a detector according to the second embodiment including an object characteristic data detection unit 530 .

FIG. 40 is a diagram showing detection areas where differences have occurred in the second embodiment.

FIG. 41 is a diagram showing the second object extractor 510 .

42 is a table showing detection data from the x-axis detector and the y-axis detector of the second embodiment.

FIG. 43 is a diagram showing a detection area subjected to a masking process in the second embodiment.

FIG. 44 is a diagram illustrating an example of a menu screen in which an image of an operator and a menu image are mixed in one embodiment of the present invention.

FIG. 45 is a diagram depicting the state of an operator performing an operation on a menu screen.

Detailed ways

One embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a diagram for explaining the difference between the operation mode of the conventional remote control device and the operation mode of the present invention. When the user (operator) 3 operates the television receiver 1, the user 3 has conventionally operated the television receiver 1 by pressing a key for activating a desired function with the hand-held remote control device 4 . Therefore, if there is no remote control device 4, it cannot be operated, and a large inconvenience is often experienced.

In this embodiment, as shown in FIG. 1, a camera 2 is provided on the television receiver 1, and the user 3 is photographed by the camera 2, and the actions of the user 3 are detected according to the image of the camera 2, and the television receiver 1 and its correlation are performed. operation of the device.

The detected action of the user 3 specifically refers to the use of the user 3's body ( specific movements of hands, feet, face, etc.). By detecting this specific motion, the electronic device is operated. In this embodiment, the most possible method of operating with hand movement will be described.

FIG. 2 is a block diagram showing the configuration of the television receiver 1 . The television receiver 1 has a reference sync generator 11, a timing pulse generator 12, a pattern generator 16, a video camera 2, a mirror converter 17, a scaler (scale) 15, a first mixer 17, a pixel number converter 21, A second mixer 22 , a display device 23 , a detection unit 19 , and a control information determiner (CPU) 20 .

The reference sync generator 11 generates horizontal periodic pulses and vertical periodic pulses that serve as references for the television receiver 1 . When television broadcasting is received and when a video signal is input from an external device, pulses synchronized with the synchronization signal of the input signal are generated. The timing pulse generator 12 generates pulses having arbitrary phases and widths in the horizontal and vertical directions required for each detection block (detection area) shown in FIG. 4 described later.

The camera 2 is located in front of the television receiver 1 shown in FIG. 1 , and captures images of the user (operator) 3 or the front of the television receiver 1 . The output signal of the camera 2 is a luminance (Y) signal and a color difference (R-Y, B-Y) signal, which are synchronized with the horizontal period pulse and the vertical period pulse output from the reference sync generator 1 . In addition, in this embodiment, the number of pixels of the image captured by the camera 2 matches the number of pixels of the display device 23 . If the number of pixels does not match, insert a pixel converter to make the number of pixels match.

The mirror converter 14 is used to display the subject image (the user 3 ) captured by the camera 2 on the display device 23 in a mirror-like reversed left-right manner. Therefore, when displaying a character, it is reversed left and right in the same way as a mirror. In this embodiment, mirror image conversion is performed by using a memory and by inverting an image in the horizontal direction.

When a CRT (cathode ray tube) is used as the display device 23, the same effect can be obtained by reversely operating the horizontal deflection. In this case, it is necessary to invert the graphics and other mixed side images in the horizontal direction in advance.

The scaler 15 is a device for adjusting the size of the subject image captured by the camera 2, and two-dimensionally adjusts the enlargement ratio and reduction ratio under the control of the control information judgment unit (CPU) 20 . In addition, horizontal and vertical phase adjustments can be performed without zooming in or out.

Graphics generator 16 is a device for expanding the menu screen sent from control information judging device (CPU) 20, even if the signal on the memory is in the primary colors of R (red) signal, G (green) signal, B (blue) signal The signal is expanded, and the output signal compounded or mixed with the image signal in the subsequent stage is a luminance (Y) signal and a color difference (R-Y, B-Y) signal. In addition, the number of planes of the generated figure is not limited, but one plane is used for illustration.

In this embodiment, the number of pixels is made to match the number of pixels of the display device 23 . In case of inconsistency, a pixel count converter needs to be inserted to make it consistent.

The first mixer 17 mixes the output signal Gs of the pattern generator 16 and the output signal S1 of the scaler 15 by controlling the mixing ratio using the control value α1. Specifically, the output signal M1o is represented by the following equation.

M1o=α1·S1+(1-α1)·Gs

The control value α1 is set to a value between 0 and 1. If the control value α1 is larger, the ratio of the output signal S1 of the scaler 14 becomes larger, and the ratio of the output signal Gs of the graph generator 16 becomes smaller. The example of the mixer is not limited to this, but in this embodiment, if the input signal information of the two systems is included, the same effect can be obtained.

The detection unit 19 includes a first detector 301 , a second detector 302 , a third detector 303 , ... an nth detector (300+n). The number of detectors included in the detection unit 19 is not limited, but in the first embodiment of the present invention, there are 25 detectors including the first detector 301 to the sixteenth detector that operate at timing in the horizontal direction described later. 16 of 316, and 9 of the 17th detector 317 to the 25th detector 325 that operate at the timing in the vertical direction.

The number of detectors is not limited thereto. In order to improve the detection accuracy, the more detectors the better, but it is desirable to adjust the number according to the relationship with the processing scale. In the first embodiment 25 detectors are used and in the second embodiment 144 detectors are used.

The control information judgment unit (CPU) 20 analyzes the data output from the detection part 19, and outputs various control signals. The processing content of the control information determiner 20 is realized by software, and the algorithm will be described in detail later. In the present embodiment, processing by hardware (each functional block) and processing by software (developed on CPU 20 ) are mixed, but the boundaries are not limited to those shown here.

The number-of-pixels converter 21 performs number-of-pixels conversion for making the number of pixels of an external input signal input from the outside coincide with the number of pixels of the display device 23 . The external input signal is assumed to be a signal input from the outside of the television receiver, such as a broadcast television signal (including data broadcast, etc.), video (VTR) input, or the like. In the description here, the synchronization system of the external input signal is omitted, but the configuration is such that the synchronization signals (horizontal and vertical) are obtained and the synchronization is made by the reference synchronization generator 11 .

The second mixer 22 has the same function as the first mixer 17 . That is, the output signal Mio of the first mixer 17 and the output signal S2 of the pixel number converter 21 are mixed by controlling the mixing ratio with the control value α2. Specifically, the output signal M2o is represented by the following equation.

M2o＝α2·M1o+(1-α2)·S2

The control value α2 is set to a value between 0 and 1. If the control value α2 is larger, the ratio of the output signal M1o of the first mixer 17 becomes larger, and the ratio of the output signal S2 of the pixel number converter 21 becomes smaller. . The example of the mixer is not limited to this, but in this embodiment, if the input signal information of the two systems is included, the same effect can be obtained.

The display device 23 is assumed to be a CRT, a liquid crystal display (LCD), a plasma display (PDP), or a projection display device, but the display method of the display is not limited. The input signal of the display device 23 is a luminance signal and a color-difference signal, which are matrix-converted into RGB primary color signals inside the display device 23 for display.

The operation of the television receiver 1 configured as above will be described including the operation of the user 3 . FIG. 3 is a diagram for explaining the hand movement of the user 3 and the control content of the television receiver 1 corresponding to the movement. A diagram of hand motions performed by a user 3 standing in front of the television receiver 1 is shown by arrows in FIG. 3(A). In this embodiment, it is assumed that the motions performed by the user are two motions of "swinging the hand vertically (up and down)" and "swinging the hand horizontally (left and right).

3(B)(1) to (B)(3) (hereinafter, (B)(1) will be denoted as B-1, and the others are the same) each shows a TV that performs control corresponding to the manual motion of the user 3. Transition of the contents displayed on the display device 23 of the receiver 1 . In the present embodiment, the control contents for the television receiver 1 are three controls of "turning the power from off to on", "displaying the menu screen", "clearing the menu screen", and "turning the power from on to off". .

The corresponding relationship between the manual motion performed by the user 3 and the control content of the television receiver 1 is that the action of "swinging the hand vertically" of the user 3 corresponds to "when the power supply of the television receiver 1 is turned off, turn on the power" and " When the power of the television receiver 1 is turned on, a control to display the menu screen ". The action of "swaying the hand" corresponds to the control of "turning off the power of the television receiver 1 no matter what screen state it is in".

FIG. 3(B-1) shows a state where the power of the television receiver 1 is turned off and nothing is displayed on the display device 23 . In this state, if the user 3 swings his hand vertically (up and down), the camera 2 will capture the action, turn on the power of the television receiver 1, and display a television picture as shown in Figure 3 (B-2) on the display device 23 (programme).

First, as shown in FIG. 3(B-1), nothing is displayed on the display device 23 , so the user cannot confirm the self-image captured by the camera 2 . Therefore, the user 3 must stand at a position that the camera 2 can definitely capture. Even if the user 3 is photographed at a certain position in the video image captured by the camera 2, the television receiver 1 also needs to be able to recognize the action of the user 3. In this case, there is no problem even without the display device 23 and the graphics generator 16 .

Further, in the state (viewing and listening state) of the TV picture shown in FIG. The menu screen shown in (B-3) then shifts to selection operations such as channel change. In this case, too, the display device 23 only initially displays the television screen, and the user cannot display his/her own video captured by the camera 2 on the display device 23 for confirmation. Therefore, the television receiver 1 must recognize the motion of the user 3 no matter where the user 3 is located in the image captured by the camera 2, similarly to the above.

Under the state (Fig. 3 (B-2)) that display device 23 displays the television screen, when user 3 swings the hand laterally (left and right), the power of television receiver 1 is turned off, and becomes as shown in Fig. 3 (B-1). status. Under the state (Fig. 3(B-3)) that shows all the screens such as menu, data broadcasting, EPG, etc., when the user swings the hand horizontally, it becomes as shown in Fig. 3(B-2) or Fig. 3(B-1). displayed status.

The action of swinging the hand vertically or horizontally used in this embodiment is the action content that people perform in daily life. The action of swinging the hand vertically generally has the meaning of greeting such as "come here, come here". It can also be said to be an appropriate action in terms of the meaning of entering (moving to) the next state. In addition, the movement of swinging the hand sideways is a movement at the time of parting to say "goodbye", and it can be said to be an appropriate movement in view of giving it the meaning of leaving a specific state. The meaning of this action varies depending on the country and race, and other actions can also be considered, but from the perspective of ease of use, I hope to follow the meaning of the action as much as possible.

Here, a simple and easy-to-understand control example of the television receiver 1 is given, but the content of the control may be appropriately changed according to the product design according to the functions of the television receiver 1 .

In addition, considering that the user 3 is moving away from the best viewing point when the power of the television receiver 1 is switched from off to on, it is desirable to make the shooting range of the camera 2 wide-angle to increase the range of the recognition action as much as possible. In addition, when changing the display screen from the state of watching a TV program to the menu screen, etc., it can be imagined that the user 3 is located near the sweet spot, so the shooting range of the camera 2 can be narrowed to a certain extent.

FIG. 4 is a diagram for explaining a detection area for detecting the hand movement of the user 3 . FIG. 4 shows the image of the user 3 captured by the camera 2 and the coordinates in the horizontal direction x and the vertical direction y. In this embodiment, the hand movement of the user 3 is recognized in a plurality of detection areas provided by dividing the screen of the image output from the camera 2 into 16 in the horizontal direction and 9 in the vertical direction. As shown in FIG. 4 , in a television receiver 1 having an aspect ratio of 16:9 horizontally: vertically for displaying an image on a display device 23, if the screen is divided into 16 horizontally and 9 vertically as described above, then One section formed by division in the vertical (y-axis) direction and the horizontal (x-axis) direction is a square. Each number of divisions is an integer of 2 or more, and may be appropriately set.

When detecting the hand movement of the user 3, the following two situations can be considered: use all 25 detection areas (16 detection areas divided into 9 detection areas set in the y-axis direction and 16 detection areas divided by the screen in the x-axis direction) Dimensions); and, the screen is divided into 16 in the x-axis direction and 9 in the y-axis direction, and one section formed by these divisions is used as one detection area, and all 144 detection areas (two-dimensional) are used Condition. If the number of detection areas is 25, the scale of hardware can be reduced, which is desirable. Also, when the number of detection areas is 144, the same processing as when the number of detection areas is 25 can be applied by converting the information from each detection area into information on the x-axis and y-axis.

First, as a first embodiment of the present invention, a case where all 25 detection areas are provided on a screen will be described. FIG. 5 is a diagram for explaining the division of the image screen output from the camera 2 into nine detection areas in the y-axis direction. 5 shows the image of the hand of the user 3 taken by the camera 2, nine detection areas and timing pulses divided in the y direction and represented by a dotted square, and the seventeenth to twenty-fifth detectors 317 to 25 corresponding to each detection area. Detector 325 (y-axis detector).

In each detection area, the center of the screen in the y-axis direction is set to 0, and coordinates indicating a positional relationship from -4 to +4 are given. The detection area whose y-axis coordinate is -4 corresponds to the 17th detector 317, the detection area whose y-axis coordinate is -3 corresponds to the 18th detector 318, the detection area whose y-axis coordinate is -2 corresponds to the 19th detector 319, and the y-axis The detection regions whose coordinates are −1 to +4 correspond to the 20th detector 320 to the 25th detector 325 respectively. Each of the y-axis detectors 317 to 325 is a detector that generates a detection signal based on a manual motion performed by the user 3 .

The respective y-axis detectors 317 to 325 operate based on timing pulses supplied from the timing pulse generator 12 . In Fig. 5, in the y-axis direction (vertical) and the x-axis direction (horizontal), respectively show the timing pulse for making the 19th detector 319 corresponding to the detection area whose y-axis coordinate is -2, and the timing pulse for making the 19th detector 319 act. 25 The detector 325 corresponds to the timing pulse of the detection area whose y-axis coordinate is 4.

The timing pulse shown in the x-axis direction is a pulse having a width corresponding to the width of the effective video period in the horizontal direction, and the timing pulse shown in the y-axis direction is a width that divides the width of the effective video period in the vertical direction into nine. Quite a pulse. The same timing pulses are also input to each of the other y-axis detectors.

FIG. 6 is a diagram for explaining that the screen of the image output from the camera 2 is divided into 16 detection areas in the x-axis direction. 6 shows the image of the hand of the user 3 taken by the camera 2, 16 detection areas and timing pulses divided in the x direction and represented by a dotted rectangle, and the first detectors 301 to 16 corresponding to each detection area. Detector 316 (x-axis detector).

In each detection area, the approximate center of the screen in the x-axis direction is set to 0, and coordinates indicating a positional relationship from -8 to +7 are given. The detection area whose x-axis coordinate is-8 corresponds to the first detector 301, the x-axis coordinate is the detection area of-7 corresponds to the second detector 302, and the x-axis coordinate is-6 The detection area corresponds to the third detector 303, x The detection areas whose axis coordinates are −5 to +7 correspond to the fourth detector 304 to the sixteenth detector 316 , respectively. Each of the x-axis detectors 301 to 316 is a detector that generates a detection signal based on a manual motion performed by the user 3 .

Each of the x-axis detectors 301 to 316 operates based on a timing pulse supplied from the timing pulse generator 12 . In Fig. 6, in the x-axis direction (horizontal) and the y-axis direction (vertical), respectively show the timing pulse for making the second detector 302 corresponding to the detection area whose x-axis coordinate is -7, and the timing pulse for making the second detector 302 act. 16 The detector 316 corresponds to the timing pulse of the detection area whose x-axis coordinate is 7. The timing pulse shown on the y-axis is a pulse having a width corresponding to the vertical width of the effective video period, and the timing pulse shown on the x-axis is equivalent to dividing the horizontal width of the effective video period into 16. pulse. The same timing pulses are also input to the other x-axis detectors, respectively.

As shown in FIG. 7 , the first detector 301 to the twenty-fifth detector 325 each have a first object extractor 51 , a timing gate 52 , and an object characteristic data detection unit 53 . The timing gate 52 controls the passage of the image signal from the camera 2 according to the timing pulses shown in FIGS. 5 and 6 .

The area through which the image signal passes is located in each detection area indicated by a dotted rectangle in FIGS. 5 and 6 . Various filtering processes described later are performed on the signal limited within the detection area, and the hand of the user 3 captured by the camera 2 is extracted.

The first object extractor 51 has a filter to which image features are added. In this embodiment, in order to detect the hand of the user 3 , in particular, it performs filtering processing focusing on skin color and motion detection filtering processing to detect motion. As shown in FIG. 8 , the first object extractor 51 specifically includes a specific color filter 71 , a grayscale limiter 72 , a motion detection filter 75 , a compounder 73 , and an object selector 74 .

The specific color filter 71 will be described with reference to FIG. 9 . FIG. 9 is a plane view of color difference, with R-Y on the vertical axis and B-Y on the horizontal axis. All color signals of TV signals can be represented by vectors on this coordinate, and can be evaluated by polar coordinates. The specific color filter 71 is means for limiting the hue and color density (saturation) of a color signal input with a color difference signal. To specify it, the hue is represented by an angle turned left with respect to the B-Y axis of the first quadrant (0 degrees). In addition, the saturation is a scalar quantity of the vector, and the origin of the color difference plane is a state where the saturation is 0 and there is no color. As the distance from the origin increases, the saturation increases, indicating that the color becomes thicker.

In FIG. 9, the range extracted by the specific color filter 71 is set to be smaller than the angle θ1 corresponding to the isochromatic line L1 and the range of the angle θ2 corresponding to the isochromatic difference line L2. In addition, the color density is set to The range is set to be larger than the iso-saturation line L3 and smaller than L4. This range of the second quadrant corresponds to the color of the human hand extracted in this embodiment, that is, the skin color area, but the extracted color area is not particularly limited thereto.

The specific color filter 71 calculates the angle and saturation from the color difference signal (R-Y, B-Y) input from the camera 2, and detects whether the color difference signal enters an area surrounded by isochromatic lines and iso-saturation lines.

As an example of angle calculation, the angle formed on the color difference plane shown in FIG. 9 is calculated for each input pixel by the angle calculation process shown in the flowchart of FIG. 10 . In this embodiment, the angle calculation processing is realized by hardware, but it may be realized by either software or hardware.

First, in step S401 shown in FIG. 10 , according to the signs of the R-Y and B-Y components of the color-difference signal of each input pixel, it is detected which quadrant the hue of the input pixel is located on the color-difference plane.

Next, in step S402, the absolute values |R-Y|, |B-Y| of the R-Y and B-Y components of the color-difference signal are compared, and the larger one is regarded as A and the smaller one is regarded as B for calculation.

Then, in step S403, the angle T1 is detected according to B/A. As can be seen from the processing in step S402, the angle T1 is 0° to 45°. Angle T1 can be calculated by polygonal line approximation, ROM table, etc.

In step S404, it is judged whether A is |RY|, that is, whether |RY|>|BY|. If the judgment is "No", that is, it is not |RY|>|BY|, then go directly to step S406. If the judgment is "Yes", that is |RY|>|BY|, then go to step S405 and replace the angle T1 with the angle T which is (90-T1). From this, tan ^-1 ((RY)/(BY)) is obtained.

The reason for setting the angle T1 detected in step S403 to 0° to 45° is that if the curvature of tan ^-1 ((RY)/(BY)) exceeds 45°, the gradient becomes sharply large, which is not suitable for calculating the angle.

Furthermore, in step S406, use the quadrant data detected in step S401 to determine whether it is the second quadrant, if it is the second quadrant, proceed to step S407, and calculate T=180-T1. If it is not the second quadrant, go to step S408 to judge whether it is the third quadrant, if it is the third quadrant, go to step S409 to calculate T=180+T1.

If it is not the third quadrant, go to step S410 to judge whether it is the fourth quadrant, if it is the fourth quadrant, go to step S411 to calculate T=360-T1. If it is not the fourth quadrant, that is, the first quadrant, the angle T is set to T1 in step S412. Then, finally in step S413, the angle T formed by each input pixel on the color difference plane in FIG. 9 is output.

Through the above processing, the angles of the input color difference signals R-Y, B-Y on the color difference plane can be obtained within the range of 0° to 360°. Steps S404 to S411 are processes for correcting the angle T1 detected in step S403 to an angle T. FIG. In addition, steps S404 to S411 correct the angle T1 according to the first quadrant to the fourth quadrant.

Next, calculation of saturation as color density is performed using the following formula.

Vc=sqrt(Cr×Cr+Cb×Cb)

Vc is a vector of scalars, here denoting saturation. Cr is the (R-Y) axis component of the color signal shown in FIG. 9, and Cb is the (B-Y) axis component. Also, sqrt() is an operator that takes a square root.

The processing here is not specific to software or hardware, but multiplication and square root are not easy to implement with hardware, and since there are many calculation steps in software, it is not desirable, so the following approximate formula can also be used.

Vc=max(|Cr|, |Cb|)+0.4×min(|Cr|, |Cb|)

Among them, max(|Cr|, |Cb|) is to select the larger one of |Cr| The operation processing of the smaller one.

Evaluate whether the angle (hue) T and saturation Vc obtained through the above steps fall within the range of the angle θ1 to θ2 of the isochromatic line, and whether the color density falls within the range smaller than the iso-saturation line L4 and greater than L3. The role of the specific color filter 71 shown in FIG. 8 is to pass signals entering this range.

As shown in FIG. 11 , the grayscale definer 72 of FIG. 8 defines a specific grayscale range of the luminance signal. In the case of an 8-bit digital signal, the maximum level Lmax and the minimum level Lmin of the gradation are arbitrarily set in 256 levels from 0 to 255, and the brightness of the gradation level included between Lmax and Lmin is output. Signal.

The motion detection filter 75 in FIG. 8 will be described using FIGS. 12 and 13 . As shown in FIG. 12, the motion detection filter 75 has a one-frame delayer 75-1, a subtractor 75-2, an absolute valuer 75-3, a nonlinear processor 75-4, and a quantizer 75-5. The luminance signal detects movement of the image.

In the one-frame delayer 75-1, the image signal from the camera 2 is delayed by one frame, and input to the subtractor 75-2. The subtracter 75-2 calculates the difference between the image signal from the camera 2 and the image signal from the one-frame delayer 75-1, and outputs it to the absolute valuer 75-3. The sign direction of the subtraction operation is not specified. Since the differential signal outputs values in both positive and negative directions according to the level of the signal, the absolute value unit 75-3 performs absolute value conversion on the differential value input from the subtractor 75-2, and supplies the value to the nonlinear processor 75-4. output.

The nonlinear processor 75-4 performs nonlinear processing on the input absolute-valued differential signal based on the input-output characteristics shown in FIG. 13 . In FIG. 13(A), the horizontal axis represents the absolute valued differential signal input from the absolute value unit 75-3, and the vertical axis represents the signal output from the nonlinear processor 75-4. The a value and the b value are variable within the ranges R1 and R2, respectively.

The output signal of the nonlinear processor 75-4 is input to a quantization unit 75-5, and binarized based on a predetermined threshold shown in FIG. 13(B).

The multiplexer 73 in FIG. 8 multiplexes the signals input from the specific color filter 71, the gradation limiter 72, and the motion detection filter 75, and converts them into area pulses. In the present embodiment, when all of the signal passing through the specific color filter 71 , the signal passing through the gradation limiter 72 , and the signal passing through the motion detection filter 75 exist, an area pulse at a high level is output.

The area pulse generated by the multiplexer 73 is supplied to the object gate 74 . When the area pulse is high, the object gate 74 passes the luminance signal and the color difference signal. When the area pulse is at a low level (a range other than the area pulse), the input signal (luminance signal and color difference signal) is not passed, and a signal of a predetermined value is output. In this embodiment, a luminance signal of a black level and a color difference signal of 0 saturation are output.

The specific color filter 71 limits the hue (angle) and saturation of the color signal input by the color difference signal, the gradation limiter 72 limits the specific gradation range of the luminance signal, and the motion detection filter 75 limits the luminance signal according to the movement of the image.

By limiting the hue and saturation with the specific color filter 71, it is possible to focus on the skin color of a person. However, the skin color of a person varies depending on the sun exposure and also varies according to race. Therefore, if the specific color filter 71 is used to adjust the hue and saturation according to the control signal input from the control information determiner 20, and the grayscale limiter 72 is used to adjust the grayscale limit range of the luminance signal, then a human hand can be roughly detected. Furthermore, a human hand can be extracted and recognized by the motion detection filter 75 based on the motion of the image.

FIG. 14(A) is a diagram in which the output signal of the first object extractor 51 is displayed on the display device 23 . In the image captured by the camera 2, the image of the hand is displayed based on the signal extracted by the first object extractor 51, and since the luminance signal is set to a black level in parts other than the image of the hand, nothing is displayed. Based on the extracted signal, the feature of the image, the position on the screen, and the content of the action are analyzed, and it is recognized that the user 3 has made an intentional action.

FIG. 14(B) shows an image of an image signal gated by the timing gate 52 in FIG. 7 based on a timing pulse for operating each detector provided corresponding to each detection area. Here, as a representative example, the slave timings of the 21st detector 321 corresponding to the detection area whose y-axis coordinate is 0 (zero) and the 20th detector 320 corresponding to the detection area whose y-axis coordinate is -1 are shown. The output signal output by the gate 52.

The object characteristic data detecting unit 53 performs filtering processing for detecting the characteristic from the image signal shown in FIG. 14(A). As shown in FIG. 15 , the object feature data detection section 53 has functional blocks for detecting various features from an image, that is, a histogram detector 61, an average luminance (APL) detector 62, a high-frequency generation amount detector 63, Minimum value detector 64 and maximum value detector 65 . Although there are other requirements for specifying image features, in this embodiment, based on the detection signals generated from these detectors 61-65, it is generated to indicate that the first motion detector 20-1 to the fifth motion detector 20-1 5. With the detection signal of the hand area detected in the detection area, it is judged that the video signal is a signal of a hand, and the movement of the hand is recognized.

The histogram detector 61, the average luminance (APL) detector 62, the high-frequency generation amount detector 63, the minimum value detector 64, and the maximum value detector 65 are constituted by hardware in this embodiment, and each screen unit (field and frame) Unit: vertical cycle unit) Generates data (detection signal) representing each feature in the detection area in the space, and sends it to the control information determiner 20 via the CPU bus.

The control information judging unit 20 stores the data transmitted from the detectors 61 to 65 as variables on software, and performs data processing.

The histogram detector 61 divides the gradation of the luminance signal output from the timing gate 52 into, for example, eight levels, counts the number of pixels existing in each level, and represents each screen (one field or one frame) The data of the histogram is output to the first motion detector 20-1. The average luminance detector 62 outputs to the second motion detector 20-2 the average luminance value of one screen obtained by adding the luminance levels in one screen in the same way and dividing by the number of all pixels.

The high-frequency generation amount detector 63 extracts high-frequency components through a spatial filter (two-dimensional filter), and outputs the generation amount of high-frequency components within one screen to the third motion detector 20-3. The minimum value detector 64 outputs the minimum gradation value of the luminance signal in one screen to the fourth motion detector 20-4, and the maximum value detector 65 outputs the gradation value of the luminance signal in one screen to the fifth motion detector 20-5. Maximum grayscale value.

The first motion detector 20-1 to the fifth motion detector 20-5 store the received data as variables, and process the data by software. The processing of detecting a manual motion to be described later is performed by software in this embodiment. The control information determiner 20 has a control information generator 20-10 that generates a control signal based on detection signals from the first motion detector 20-1 to the fifth motion detector 20-5.

FIG. 16 shows the results of modeling the data output from the histogram detector 61 and the average luminance detector 62 in the object characteristic data detection section 53 . FIG. 16 is a histogram in which the horizontal axis represents 8-level gradation divided into 0 to 7, and the vertical axis represents frequency. The average luminance (APL) is represented by an arrow to give a sense of its size.

FIG. 16(A) shows the outputs of the histogram detector 61 and the average luminance detector 62 constituting the twentieth detector 320 of FIG. 14(B). As shown in FIG. 14(B), the hand is not placed in the detection area in the twentieth detector 320, so the hand is not detected in the first object detector 51, and the signal output from the first object extractor 51 is processed with a black level. shield. Therefore, the histogram shown in FIG. 16(A) contains only the lowest grayscale (0) portion. Also, basically the signal is black and the APL is 0 (zero), but it is indicated by a short arrow to show that the signal level is low.

FIG. 16(B) shows the outputs of the histogram detector 61 and the average luminance detector 62 constituting the 21st detector 321 of FIG. 14(B) . As shown in FIG. 14 (A), in the 21st detector 21, the hand placed in the detection area is detected with the first object extractor 51, therefore, the output of the histogram detector 61 shown in FIG. 16 (B) is , except for the gray level 0 of the masked black level, the frequency distribution is on the gray level of the brightness of the hand. In addition, APL is indicated by a long arrow because the average luminance increases due to the signal component of the hand.

In the present embodiment, the sum of the data output from the histogram detector 61 other than the lowest gradation (0) is obtained as data representing the area of the hand placed in the detection area. That is, the object extractor 51 of the detector set according to the corresponding detection area extracts the output signal of the moving hand, the histogram detector 61 generates the first detection data, and the first motion detector 20-1 based on the first detection data, Second detection data representing the area of the hand extracted from the detection area is generated.

Furthermore, the frequency is calculated by the histogram detector 61 from the two-level gradation divided into black and other components, and it is also possible to extract the hand placed in the detection area by performing a predetermined movement. Therefore, the second detection data representing the region of the hand can also be obtained from the first detection data from the histogram detector 61 which is simplified to 0 gray level and other two gray levels.

Furthermore, in this embodiment, based on the first detection data output from the histogram detector 61, the second detection data is generated in the first motion detector 20-1. The first detection data output by the object feature data detector 53 included in the devices 301 to 325 may be used to generate the second detection data in the control information determiner 20 .

FIG. 17 is an example of a hand image captured by the camera 2 when the user 3 moves the hand vertically (up and down) within the area captured by the camera 2 . At the same time, arrows indicating the direction of hand movement and xy coordinates of the detection areas arranged in the screen are shown. In Fig. 17(A), (B), (C), and (D), four positions of the moving hand are drawn and shown. Fig. 17(A) shows the situation where the hand exists at the top, Fig. 17(B) shows the situation where the hand is swung slightly downward, Fig. 17(C) shows the situation where the hand is further swung down, Fig. 17(D) shows The case where the hand is at the bottom.

In this embodiment, the hand is moved up and down four times. That is, taking (A), (B), (C), (D), (D), (C), (B), and (A) of FIG. 17 as one cycle, the hand is moved for four cycles. In the case of such up and down motions, the hand hardly moves in the x-axis direction and is located on the same coordinates. On the other hand, in the y-axis direction, the coordinates of the hand fluctuate up and down. Therefore, the detected data has four cycles in which the upper and lower peaks are repeated, and is expressed as a fluctuation value of the output data from each detector provided corresponding to the detection area of each coordinate.

FIG. 18 shows, as a table, the data values output by the histogram detectors 61 of the detectors 301 to 325 among the detection results of the vertical movement of the hand shown in FIG. 17 and the contents of the processed data. The leftmost column of this table is the item name, and the data value of each item that changes with the lapse of time is shown on the right side of the item name column.

The cycle of the item represents the above-mentioned cycle of the up and down movement of the hand, and is represented by writing the first two cycles out of all four cycles in this table. The item n indicates the frame number of the image, and in the case of a general video signal, it is set to a 60 Hz cycle, and in the case of an interlaced scan, one frame is composed of two fields, and one vertical cycle is set to a 60 Hz cycle.

The item ph indicates the position of the hand moving up and down, and A, B, C, and D correspond to (A), (B), (C), and (D) in Figure 17, respectively. The items x(i) (i=-8 to +7) respectively represent the corresponding detection regions obtained from the histogram detectors 61 of the first detector 301 to the sixteenth detector 316 as described above. The first detection data represents the second detection data of the area of the hand. Similarly, y(j) (j=-4 to +4) in the item represents the value obtained from the histogram detector 61 of the seventeenth detector 317 to the twenty-fifth detector 325 based on the corresponding detectors 301 to 325 The first detection data of the hand extracted from the detection area represents the second detection data of the hand area. In addition, the items XVS, XVSG, XG, YVS, YVSG, and YG are contents in which data obtained from the respective detectors 301 to 325 are processed, and will be described in detail later.

In an example shown in FIGS. 17(A) to (D), the hand moves up and down, but the position of the moving hand does not change in the x-axis direction, so the data of the item x(i) does not change. As shown in Figure 17(A)-(D), the hand is located on x-coordinates 4-6 with x-coordinate 5 as the center, and on items x(4), x(5) and x(6) of the table in Figure 18 Values for which a hand was detected are shown. The other items x(i) are masked by the first object extractor 51, so the value is 0 (except the items x(1), y(-2), y(-3) of frame number 11).

This is only an ideal situation. If the part representing the skin color other than the hand of user 3 is moving, the detected value will also be generated in the coordinates other than the coordinates of the detection area where the hand is placed, which is noise for the detection of hand movement. . The focus is how to suppress such noise and recognize hand movements as operational information.

In the y-axis direction, the data of the item y(j) changes as the hand is moved up and down. In FIG. 17(A), the hand is located on the y coordinates 2 and 3, so the detected values are shown in the column of frame number 0 in the items y(2) and y(3) of FIG. 18 . Similarly, in FIGS. 17(B), (C), and (D), the respective detected values are shown in the item (y) corresponding to the respective y-coordinates where the hand is placed.

The values of data (second detection data) of items x(i) and y(j) shown in FIG. 18 are based on signals detected by the histogram detector 61 . In this embodiment, the first detection is performed by setting a value of 100 (area) in one section where the screen is divided into 16 in the x-axis direction and 25 detection areas in the y-axis direction. Data scaling. The second detection data is data representing the size of the area of the hand placed in the detection area based on the first detection data generated based on the output signal of the extracted hand in the detection area.

In this embodiment, it is more important than the change in the value indicated by each item with the passage of time, that is, the change in the second detection data based on the outputs of the first detector 301 to the twenty-fifth detector 325 . That is, the fluctuation of the data indicating the movement of the center of gravity is calculated based on the sum of the second detection data based on the first detection data output from the plurality of detectors. Therefore, the center of gravity (hereinafter simply referred to as "the center of gravity of the hand") in which the hand is placed is obtained based on the output signals extracted from each of the plurality of detection areas based on the placed hand, and is compared. did an evaluation.

The center of gravity XG of the hand on the x-coordinate with the frame number n can be obtained by the following mathematical expression 1. XVS is the sum of the second detection data of each x-axis detector (the first detector 301 to the sixteenth detector 316), and is based on the output extracted by the first object extractor 51 in a plurality of detection areas by the moving hand The value of the signal. XVSG is the sum of weighted detection data obtained by multiplying the second detection data of each x-axis detector by the x-coordinate of the corresponding detection area.

[mathematical formula 1]

$\mathrm{<span\; class="notranslate">\; XG</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; XVSG</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; XVS</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 8</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 7</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 8</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 7</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

In this embodiment, the item XG in FIG. 18 is 5 in almost all frames (except frame number 11), so the x-coordinate of the center of gravity of the hand is 5, and the data increases around the x-coordinate 5.

The center of gravity YG of the hand on the y-coordinate with the frame number n is obtained by the following Mathematical Expression 2. YVS is the sum of the second detection data of each y-axis detector (the 17th detector 317 to the 25th detector 325), and YVSG is the y coordinate of the corresponding detection area multiplied by the second detection data of each y-axis detector , is the sum of the weighted detection data.

[mathematical formula 2]

$\mathrm{<span\; class="notranslate">\; YG</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; YVSG</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; YVS</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 4</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 4</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>}{\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 4</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 4</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

In this embodiment, the item YG in FIG. 18 becomes 2.5 at frame number 0. This means that the y-coordinate of the center of gravity of the hand is 2.5. For other frames, the value of the item YG also indicates the y-coordinate of the center of gravity of the hand in that frame. In the present embodiment, the value of the item YG is a value in the range of 0 to 2.5 (except frame number 11), and the change in the value of the item YG indicates the vertical movement of the hand on the coordinates.

In this embodiment, the fluctuation of the center of gravity YG is analyzed, and the motion of the hand is recognized as operation information. FIG. 19 is a timing chart showing the change in the coordinates of the center of gravity of the hand along the passage of time. FIG. 19(A) shows the change in the y-coordinate of the center of gravity of the hand, that is, the change in the value of the item YG in FIG. 18 , which shows fluctuations over four periods in the range of 0 to 2.5. FIG. 19(B) shows the change in the x-coordinate of the center of gravity of the hand, that is, the change in the value of the item XG in FIG. 18 . As shown in FIG. 17, the hand is swung vertically with the x-coordinate 5 as the center of gravity, but does not change in the horizontal direction. Therefore, in principle, it becomes a straight line with a constant level as shown in FIG. 19(B).

Although the waveforms of the x and y axes should be analyzed, the protection against misidentification will be described first. The first cycle of the watch shown in FIG. 18 is ideal data when the hand is swung vertically. The data of the item x(i) of the x-coordinates other than the x-coordinates 4, 5, and 6 of the hand extracted is 0. The same is true for the y coordinate, and the data outside the detection area where the hand is drawn is 0. However, actually, even if various filtering processes are performed by the first object extractor 51, there may be omissions, so it is conceivable that unexpected data (noise) other than hand motions may be generated.

In the frame number 11 of the second period, in addition to the data related to the hand, there is also the second detection data, which indicates that in each detection area, the detector corresponding to x(1) detects the equivalent The region of the hand (object) in region 100 is detected by the detector corresponding to y(-2) and the region of the hand (object) corresponding to region 50 is detected by the detector corresponding to y(-3). The area of the hand (object). These data may disturb the center-of-gravity coordinates of the detected hands. As shown in FIGS. 17(A) to (D), the x-coordinate of the center of gravity of the hand is 5 and is constant, but the value of the item XG of frame number 11 shows 3.361. Also, in the y-coordinate of the center of gravity of the hand at frame number 11, the value of the item YG should be 0 as in frame number 3, but it is -1.02, and both the x-axis and the y-axis are values affected by noise.

If the noise occurs accidentally, it can be suppressed by an outlier removal filter (intermediate value filter) often used in digital signal processing, but if it can pass through the filter, or the number and amount of noise are large This is the main reason for the reduction of recognition rate.

In the present embodiment, in order to effectively suppress this noise, a process of turning off the timing gate 52 of unnecessary detectors is performed. In the table shown in FIG. 18 , the control information determiner 20 confirms that the added value of the second detection data accumulated during a predetermined period exceeds a certain value (threshold value th1x in the case of an x-axis detector; threshold value th1x in the case of a y-axis detection A detector is a detector of a threshold value th1y), that is, a detector representing a maximum value, wherein the second detection data is based on the outputs of the respective detectors from the x-axis detectors 301-316 and the y-axis detectors 317-325.

As shown in the table of FIG. 18 , there is no detector exceeding the threshold value th1y based on the fluctuation of the second detection data (output signal) of the first detection data output from the y-axis detectors 317 to 325 . On the other hand, among the second detection data based on the first detection data output from the x-axis detectors 301 to 316, the second detection data x(5) based on the output from the fourteenth detector 314 corresponding to the x-coordinate 5 represents The maximum value, the accumulated result value exceeds the threshold th1x at a certain moment, so it is judged to be the corresponding detector. From this, it is found that the movement of the hand is a movement of swinging up and down. In addition, for hereinafter convenience, the second detection data based on the first detection data output from a predetermined detector will only be shown as the second detection data of a predetermined detector.

FIG. 19(C) shows the process of accumulating the second detection data x(5) of the fourteenth detector 314 corresponding to the x-coordinate 5 . When the accumulated value exceeds the threshold th1x (frame 9), the activation flag Flg_x changes from 0 to 1 for a predetermined period. When the added value exceeds the threshold th1x, the control information determiner 20 as a flag generator generates an active flag Flg_x. While the activation flag Flg_x is at 1, unnecessary sections or hands (objects) in the detection area are not detected as will be described later. Here, the accumulated value exceeds the threshold th1x in frame 9, but it only needs to exceed the threshold th1x within a predetermined period.

The predetermined period during which the activation flag Flg_x rises is taken as the activation period, and its length is set to a period of four cycles required to recognize a manual motion. FIG. 19(D) will be described later.

FIG. 21 is a diagram for explaining which position on the screen the detection area is valid. In the present embodiment, it is effective when the detection area (section) is used to detect the motion of the placed hand. FIG. 21 depicts an image of a hand swaying longitudinally on the x-coordinate 5 captured by the camera 2 , noise components represented by black boxes, and timing pulses supplied to control the 21st detector 321 .

The first x-axis timing pulse drawn by a dotted line in the x-axis direction is a pulse having a width corresponding to the width of the effective video period in the horizontal direction, and is supplied to all the y-axis detectors ( 17th detector 317 to 25th detector 325).

The second x-axis timing pulse drawn with a solid line is generated when the active flag Flg_x is generated (becomes 1) based on the added value which is a detection corresponding to the detection area where the swinging hand is extracted within a predetermined period. The value obtained by accumulating the output signal of the device. The second x-axis timing pulse is a pulse having a width corresponding to a predetermined width in the horizontal direction of the effective video period, and is supplied to all the y-axis detectors 317 to 325 . Based on the second x-axis timing pulse, each of the y-axis detectors 317 to 325 outputs only the detection signal of the minimum detection area required to detect the hand placed in the detection area.

A method of generating the second x-axis timing pulse will be described using FIG. 22 . The x-axis timing pulse initially supplied to each of the y-axis detectors 317-325 is the first x-axis timing pulse. The first x-axis timing pulse is a timing pulse for validating the entire width in the x-axis direction of each detection area corresponding to each of the y-axis detectors 317 to 325 .

When the hand is extracted from the detection area at the x-coordinate 5 by the movement of the hand shown in FIG. The detection data, compared with the second detection data of other detectors, continuously takes the maximum value (refer to FIG. 18). When the value of the second detection data accumulated by the fourteenth detector 314 exceeds the threshold th1x, an activation flag Flg_x is generated (turned to 1). The control information determiner 20 confirms that the active flag Flg_x has been generated, and sets the x-axis control data of the detection area at the x-coordinate 5 to 1.

In this embodiment, considering that the size of the hand on the screen changes slightly depending on the distance between the television receiver 1 (camera 2) and the user 3, at least the detector corresponding to the activation flag Flg_x is included. The x-axis control data of the detection area of , and the detection area adjacent to the detection area is set to 1. For example, the x-axis control data of the detection area with x-coordinates 4 and 6 is set to 1. In addition, the x-axis control data of other detection areas are set to 0.

Control information controller 20 supplies x-axis control data as described above to timing pulse generator 12; the x-axis timing pulse within timing pulse generator 12 activates controller 12x to generate a second x-axis control data based on the input x-axis control data. Timing pulses are supplied to all y-axis detectors 317-325. Therefore, in the state shown in FIG. 21 , the second x-axis timing pulse having a width corresponding to the width of the detection area whose x-coordinates are 4 to 6 is generated. That is, the timing generator 12 generates the second x-axis timing pulse narrowing the width of the first x-axis timing pulse. Each of the y-axis detectors 317 to 325 to which the second x-axis timing pulse is supplied outputs a detection signal only from the section where the x-coordinates of the corresponding detection areas are 4 to 6 . As a result, noise components occurring at coordinates (x, y)=(1, -2), (1, -3) shown in FIG. 21 were not detected.

When the second x-axis timing pulse is generated, the control information determiner 20 performs subsequent control based on outputs from the respective y-axis detectors 317 to 325 . The detection signals detected by the respective x-axis detectors 301 to 316 are not referred to. In addition, the output of the detection signal may be stopped without supplying a timing pulse to the timing gate 52 of each of the x-axis detectors 301 to 316 .

23 shows the second x-axis timing pulse supplied to the 17th detector 317 to the 25th detector 325 of the y-axis detector and the timing pulse (y axis direction). Each of the y-axis detectors 317 to 325 outputs only signals detected from three intervals that overlap with the corresponding detection area and the detection area corresponding to x-coordinates 4 to 6 based on the second x-axis timing pulse. By doing so, it is possible not to detect a section in which the hand is not drawn and which does not need to be detected within the detection area.

In addition, in this embodiment, the method of controlling the pulse width in units of detection regions as shown in Fig. 22 and Fig. 23 is adopted, but the pulse width can also be flexibly controlled by using a method such as specifying the start point of the pulse and the pulse width. circuit method.

The table shown in FIG. 24 is substantially the same as the table shown in FIG. 18 , but shows the second x generated after the activation flag Flg_x of the fourteenth detector 314 shown in FIG. 19(C) becomes 1. axis timing pulse, and limit the second detection data based on the output signals from the respective detectors 301 to 325 obtained from the detection in the section that does not require detection or the detection in the self-detection area. In FIG. 19(C), the second detected data of frame number 10 and later exceeding the threshold th1x correspond to this, and x(1), y(- 3) and y(-2) become 0. This is because, in the interval of coordinates (x, y)=(1,-2), (1,-3), each timing gate device of the 18th detector 318 and the 19th detector 319 correspondingly arranged 52 supplies the second x-axis timing pulse, which is not detected.

By removing the noise components, the disturbance of the values of the center of gravity XG and YG is eliminated, and the recognition of the first motion detector 20-1 to the fifth motion detector 20-5 in the subsequent stage of each of the y-axis detectors 317 to 325 is improved. Rate. In addition, in the present embodiment, noise components are affected up to frame 9, but until this point, the activation flag Flg_x is set to 1, so that the maximum value of the accumulated value does not vary. Composition has no effect on detection.

The first motion detector 20-1 to the fifth motion detector 20-5 in the control information determiner 20 receive and process the data shown in FIG. 24 . Returning to Fig. 19, the processing for detecting how the hand moves will be described.

FIG. 19(A) shows the variation of the y-coordinate YG of the center of gravity, and FIG. 19(B) shows the variation of the x-coordinate XG of the center of gravity, respectively showing waveforms without noise. The activation flag Flg_x becomes 1 when the value of the accumulated output signal of the x-axis detector (fourteenth detector 314 ) shown in FIG. 19(C) becomes equal to or greater than the threshold th1x. Each interval in the y-axis direction other than a plurality of intervals formed by the intersection of the detection area in the x-axis direction and the detection area in each y-axis direction is passed by the second x-axis timing pulse supplied to each of the y-axis detectors 317-325. is disabled, the nearby detection area in the x-axis direction includes at least a detection area adjacent to the detection area corresponding to the detector that generated the activation flag Flg x. That is, it is not used for hand detection. Therefore, it is not affected by noise.

If the waveform shown in FIG. 19(C) continues to be above the threshold th1x, the second x-axis timing pulse is continuously supplied to each of the y-axis detectors 317 to 325. Therefore, the unnecessary section continues to be an invalid period and continues to receive no noise. the effect of the influence. When the waveform shown in FIG. 19(C) becomes equal to or less than the threshold value th1x, the accumulated value is reset. However, the value used as the reset reference is not limited to the threshold th1x.

Next, processing for suppressing the DC offset of the waveform in FIG. 19(A) is performed so that the average value of the waveform becomes substantially zero. The high-pass filter shown in FIG. 20 is used for this processing.

The delay unit 81 in FIG. 20 performs a delay of 4 frames (time Tm) in this embodiment. Subtractor 82 takes the difference between the delayed signal and the undelayed signal. Here, the sign is not important and does not affect the final result. Finally, a 1/2 multiplier 83 is used to adjust the ratio. The waveform in FIG. 19(A) passes through the high-pass filter in FIG. 20 , and as a result, as shown in FIG. 19(D ), the average value of the waveform becomes approximately zero. In this way, the position information of the hand on the y-axis of the swing is excluded, and a waveform suitable for analysis of the content of the hand movement can be obtained. Note that the center of gravity YGH shown on the vertical axis of FIG. 19(D) is a value obtained by performing high-pass filter processing on the center of gravity YG shown on the vertical axis of FIG. 19(A).

Next, returning to FIG. 15, the first motion detector 20-1 to the fifth motion detector 20-5 will be described. The first motion detector 20-1 to the fifth motion detector 20-5 have cross-correlation digital filters not shown. In this embodiment, in order to recognize the operation information of the hand movement, it is only necessary to swing the hand up and down or left and right four times at most. That is, it is determined in advance what kind of movement is to be recognized, and therefore, the cross-correlation digital filter obtains a representative detection signal waveform of a certain predetermined predetermined movement (vertical swing movement) determined in advance, and an The detection signal generated by the actual motion is correlated with the detection signal waveforms generated by the first motion detector 20-1 to the fifth motion detector 20-5, and the consistency is evaluated to recognize the operation information of the hand motion.

In this embodiment, the waveform shown in FIG. 25(G) is used as the reference signal waveform of the pitching motion (representative detection signal waveform for a predetermined motion), and the cross-correlation digital filter shown in FIG. 25(F) Among the tap coefficient values k0 to k40 of , values corresponding to the waveform of the reference signal are used. 25(D) shows the detection signal waveform input to the cross-correlation digital filter kn and generated by actual operation, but this is the same waveform as the signal waveform in FIG. 19(D) . The cross-correlation digital filter multiplies the tap coefficient by the second detection signal based on the signal that detected the actual motion, and the first motion detector 20-1 to the fifth motion detector 20-5 based on the output from the cross-correlation digital filter The signal waveform is used to detect whether the action performed by the user 3 is a vertical swing action. The output signal wv(n) of the cross-correlation digital filter kn is obtained using the following Mathematical Expression 3.

[mathematical formula 3]

$\mathrm{<span\; class="notranslate">\; wv</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

N is the number of taps of the digital filter, and here is 41 taps (0 to 40). y(n+i) is the filtered center of gravity YGH shown on the vertical axis of FIG. 25(D). The cross-correlation digital filter kn realizes its function by operating only when the activation flag Flg_x becomes 1.

The cross-correlation digital filter output signal wv(n) has a waveform as shown in FIG. 25(E), and the amplitude of the cross-correlation increases as the consistency of the cross-correlation increases. In addition, FIG. 26(D) is the same as FIG. 19(D) and FIG. 25(D), and is shown as a comparison with FIG. 26(E). Take the absolute value of the output signal wv(n) and carry out cumulative integration. When the value reaches the threshold value th2v or more, it is judged that the cross-correlation with the reference signal waveform is sufficient, and it is recognized that a predetermined action (here, vertical swing action). The first motion detector 20 - 1 to the fifth motion detector 20 - 5 are motion detectors that detect whether or not the motion of the user 3 is a predetermined motion based on the detection signal output from the detection unit 19 .

While identifying this action, confirm that it is a vertical swing action and that the activation flag Flg_x that takes over the role of the protection window is "1", and the vertical swing operation of the hand is determined, and an event corresponding to the state of the television receiver 1 is performed (control ). This event is performed based on a signal output by the control information generator 20-10 after logically judging that any one of the plurality of motion detectors 20-1 to 20-5 shown in FIG. 15 is confirmed.

Next, the movement of swinging the hand sideways (bye-bye) will be described. In this embodiment, vertical and horizontal motions are automatically distinguished and can be executed simultaneously. FIG. 27 is an example of a hand image captured by the camera 2 when the user 3 moves the hand laterally (left and right) within the area captured by the camera 2 . At the same time, arrows indicating the direction of hand movement and xy coordinates of the detection areas arranged in the screen are shown. In Fig. 27 (A), (B), (C), (D), four positions of the moving hand are shown. Figure 27(A) shows the situation where the hand is at the far left, Figure 27(B) shows the situation where the hand is moved slightly to the right, Figure 27(C) shows the situation where the hand is moved further to the right, and Figure 27(D) shows the situation where the hand is moved further to the right. The rightmost case.

In this embodiment, the hand is moved left and right four times. That is, taking (A), (B), (C), (D), (D), (C), (B), and (A) of FIG. 27 as one cycle, the hand is moved for four cycles. When performing such a right-and-left movement, the hand hardly moves in the y-axis direction, and is located on the same coordinates. On the other hand, in the x-axis direction, the coordinates of the hand fluctuate left and right. Therefore, the detected data has four cycles in which the left and right peaks are repeated, and is expressed as a fluctuation value of output data output from each detector provided in a detection area corresponding to each coordinate.

FIG. 28 is a table showing the data values output by the histogram detectors 61 of the detectors 301 to 325 in the detection results of the left and right hand movements shown in FIG. 27 and the contents of the processed data values. This table is made in the same format as the table of Fig. 18, and the data values correspond to the left and right movements of the hand.

In the example shown in FIG. 27(A) to (D), the hand moves in the left and right directions, but the position of the moving hand does not change in the y-axis direction, so the item y(j) (j=-4 to +4) data does not change. As shown in Figure 27(A)-(D), the hand is located on y-coordinates 1-3 centered on y-coordinate 2, and items y(1), y(2) and y(3) in Figure 28 show The hand value is detected. The other items y(j) are masked by the first object extractor 51, and their values become 0 (except the items x(7), x(4), and y(-1) of frame number 11).

In the x-axis direction, the hand moves left and right, so the data of the item x(i) changes. In FIG. 27(A), the hand is located on x coordinates -6, -5, -4, and in the column of frame number 0 of items x(-6), x(-5) and x(-4) in FIG. 28 The detected values are shown in . Similarly, in FIGS. 27(B), (C) and (D), each detected value is shown in the item x(i) corresponding to the x-coordinate of each hand placement.

The center of gravity XG of the hand on the x-coordinate with the frame number n can be obtained by using the above-mentioned mathematical expression 1. In this embodiment, the item XG in FIG. 28 is -5.3 at frame number 0. This means that the x-coordinate of the center of gravity of the hand is -5.3. For other frames, the value of the item XG indicates the x-coordinate of the center of gravity of the hand in that frame. In the present embodiment, the value of the item XG is a value in the range of -5.3 to -2.3 (except frame number 11), and the change in the value of the item XG indicates the left and right movement of the hand on the coordinates.

The center of gravity YG of the hand on the y-coordinate with the frame number n can be obtained by using the above-mentioned Mathematical Expression 2. In the present embodiment, the item YG in FIG. 28 is 2.19 in almost all frames (except frame number 11). Therefore, the y-coordinate of the center of gravity of the hand is 2.19, and the y-coordinate 2.19-bit center data increases.

FIG. 29 is a timing chart showing changes in the coordinates of the center of gravity of the hand over time. Fig. 29(A) shows the change of the y coordinate of the center of gravity of the hand, that is, the change of the YG value of the item in Fig. 28. As shown in Fig. 27, the y coordinate 2.19 is used as the center of gravity to swing the handle horizontally, so there is no change in the vertical direction. From the principle In other words, as shown in FIG. 29(A), it becomes a straight line of a certain level. FIG. 29(B) shows the change of the x-coordinate of the center of gravity of the hand, that is, the change of the item XG value of FIG. 28 , which shows fluctuations in four cycles between -5.3 and -2.3.

Although the waveforms of the x and y axes should be analyzed, the first cycle of the table shown in FIG. 28 becomes ideal data when the hand is swung sideways. The data of the item y(j) of the y-coordinates other than the y-axis coordinates 1, 2, and 3 of the hand are extracted is 0. The same is true for the x-coordinate, and the data outside the detection area where the hand is drawn is 0.

However, in frame number 11 of the second cycle, in addition to the hand-related data, there is second detection data indicating that in each detection area, the detector corresponding to y(-1) detects The area of the hand (object) corresponding to area 120 is detected, and the area of the hand (object) corresponding to area 50 is detected by the detector corresponding to x(4), and the area corresponding to the hand (object) is detected by the detector corresponding to x(7). 70 for the hand (object) area. These data disturb the center-of-gravity coordinates of the detected hands. As shown in FIGS. 28(A) to (D), although the y-coordinate of the center of gravity of the hand is fixed at 2.19, the value of the item YG of frame number 11 is indicated as 1.351. Also, the x-coordinate of the center of gravity of the hand at frame number 11 should be the same as frame number 3, and the value of the item XG should be -2.3, but it becomes -0.45, and both the x-axis and the y-axis are affected by noise. value.

When the hand is moved horizontally, as in the case of moving the hand vertically, the process of turning off the timing gate 52 of unnecessary detectors is performed. In the table shown in FIG. 28, it is confirmed that the added value of the second detection data accumulated during the specified period exceeds a certain value for the first time (in the case of an x-axis detector, it is a threshold th1x; in the case of a y-axis detector, it is a threshold th1y ), that is, a detector indicating the maximum value, and the second detection data is based on the first detection data output from each detector of the x-axis detectors 301-316 and the y-axis detectors 317-325.

As shown in the table of FIG. 28 , among the x-axis detectors 301 to 316 , there is no detector whose second detection data (output signal) fluctuates and exceeds the threshold value th1x. On the other hand, among the y-axis detectors 317 to 325, the second detected data (y(2)) of the 23rd detector corresponding to the y-coordinate 2 shows the maximum value, and the accumulated value exceeds the threshold value at a certain moment. th1y, is judged as the corresponding detector. From this, it was found that the movement of the hand is a movement of swinging from side to side.

FIG. 29(C) shows the process of accumulating the second detection data y(2) of the 23rd detector 323 corresponding to the y-coordinate 2 . When the accumulated added value exceeds the threshold value th1y (frame 9), the active flag Flg_y changes from 0 to 1 for a predetermined period. When the added value exceeds the threshold th1y, the control information determiner 20 as a flag generator generates an active flag Flg_y. While the activation flag Flg_y is at 1, unnecessary sections or hands in the detection area are not detected as will be described later. In addition, in this figure, the cumulative addition value exceeds the threshold value th1x in frame 9, however, it is only necessary to exceed the threshold value th1x within a predetermined period.

The predetermined period during which the activation flag Flg_y rises is taken as the activation period, and its length is set to a period of about four cycles required to recognize the movement of the hand. FIG. 29(D) will be described later.

FIG. 30 is a diagram for explaining at what position on the screen the detection area is valid. 30 depicts the image of the hand moving laterally at the y-coordinate 2.19 captured by the camera 2, the 2 noise components indicated by black boxes, and the timing pulses supplied by the sixth detector 306 for control. The first y-axis timing pulse drawn by a dotted line in the y-axis direction is a pulse having a width corresponding to the width in the vertical direction of the effective image period, and is transmitted to all x-axis detectors at the moment when the user 3 starts to shake his hand. (1st detector to 16th detector) supply.

When the activation flag Flg_y is generated (turned to 1) based on the added value corresponding to the detection area where the swinging hand is extracted for a predetermined period, a second y-axis timing pulse drawn by a solid line is generated. The summed value of the output signal of the detector. The second y-axis timing pulse is a pulse having a width corresponding to a predetermined width in the vertical direction of the effective video period, and is supplied to all the x-axis detectors. Based on the second y-axis timing pulse, each of the x-axis detectors 301 to 316 outputs a detection signal of only the minimum detection area necessary to detect a hand placed in the detection area.

A method of generating the second y-axis timing pulse will be described with reference to FIG. 31 . The first y-axis timing pulse supplied to each of the y-axis detectors 301 to 316 is the first y-axis timing pulse. The first y-axis timing pulse is a pulse valid for the entire width in the y-axis direction of each detection area corresponding to each of the x-axis detectors 301 to 316 .

When the hand is extracted from the detection area of y-coordinate 2 by the movement of the hand shown in FIG. The second detection data of the corresponding 23rd detector 323 continuously takes the maximum value (refer to FIG. 28 ). When the value of the second detection data accumulated by the 23rd detector 323 exceeds the threshold th1y, an activation flag Flg_y is generated (becomes 1). The control information determiner 20 confirms that the activation flag Flg_y is generated, and sets 1 to the y-axis control data of the detection area at the y-coordinate 2 .

In this embodiment, considering that the size of the hand on the screen varies slightly with the distance between the television receiver 1 (camera 2) and the user 3, at least the detection area corresponding to the detector that generated the activation flag Flg_y will be included The y-axis control data of the detection area adjacent to the detection area adjacent to the detection area is set to 1. For example, set the y-axis control data of the detection area of y-coordinates 1 and 3 to 1. In addition, the y-axis control data of other detection areas are set to 0.

The control information determiner 20 supplies the y-axis control data as described above to the timing pulse generator 12; the y-axis timing pulse activation controller 12y within the timing pulse generator 12 generates a second y-axis timing pulse based on the input y-axis control data , are supplied to all the x-axis detectors 301-316. Therefore, in the state shown in FIG. 30 , a second y-axis timing pulse having a width corresponding to the width of the detection region whose y-coordinates are 1 to 3 is generated. That is, the timing generator 12 generates the second y-axis timing pulse narrowing the width of the first y-axis timing pulse. Each of the x-axis detectors 301 to 316 to which the second y-axis timing pulse is supplied outputs a detection signal only from an interval of y-coordinates 1 to 3 of each corresponding detection area. As a result, noise components occurring at coordinates (x, y)=(4, -1), (7, -1) shown in FIG. 30 cannot be detected.

When the second y-axis timing pulse is generated, the control information determiner 20 performs subsequent controls based on outputs from the x-axis detectors 301 to 316 . The detection signals detected from the respective y-axis detectors 317 to 325 are not referred to. In addition, the timing pulse may not be supplied to the timing gate 52 of each y-axis detector 317-325, and the output of a detection signal may be stopped.

32 shows the second y-axis timing pulses supplied to the first detectors 301 to 16 detectors 316 of the x-axis detectors, and the timing pulses (x-axis direction). Each of the x-axis detectors 301 to 316 only needs to output signals detected from three intervals that overlap with the corresponding detection area and the detection area corresponding to the y-coordinates 1 to 3 based on the second y-axis timing pulse. By doing so, it is possible not to detect a section in the detection area where the hand cannot be drawn and does not need to be detected.

In addition, in this embodiment, the method of controlling the pulse width in units of detection areas as shown in Fig. 31 and Fig. 32 is adopted, but the pulse width can also be flexibly controlled by using a method such as specifying the start point of the pulse and the pulse width. circuit method.

The table shown in FIG. 33 is substantially the same as the table shown in FIG. 28 , but shows the second y-axis generated after the activation flag Flg_y of the 23rd detector 323 shown in FIG. 29(C) becomes 1. The timing pulse is the second detection data from each of the detectors 301 to 325 obtained by restricting detection from a section or detection area that does not require detection.

In FIG. 29(C), the second detection data of frame number 10 and later exceeding the threshold th1y belong to this, and x(4), x(7), and y of frame number 11 existing as noise components in the table of FIG. 28 (-1) becomes 0. This is because, since the second y-axis timing pulses are supplied to the respective timing gates 52 of the correspondingly arranged 13th detector 13 and the 16th detector 316, the coordinates (x, y)=(4,-1) , (7, -1) intervals are not detected. By removing the noise components, the confusion of the values of the center of gravity XG and YG is eliminated, and the recognition rate of the first motion detector 20-1 to the fifth motion detector 20-5 in the subsequent stage of each x-axis detector 301-316 is improved. .

The first motion detector 20-1 to the fifth motion detector 20-5 in the control information determiner 20 receive and process the data shown in FIG. 33 . Returning to Fig. 29, the processing for detecting how the hand moves will be described.

FIG. 29(A) shows the variation of the y-coordinate YG of the center of gravity, and FIG. 29(B) shows the variation of the x-coordinate XG of the center of gravity, respectively showing waveforms without noise. The activation flag Flg_y becomes 1 when the value of the accumulated output signal of the y-axis detector (the 23rd detector 323 ) shown in FIG. 29(C) becomes equal to or greater than the threshold th1y. The intervals in the x-axis direction other than the plurality of intervals formed by the intersection of the detection area in the y-axis direction and the detection area in the x-axis direction are passed by the second y-axis timing pulse supplied to the x-axis detectors 301-316. The detection area in the y-axis direction in the vicinity includes at least a detection area adjacent to the detection area corresponding to the detector that generated the active flag Flg_y. That is, it is not used for hand detection. Therefore, it is not affected by noise.

If the waveform in FIG. 29(C) continues to be above the threshold value th1y, the second y-axis timing pulse is continuously supplied to each of the x-axis detectors 301 to 316. Therefore, unnecessary intervals continue to be invalid periods, and continue to be inactive. Effects affected by noise. When the waveform in FIG. 29(C) becomes equal to or less than the threshold value th1y, the accumulated value is reset. However, the value used as the reset reference is not limited to the threshold th1y.

Next, processing for suppressing the DC offset of the waveform in FIG. 29(B) is performed so that the average value of the waveform becomes substantially zero. The high-pass filter shown in FIG. 20 is used for this processing.

The waveform shown in FIG. 29(B) passes through the high-pass filter shown in FIG. 20. As a result, the average value of the waveform shown in FIG. 29(D) becomes approximately zero. In this way, the position information on the x-axis of the hand swing is excluded, and a waveform suitable for analyzing the content of the hand movement is obtained. In addition, the center of gravity XGH shown on the vertical axis of FIG. 29(D) is a value obtained by high-pass filtering the center of gravity XG shown on the vertical axis of FIG. 29(B).

In order to analyze the motion content of the hand swinging laterally, similarly to the vertical swinging motion of the hand, representative detection signal waveforms of predetermined predetermined motions (lateral swinging motions) determined in advance and signals generated by actual motions output from the detectors 301 to 325 are acquired. Cross-correlation between signal waveforms is detected to evaluate consistency.

In this embodiment, the waveform shown in FIG. 34(G) is used as the reference signal waveform of the wobble operation (a representative detection signal waveform for the predetermined operation), and the cross-correlation digital filter shown in FIG. 34(F) Among the tap coefficient values k0 to k40, values corresponding to the reference signal waveform are used. In addition, although the actual detection signal waveform input to the cross-correlation digital filter kn is shown in FIG. 34(D), this is the same waveform as the signal waveform in FIG. 29(D). The cross-correlation digital filter multiplies the tap coefficient by the second detection signal based on the signal that detected the actual motion, and the first motion detector 20-1 to the fifth motion detector 20-5 based on the output from the cross-correlation digital filter Signal waveform, it is detected that the motion performed by the user 3 is a lateral swing motion. The output signal wh(n) of the cross-correlation digital filter kn is obtained by the following Mathematical Expression 4.

[mathematical formula 4]

$<span\; class="notranslate">\; wh</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

N is the number of taps of the digital filter, and here is 41 taps (0 to 40). x(n+i) is the center of gravity XGH subjected to the filter processing shown on the vertical axis of FIG. 34(D). The function of the cross-correlation digital filter kn is realized by operating only when the activation flag Flg_y becomes 1.

In addition, in this embodiment, a cross-correlation digital filter having tap coefficients corresponding to the vertical wobble operation and a cross-correlation digital filter having tap coefficients corresponding to the horizontal wobble operation are used, however, it may be determined in the control information The device 20 and the like store the tap coefficients corresponding to the vertical wobble operation and the tap coefficients corresponding to the horizontal wobble operation, and switch to one cross-correlation digital filter for supply according to the operation. However, in the case where the vertical wobble operation and the horizontal wobble operation are the same operation, the same tap coefficient may be used.

Next, the speed of hand movement and the number of frames will be described. Regarding the relationship between the movement speed of the hand and the number of frames, the way the hand swings is the same whether it is vertical (up and down) or horizontal (left and right).

In this embodiment, 1 second is defined as 60 frames, and for the purpose of simplification of the description and the drawings, the motion of shaking the hand up and down or left and right four times is defined as 32 frames. The tap coefficient of the correlation calculation also becomes smaller.

However, when 32 frames are converted into time, it is about 0.5 seconds, which is too fast for a real human being. It is conceivable that the actual hand movement is slower. For example, if it takes 2 seconds to swing the hand 4 times, 120 frames are required. In order to detect this motion, the number of taps may be increased in the correlation calculation, and the number of taps may be appropriately adjusted according to the time required for the motion.

The cross-correlation digital filter output signal wh(n) related to the lateral swing of the hand has a waveform as shown in FIG. 35(E), and the consistency of the cross-correlation is increased, and the amplitude is increased. In addition, FIG. 35(D) is the same waveform as FIG. 29(D) and FIG. 34(D), and is shown as a comparison with FIG. 35(E). The absolute value of the output signal wh(n) is obtained and integrated, and when the value reaches the threshold value th2h or more, it is determined that the cross-correlation with the reference signal waveform is sufficient, and it is recognized that a predetermined operation has been performed. The first motion detector 20 - 1 to the fifth motion detector 20 - 5 are motion detectors that detect whether or not the motion of the user 3 is a predetermined motion based on the detection signal output from the detection unit 19 .

While recognizing this motion, it is confirmed that the activation flag Flg_y indicating that it is a lateral swing motion and that it functions as a protective window is “1”, and it is determined to be a lateral swing operation of the hand, and an event corresponding to the state of the television receiver 1 is performed. This event is performed based on a signal output by the control information generator 20-10 after logically judging that any one of the plurality of motion detectors 20-1 to 20-5 shown in FIG. 15 is confirmed.

FIG. 36 is a flowchart showing the processing steps of the method of detecting the above-mentioned vertical swing and lateral swing of the hand. The processing in each step shown in the flow chart of FIG. 36 has been described in detail, so here is an explanation of what functions each step implements as a whole, and it is also explained that the vertical swing and horizontal swing of the hand are received by the TV as operation information. Machine 1 recognizes and executes the process of controlling (event) content.

The flowchart shown in FIG. 36 is divided into two types: a vertical swing processing system and a horizontal swing processing system, according to the hand swinging motion performed by the user 3 . When the X-axis of the longitudinal processing system is activated, 16 pieces of second detection data x(-8) to x(7) based on the first detection data output from the respective x-axis detectors 301 to 316 are input. First, in step A501, the respective second detection data x(-8) to x(7) based on the outputs of the respective x-axis detectors 301 to 316 are accumulated for each frame.

Next, the process proceeds to step A502, and it is judged whether or not each accumulated value msx(i) (i=-8 to +7) is equal to or greater than the threshold value th1x. When the answer in step A502 is "No", that is, when all added values msx(i) are smaller than the threshold th1x, return to step A501 to perform accumulation. If the answer in step A502 is "Yes", that is, if any of the added values msx(i) is equal to or greater than the threshold th1x, the process proceeds to the next step A503.

The added value msx(i) from a certain x-axis detector is above the threshold th1x, which means that the hand is swinging vertically. Therefore, in step A503, the activation flag Flg_x is set from 0 to 1, and sent to each x-axis detector 301 ~316 supplies the second x-axis timing pulse. In this way, the outputs of the x-axis detectors 301 to 316 are controlled, and processing (masking) not to extract the object (hand) is performed in an unnecessary detection area or section, thereby improving noise resistance.

The same is true for the yaw processing system. When the Y-axis is activated, nine pieces of second detection data y(-4)-y(4) based on the outputs of the y-axis detectors 317-325 are input, and are compared with each other in steps B501-B503. The pitching process is exactly the same process as steps A501 to A503.

Then, in step B502, when the accumulated value msy(j) (j=-4～+4) in a certain y-axis detector reaches the threshold value th1y or more, the activation flag Flg_y changes from 0 to 1, and it is recognized that the hand movement is swing sideways.

In the present embodiment, when one of the active flags Flg_x and Flg_y becomes 1, the other active flag is suppressed. In step A504 or B504, the activation flag is determined. For example, in the pitching processing system, when the activation flag Flg_x becomes 1, the process proceeds to step A504, and it is determined whether the activation flag Flg_y of the lateral rolling processing system is 0 or not.

If the answer in step A504 is "Yes", that is, if the activation flag Flg_y is 0, it is determined that the subsequent processing is a vertical swing processing system, and the process proceeds to step A505. On the other hand, if the answer of step A504 is "No", that is, the horizontal swing processing system is activated, and the activation flag Flg_y is 1, then proceed to step A509, and the accumulated value msx(i ) and the activation flag Flg_x are reset to 0, and return to step A501.

In addition, in the panning processing system, when the active flag Flg_y becomes 1 in step B503, the process proceeds to step B504, and it is determined whether the active flag Flg_x in the vertical rolling processing system is 0 or not.

If the answer in step B504 is "Yes", that is, if the activation flag Flg_x is 0, it is determined that the subsequent processing is the yaw processing system, and the process proceeds to step B505. On the other hand, if the answer of step B504 is "No", that is, the longitudinal swing processing system is activated, and the activation flag Flg_x is 1, then proceed to step B509, and the accumulated value msy(j ) and the activation flag Flg_y are reset to 0 respectively, and return to step B501.

When the answer of step A504 is "yes", or the answer of step B504 is "yes", go to step A505 or B505 respectively to calculate the center of gravity of the y-axis or the center of gravity of the x-axis. The y-axis gravity center calculation or the x-axis gravity center calculation calculates the item YG of the table shown in FIG. 24 or the item XG of the table shown in FIG. 33 using the above-mentioned formula 1 or formula 2. In the cross-correlation calculation in step A506 or B506, cross-correlation digital filtering is performed on the obtained value of the center of gravity (XG, YG), and the cross-correlation digital filter output signal wv(n) or wh(n) is calculated.

In step A507 or B507, the cross-correlation digital filter output signals wv(n) or wh(n) are converted into absolute values and then accumulated to calculate the accumulated swv of wv(n) or the accumulated swh of wh(n).

Next, in step A508 or B508, it is judged whether the value of the accumulated swv is greater than the threshold th2v, or whether the value of swh is greater than the threshold th2h. When the answer of step A508 or B508 is "yes", a vertical swing event or a horizontal swing event is initiated. In addition, steps A504 to A508 and steps B504 to B508 are described here at the same time. However, as described above, the pitch processing system and the lateral motion processing system are not processed at the same time, but only one of the processes is performed.

In addition, in consideration of easy-to-understand description, the processing before the cross-correlation calculation processing of step A506 or B506 is separated into two systems in FIG. Oscillation is also lateral oscillation, so the treatment can be made into a system. In addition, when the answer in step A504 or B504 and step A508 or B508 is "No", proceed to step A509 or B509, and the accumulated value msx (i) and Flg_x, or the accumulated value msy (j) and Flg_y are respectively Reset to 0, and return to the beginning moment.

In this way, in this embodiment, the operation of shaking the hand vertically and the operation of shaking the hand horizontally are processed at the same time, and are recognized separately. As an example of this application, if it is the action of "Come here, come here" as shown in FIG. 3, the event (control) corresponding to it is, for example, turning on the power or starting a menu screen. In addition, if it is a "bye" movement of swinging the hand sideways, it can be applied to turning off the power.

In addition, only one of the vertical swing motion and the horizontal swing motion may be used as a predetermined motion for controlling the electronic device, and in this case, step A504 or B504 may be omitted.

In the first embodiment described above, as shown in FIGS. 5 and 6 , the detection areas provided on the screen are divided into 16 in the horizontal direction and 9 in the vertical direction. There are a total of 25 detection areas. The number of detectors corresponding to each detection area is 25 from the first detector 301 to the twenty-fifth detector 325, and the first embodiment has the advantage that it can be realized with a smaller hardware scale.

On the other hand, if a more advanced and complicated recognition operation is considered, a second embodiment using the detection area described below can be considered. The case where the algorithm explained by the flowchart of FIG. 36 also functions in the second embodiment will be described in detail. In addition, only the part which differs from 1st Embodiment is demonstrated about 2nd Embodiment.

37 shows a second embodiment in which a total of 144 (16×9) detection areas are provided. The 144 detection areas are divided into 16 in the horizontal direction and 9 in the vertical direction. And the area divided in the horizontal direction intersects the area divided in the vertical direction. Therefore, 144 detectors constituting the detection unit 19 are also required, and 144 pieces of data are input to the control information determiner 20 . The first detector 301 corresponds to the detection area located on the coordinates (x, y)=(-8, 4) in FIG. 37, and outputs the first detection data detected from the detection area.

In the description of the second embodiment, each detector is assigned to each detection area for the purpose of obtaining output signals extracted from each detection area for each screen (each vertical period), and each detection signal is input to the control information determiner 20. Area data is processed with software. In addition, each detector can be realized with less than the amount of data necessary for the hardware configuration by providing a buffer memory.

FIG. 38 shows a screen in which an image of a vertically swinging hand captured by the camera 2 is superimposed and drawn on 144 detection areas. The detection area where the frame difference occurs due to the movement of the hand is hatched (the area of the hand is also hatched). In the first embodiment, the hatched detection area is converted into data by the histogram detector 61 of the object feature data detection unit 53 shown in FIG. .

The same structure can also be applied to the second embodiment, but the number of data obtained from each detector is 144, so the data is simplified in consideration of the increase in hardware scale and the traffic volume of the bus. In addition, for comparison and description, the position of the hand movement in FIG. 38 is the same position as that of the first embodiment shown in FIG. 17 .

FIG. 39 is a block diagram of the detection unit 19 and the control information determiner 200 of the second embodiment. The first detector 301 to the 144th detector 444 constituting the detection unit 19 process the data to be processed and send it to the sixth motion detector 20 - 6 of the control information judging unit 200 . The output of the first object extractor 51, as shown in FIG. 8, is a composite signal of a specific color filter 71, a grayscale limiter 72, and a motion detection filter 75, and is an output signal in which an object is extracted from the output image of the camera 2. .

Composite methods can consider a wide variety of logical operations, but here we consider logical products. The output of the object selector 74 is that only the detection area of the hatched portion in FIG. 38 has gray scale, and the other detection areas do not have objects extracted, and the gray scale is set to 0 (black level). The black level of the camera 2 at this time is assumed to be equal to or higher than 0 level.

The object feature data detection unit 530 has a block counter 66 and a block quantizer 67 . A histogram detector 61, an APL detector 62, and the like may also be provided as needed.

The block counter 66 and the block quantizer 67 perform 1-bit conversion of the information of the detection area indicated by the hatching in the screen of FIG. 38 from which the output signal of the first object extractor 51 is obtained, and outputs it. The block counter 66 is a device that counts detection areas other than the mask level in the entire detection area. In the detection area counted by the block counter 66, the output signal output from the first object extractor 51 is compared with the threshold value set in the block quantizer 67, and 1 is output when the block quantizer 67 is above the threshold value, and the following output 0.

For example, when the output signal output from the first object extractor 51 is obtained in 1/2 of the entire detection area, a threshold value is set, and if the block quantizer 67 is input from the hatched part in FIG. 38 according to the threshold value The output signal of the detection area becomes the output signal of the detection area hatched as shown in FIG. 40 . That is, two detection areas whose coordinates (x, y) of the detection area are (5, 3) and (5, 2) output 1, and other detection areas output 0.

By setting the threshold in this way, the output from the detection unit 19 generated by the block counter 66 and the block quantizer 67 can be realized at a minimum of 144 bits.

144 pieces of data for one screen (per vertical cycle) are stored as variables in the control information judging section 20, and are processed according to the motion recognition algorithm. This content is shown in FIG. 41 . Items x(-8) to x(7) respectively become the sum of outputs of all detectors corresponding to the entire detection area in the longitudinal (y-axis) direction of each x-axis. For example, the item x(0) is the summed value of the second detection data based on the first detection data output from each detector whose corresponding detection area coordinates (x, y) are (0, -4), (0,-3), (0,-2), (0,-1), (0,0), (0,1), (0,2), (0,3), (0,4) detection area. The detection area is divided into 9 in the y-axis direction, so the maximum value is 9.

Items y(−4) to y(4) are also the sum of outputs of all detectors corresponding to all detection areas in the horizontal direction (x axis) of each y axis, and the maximum value is 16. Thus, as a result of the hand movement shown in FIG. 38, the shift of the center of gravity is the same as that shown in FIG. 18, and can be processed by the same algorithm to recognize the movement.

Comparing the table of FIG. 41 and the table of FIG. 42, in the column of the first frame n=0, each item x(6)=x(4)=12, x(5)=120, y(3) of FIG. 18 =y(2)=72 corresponds to each item x(6)=x(4)=0, x(5)=2, y(3)=y(2)=1 in FIG. 41 .

FIG. 41 shows values corrected to be binary after quantization, and the scale is different from FIG. 18 . However, the center of gravity representing the position is the same. Therefore, the sixth motion detector 20-6 of the second embodiment can recognize a hand motion using the same algorithm as that of the first motion detector 20-1 to the fifth motion detector 20-5 of the first embodiment. The algorithm of the sixth motion detector 20-6 is the calculation of the center of gravity of Mathematical Expression 1 and Mathematical Expression 2, the calculation of the correlation digital filter of Mathematical Expression 3, and the timing pulse is not output to the detector corresponding to the unnecessary detection area. Figure 36 is a flow chart showing the algorithm for closing the timing gate 52 and the like. The sixth motion detector 20 - 6 is a motion detector that detects whether or not the motion of the user 3 is a predetermined motion based on the detection signal output from the detection unit 19 .

Here, the processing of closing the timing gate 52 of each detector corresponding to each detection region in the second embodiment is referred to as masking processing, which will be described later.

In the second embodiment, the detection areas corresponding to the detectors 301 to 444 correspond to the intervals described in the first embodiment, so the method of closing the timing gate 52 is the same as that of the first embodiment, but unnecessary There are different methods for detecting zone invalidation.

FIG. 42 shows the motion of repeatedly swinging the hand up and down in the same manner as in FIG. 38 . In order to recognize this motion as an operation, the first object extractor 51 performs the function of detecting the motion of the hand, but unconscious motion is mixed. For example, in the detection area shown in FIG. 42 , noise indicated by black circles occurs at (x, y)=(1, -2), (1, -3).

In the table of FIG. 41 , in the frame n=11, the items x(1)=2, y(-2)=11, and y(-3)=1 are entered. This interferes with the center of gravity of the x-axis and y-axis, which gets in the way when detecting hand movements. This affects the value of the center of gravity, so it becomes a problem in the present invention that detects the movement of the hand using the change of the center of gravity.

This noise is suppressed and removed by masking the detection area other than the detection area for detecting the vertical movement of the hand.

The masking process is the same as the first embodiment. The value of each item x(-8) to x(7) on the x-axis is accumulated for a predetermined period, and when it exceeds the threshold th1x as shown in FIG. 19(C), it is activated. The flag Flg_x can be set to 1. Therefore, in the second embodiment, when the output sum of all detectors corresponding to all the detection areas in the vertical direction (y-axis) in each x-coordinate, or the total detection area in the horizontal direction (x-axis) in each y-coordinate When the output sum of all the corresponding detectors exceeds the threshold th1x or th1y respectively, it is only necessary to generate the activation flag Flg_x or Flg_y. In addition, a limit can also be set if the accumulated value exceeds the specified value.

In FIG. 19C , it is detected that the added value of the output signals of the detectors corresponding to the detection regions having the x-axis coordinate 5 exceeds the threshold th1x after a predetermined period (frame 10). Thus, a swinging hand is detected in at least a part of each detection area having an x-coordinate 5 .

After the signal output from the detector exceeds the threshold th1x, set the activation flag Flg_x to 1 within a specified period, and evaluate the variation of the center of gravity YG in the vertical direction (y axis) shown in FIG. 19(A) by using a cross-correlation digital filter. Correlation, which recognizes hand movements as manipulations.

In the second embodiment, the screen of the image output from the camera 2 is divided horizontally and vertically to set each detection area, and the first detection data of each detection area is supplied to the control information determiner 20 as variables arranged in two dimensions Processing, therefore, masking processing can be achieved by manipulating this variable to 0. Of course, the timing pulse generator 12 can also be used to control the timing pulse input to the timing gate 52 .

In the present embodiment, since the masking process is performed on the tenth frame and onwards, the noise component (frame number 11) shown in the table of FIG. 41 can be suppressed. As described above, masking has the effect of suppressing movements other than hands and finding only predetermined movements.

The hatched portion in FIG. 42 is the detection area subjected to the masking process as described above. Based on the table in Fig. 41, it is enough to perform masking on the detectors corresponding to the detection areas other than the detection area with x-axis coordinate 5, but in this embodiment, considering the unstable hand situation, the control is paired with the x-axis coordinate All the detection areas of 5 and the detectors corresponding to the detection areas whose x-axis coordinates are 5±1 do not perform masking processing, and output detection signals.

That is, a timing pulse is supplied from the timing pulse generator 12 to each detector corresponding to each detection area of x-coordinate 5 where the active flag Flg_x becomes 1, and each detector corresponding to each detection area of x-coordinate 4 and 6, The respective detection areas of x-coordinates 4 and 6 are adjacent to the respective detection areas of x-coordinate 5 .

In addition, according to the table in FIG. 41 , the detectors corresponding to the detection areas whose x-axis coordinates are 4 to 6 and y-axis coordinates are -4, -3, -2, and 4 are not involved in the up and down movement of the hand. A timing pulse is supplied to perform muting processing. The masked detection area is indicated by a mark X in FIG. 42 . Thus, the influence of noise can be further suppressed.

This masking process is performed by evaluating the center of gravity YG shown in FIG. 19(A) before the time when the activation flag flg_x is set to 1 in FIG. 19(C) . The center of gravity YG for a predetermined period is recorded in a memory (not shown) in the control information judging unit 20 , and when the activation flag Flg_x is generated, the center of gravity YG recorded before that is referred to. In this embodiment, the range indicated by the arrow 1 in FIG. 19(A) is referred to. In the detection area where the y-axis coordinates are -4, -3, -2, and 4, it can be judged that no hand is placed, and the hand movement is outside the range of the detection area where the y-axis coordinates are -4, -3, -2, and 4. occurs, the masking process is performed as described above.

That is, when the hand is moved to perform a predetermined movement, the detection area in which the placed hand is extracted is determined, and becomes the passing area through which the detection signal passes. In other detection areas, since the timing pulse is not supplied to the timing gate 52, the detected signal does not pass through. Furthermore, if the summed value of the output signals of each detector exceeds the threshold value th1x, the second detection data for a predetermined period earlier than the time when the threshold value th1x is exceeded is referred to and corresponding to the detection area other than the detection area where the hand is placed. The detector is masked so that no detection signal is output, thereby suppressing noise.

In the second embodiment, a corresponding detector is provided for each detection area provided on the screen of the image output from the camera 2, and the movement of the hand is detected using the block-shaped detection area. Since masking processing on a two-dimensional plane can be performed, the first embodiment can focus on the detection area where the moving hand is extracted, thereby improving the anti-noise performance. In addition, since masking can be performed by software, parallel processing with data not subjected to masking can be performed, and there is also an advantage of increasing the degree of freedom in processing.

In addition, the algorithm for recognizing hand movement based on the second detection data shown in FIG. 36 performs its function in the same way regardless of the embodiment of the detection area, and can control the television receiver after confirming the recognition operation.

FIG. 43 is a diagram showing a second object extractor 510 which is another embodiment of the first object extractor 51 shown in FIG. 8 . The second object extractor 510 uses a multiplexer 73 to combine the signals output from the specific color filter 71 and the gray scale limiter 72, and a motion detection filter 75 is arranged in series in the subsequent stage, and the object selector 74 detects the signal output from the camera. 2 signals with gating processing applied.

In addition, in the second embodiment, the block counter 66 of the object feature data detection unit 530 counts only in the detection area corresponding to the detector to which the timing pulse is supplied, so the output of the motion detection filter 75 in FIG. By directly adding to the block count of the object feature data detection unit 530 in FIG. 39, the output of the block quantizer 67 can also obtain the hand movement information of the detection area unit.

FIG. 44 is a diagram for explaining an example to which an embodiment of the present invention is applied. 44(A) shows the menu image (image for operation) drawn by the graphics generator 16. The image is divided into five areas (1-1)-(1-5). perform specified actions in each area. FIG. 44(B) shows a mirror-converted image of the user 3 captured by the camera 2 .

FIG. 44(C) shows how the images of FIGS. 44(A) and (B) are mixed on the display device 23, and the positional relationship between the menu image and the user 3 is known. In the mechanism of this second embodiment, the display device 23 and the graphics generator 16 shown in FIG. 2 are essential functional blocks.

FIG. 45 shows a state in which the user 3 is operating the television receiver 1 looking at the screen in which the menu screen and the mirror image of the user 3 shown in FIG. 44(C) are mixed. FIG. 45(A) shows a state where the user 3 moves his hand to select an image representing desired menu contents, that is, a desired operation button, among a plurality of menu images. In FIG. 45(A) , the user 3 has selected, for example, the operation button of "Movie".

As described in the first embodiment, when the hand is moved vertically, it is known which of the x-axis detectors shows the maximum value, and the activation flag Flg_x becomes "1". Therefore, if the graphic generator 16 that has created the menu image is associated with the detector corresponding to the detection area of each coordinate, the control corresponding to the operation button selected by the user 3 can be activated.

According to the television receiver 1 shown in each embodiment of the present invention described above, the following effects can be achieved. When the power of the television receiver 1 is turned on, if the hand movement is within the imaging range of the video camera 2, the on/off of the power and the control of the graphical menu display can be performed. At this time, the movement of swinging the hand vertically or horizontally is not difficult for people. In addition, the vertical movement means "come, come", and the horizontal movement means "bye", which are meaningful actions for human beings, and the control of the television receiver 1 can be realized by using the form of giving meaning to these actions. Easy to understand and easy to use operation.

In addition, motion detection can be performed regardless of where the user 3 is located within the imaging range of the camera 2 , and correct detection with few false recognitions can be performed further using the control of the activation flag. Furthermore, it can also be applied to an operation method of selecting a desired menu on a screen in which the menu image generated by the graphics generator 16 and the user's own image captured by the camera 2 are mixed, and various applications can be realized with the same circuit and software processing.

In each of the embodiments of the present invention described above, the television receiver 1 was shown as an example of an electronic device, but it is not limited to this, and the video camera 2 may be mounted on other electronic devices for application. In addition, the method for the user to select and operate a menu image indicating a desired control content from a menu screen in which an image of the camera 2 is mixed with a graphic menu image is any electronic device provided with a display (display device). can be applied. The present invention provides a useful device on the basis of constructing a structure in which electronic equipment can be operated without a remote controller.

Claims (6)

1. an electronic equipment is characterized in that, comprising:

Display unit;

Video camera is photographed to the operator of the front that is positioned at above-mentioned display unit;

Test section, have and carry out N in the horizontal direction with picture and cut apart, carry out a plurality of surveyed areas a plurality of detectors of corresponding settings respectively after M is cut apart, utilize first detection signal of the action that above-mentioned a plurality of detector generation carries out based on the aforesaid operations person who is photographed by above-mentioned video camera in vertical direction with the image of above-mentioned video camera output;

Timing pulse generator, supply are used to make the commutator pulse of the respectively corresponding above-mentioned a plurality of surveyed area work of above-mentioned a plurality of detector;

Maker generates second detection signal based on above-mentioned first detection signal;

Sign maker when the additive value of above-mentioned each second detection signal that added up in specified time limit surpasses predetermined threshold value, generates sign; And

Controller, carry out following control: make based on second detection signal of first detection signal of detector output in above-mentioned a plurality of detectors, corresponding with a part of surveyed area in above-mentioned a plurality of surveyed areas effective, and make based on second detection signal from first detection signal of other detector output invalid;

The specified time limit of above-mentioned timing pulse generator after above-mentioned sign maker has generated above-mentioned sign, generation in above-mentioned a plurality of detectors the specific detector of above-mentioned sign and the detector corresponding with near the surveyed area of specific surveyed area supply with above-mentioned commutator pulse selectively, wherein, above-mentioned specific surveyed area is corresponding with above-mentioned specific detector, and near the surveyed area of above-mentioned specific surveyed area comprises the surveyed area with above-mentioned specific surveyed area adjacency at least;

Wherein, M and N are the integers more than 2.

2. electronic equipment as claimed in claim 1 is characterized in that,

Have N first detector and M second detector, above-mentioned N the first detector correspondence will have been carried out the surveyed area that N is cut apart in the horizontal direction from the picture of the image of above-mentioned video camera output, and above-mentioned M second detector correspondence will have been carried out the surveyed area that M is cut apart in vertical direction from the picture of the image of above-mentioned video camera output;

The specified time limit of above-mentioned timing pulse generator after above-mentioned sign maker has generated above-mentioned sign, the width of the commutator pulse of supplying with to above-mentioned N first detector or above-mentioned M second detector based on the control of above-mentioned controller is narrowed down according to the action that the aforesaid operations person carries out.

3. electronic equipment as claimed in claim 1 is characterized in that,

Have and carry out N in the horizontal direction with picture and cut apart, carry out that M is cut apart and N * M the corresponding detector of N * M surveyed area that be provided with in vertical direction with the image of above-mentioned video camera output;

Above-mentioned controller carries out following control: the specified time limit after above-mentioned sign maker has generated above-mentioned sign, made based on the generation in above-mentioned N * M detector second detection signal of first detection signal of specific detector output of above-mentioned sign and effective based on second detection signal from first detection signal of the detector output corresponding with near the surveyed area in above-mentioned particular detection zone, it is invalid to make based on second detection signal from first detection signal of other detector output.

4. as each described electronic equipment in the claim 1 to 3, it is characterized in that,

Above-mentioned electronic equipment comprises:

The mirror transformation device carries out the mirror transformation by the image of above-mentioned video camera photography;

Image composer is used in operation, generates at least one operation image; And

Blender mixes from the mirror transformation picture signal of above-mentioned image converter output with from the operation picture signal of aforesaid operations with image composer output;

Under the state that will be presented at by the image that above-mentioned blender mixes on the above-mentioned display unit, above-mentioned test section produce be presented at above-mentioned display unit on the aforesaid operations person operate corresponding above-mentioned first detection signal of compulsory exercise of aforesaid operations usefulness image.

5. as each described electronic equipment in the claim 1 to 4, it is characterized in that,

Above-mentioned test section has:

Digital filter, tap coefficient and above-mentioned second detection signal corresponding to first reference signal waveform that generates when detecting first action carry out multiplying, and above-mentioned first action is moved the object by above-mentioned video camera photography in the vertical; And

Motion detector, according to the signal waveform from above-mentioned digital filter output, whether the action that detection aforesaid operations person carries out is above-mentioned first action.

6. as each described electronic equipment in the claim 1 to 4, it is characterized in that,

Above-mentioned test section has:

Digital filter, tap coefficient and above-mentioned second detection signal corresponding to second reference signal waveform that generates when detecting second action carry out multiplying, and above-mentioned second action is moved the object by above-mentioned video camera photography in the horizontal; And

Motion detector, according to the signal waveform from above-mentioned digital filter output, whether the action that detection aforesaid operations person carries out is above-mentioned second action.

Images