JP2004178469A - Remote control system - Google Patents

Abstract

A laser pointer indicates a control target or a menu item projected by a projector to indicate a user's intention. However, a spot drawn by a laser beam is small and difficult to detect, and an image sensor with high resolution is required. It is used in a dimly lit environment because the spot light is buried in ambient light or changes its color by mixing with the color of the object. The dimness lowers the recognition rate of the object.
An infrared remote controller and a CMOS image sensor having several hundred thousand pixels are used. The microprocessor captures an image only with the light receiving element equipped with the red filter of the CMOS image sensor, and recognizes a remote controller which is a non-natural light source to demodulate data. The user swings the infrared remote control, and the microprocessor determines the direction and acceleration of the remote control to identify the user's gesture. The menu item is switched, the selection is confirmed, the command is sent to the object, and the remote operation is performed.
[Selection diagram] Fig. 1

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to remote control of home appliances using an infrared remote controller and a CMOS image sensor.
[0002]
[Prior art]
FIG. 11 shows a conventional technique. In the figure, 2 is an image sensor, 3 is a microprocessor, 7 is a laser pointer, 70 is a spot drawn by a laser beam on a wall, 8 is a projector, and 81, 82, and 83 are menu items on a remote control screen displayed by the projector.
[0003]
In the related art, the home appliance pointed by the laser pointer 7 is photographed by an image sensor, and a spot drawn by the laser light of the laser pointer on the object is searched from the image, and the outline and the feature of the object are recognized again.
[0004]
When the target object is determined, the state and control system of the home electric appliance as the target object are acquired, and the projector 8 projects the remote control screen. The menu items 81, 82, and 83 are projected on the wall surface of the room as a remote control screen, and the laser pointer 7 moves the spot 70 drawn on the wall by the laser beam on the menu item 83, and the user selects the menu item 83 by standing still. showed that.
[0005]
After searching for the spot drawn by the laser beam in the same procedure as that for recognizing the target, the microprocessor 3 recognizes that the menu item 83 as the target is selected. The microprocessor 3 sends a selected command to the object according to the recognition result to realize remote operation.
[0006]
Further, in the “gesture recognition device” of Patent Document 1, a user's gesture is photographed by an image sensor, the user is cut out in image recognition, each part of the user's body is individually recognized, and the locus and position of each part are recognized. The gesture was identified from the relationship and the corresponding command was executed.
[0007]
[Patent Document 1]
JP-A-11-296673
[0008]
[Problems to be solved by the invention]
In the conventional technology, a light emitting device that emits a strong single light such as a laser pointer attempts to indicate a user's will by pointing to a control target or a menu item projected by a projector. However, it is necessary to use a CMOS image sensor having millions of pixels, which is small and difficult to detect because of the straightness of the laser beam and has high resolution.
[0009]
Further, the detection becomes difficult when the laser beam is buried in the light around the spot light drawn by the laser pointer or changes its color by mixing with the color of the object, so that it has to be used in a dim environment. There was also a problem that the recognition rate of the object was reduced due to the dim light. Further, it is necessary to lower the brightness of the menu items projected by the projector, and there are also restrictions on the colors projected.
[0010]
Since the laser pointer is used, burn-in occurs when the laser light directly enters the CMOS image sensor, and there is a risk that the CMOS image sensor may be damaged and the system may be damaged. In addition, the laser pointer is an unusual light projector in ordinary households, and there is a danger of damaging people and animals due to the laser light jumping directly into the eyes by mischief or carelessness.
In addition, the “gesture recognition device” disclosed in Patent Document 1 cannot be accurately recognized when a user is in a dark room, is extremely bright and white, or is buried in the background or surroundings due to the color of clothes. . Due to the nature of the system, the user always shoots the user and recognizes the gesture, so the casual and unconscious gesture of the user is recognized as a gesture, causing unintended command operation or malfunction.
[0011]
[Means for Solving the Problems]
In the present invention, an infrared remote controller which is rolled into a general household and has high recognition and receptivity and a CMOS image sensor having several hundred thousand pixels used in a digital camera or the like are used. The microprocessor performs photographing using a light receiving element to which a red filter is mounted among the light receiving elements of the CMOS image sensor, detects non-natural light, that is, a pulse-modulated non-natural light, that is, a signal from the image pickup and demodulates the data.
[0012]
In addition to demodulating data, the user swings the infrared remote controller and performs a predetermined gesture to select a menu item on the menu screen displayed by the microprocessor. The number of frames is earned by thinning out the number of light receiving elements read out by the CMOS image sensor, the trajectory of the infrared light output by the user's remote controller is reliably taken, and the frame is compared to determine the moving direction and acceleration of the remote controller. Thus, the gesture of the user is identified, and the menu item intended by the user is selected. When the selection of the menu item is confirmed, the command is sent to the object and the remote operation is performed.
[0013]
BEST MODE FOR CARRYING OUT THE INVENTION
A first embodiment of the present invention will be described with reference to FIGS.
[0014]
In FIG. 1, 1 is a remote controller, 2 is a CMOS image sensor, 3 is a microprocessor, 4 is a bus, 5 is a network, 6 is a home appliance connected to the network, and 10 is a spot of infrared light.
[0015]
In FIG. 1, a plurality of users control home electric appliances using a remote controller 1. The remote controller 1 pulse-modulates the control data corresponding to the pressed button by ASK and radiates it to the space as infrared light pulses. In the present invention, the CMOS image sensor 2 uses a CMOS image sensor in which about 300,000 pixels are arranged on a plane, and the CMOS image sensor 2 photographs the surroundings of home electric appliances. Each pixel of the CMOS image sensor 2 receives natural light in the range from visible light to infrared light, and pulse-modulated infrared light emitted by the remote controller, which is non-natural light. The CMOS image sensor 2 maps the pixels into a matrix and transmits the pixels to the microprocessor 3 via the all-column read bus 4 for each column. The microprocessor 3 rearranges the transmitted data on a plane and processes it as an image.
[0016]
When the remote controller 1 emits infrared light, a high-intensity infrared light spot 10 appears during imaging by the CMOS image sensor 2, and the microprocessor 3 demodulates control data from the infrared light spot 10. In the present invention, natural light received by pixels around the spot 10 of infrared light has little change in a long cycle, whereas received infrared light is pulse-modulated and has a high intensity. The processor 3 easily recognizes that infrared light is emitted from the remote controller.
[0017]
The microprocessor 3 includes a device management unit and a display device, an acceleration detection unit, and a gesture estimation / identification unit. The device management means searches the home electric appliances 6 on the network via the network 5 and acquires information such as a state and a control system. From this information, a command system necessary for remote control is constructed and displayed on a display device as a remote control screen. The user swings the infrared remote controller and performs a predetermined gesture to select a menu item on the menu screen displayed by the microprocessor. The acceleration detecting means extracts the acceleration from the locus of the infrared light output by the remote controller photographed by the CMOS image sensor, and the gesture estimating / identifying means identifies the user's gesture based on the acceleration information. The action corresponding to the identified gesture is compared with the displayed menu to select a menu item intended by the user. When the selection of the menu item is confirmed, the device management means sends the corresponding command to the home electric appliance 6 on the network and performs remote control.
[0018]
In the first embodiment of the present invention, the microprocessor 3 is a single network device. However, the microprocessor incorporated in, for example, an air conditioner, a lighting device, or a television in a place with a good view indoors is the same as the device management unit. Equivalent functions can be realized by mounting a display device, acceleration detection means, and gesture estimation / identification means.
[0019]
The operation mode of the CMOS image sensor 2 will be described with reference to FIG.
[0020]
In the drawing, reference numeral 41a denotes a visible image shooting mode setting command, 42a denotes imaging data in the visible image shooting mode, 41b denotes a carrier detection mode setting command, and 42b denotes imaging data in the carrier detection mode.
[0021]
The microprocessor 3 sets the operation mode of the CMOS image sensor 2 to the visible image shooting mode or the carrier detection mode by the visible image shooting mode setting command 41a or the carrier detection mode setting command 41b via the bus 4. When the setting is completed, the CMOS image sensor 2 sends data corresponding to the mode, imaging data 42a in the visible image shooting mode, and imaging data 42b in the carrier detection mode to the bus 4.
[0022]
In the visible image capturing mode, the drawing is set to progressive, and one frame of imaging data is composed of read data of all columns. At this time, the frame rate is 30 frames per second. When the mode is set to interlace, image data is read out every other column, and 60 fields are read out per second. When the carrier detection mode is set, the imaging data is read out every eight columns and every eight pixels, and the frame rate is 1920 fields per second. In the carrier detection mode of the present invention, a carrier is detected in a field unit and a command intended by a user is processed, so that a field is referred to as a frame for convenience.
[0023]
In any mode, the transfer rate of the image data on the bus 4 is 8.79 Mbps, and it is not necessary to change the setting of the interface of the bus 3 of the microprocessor 3.
[0024]
The reading order of the pixels of the CMOS image sensor 2 in the carrier detection mode will be described with reference to FIG.
[0025]
In the figure, reference numeral 20 denotes a section composed of 64 8 × 8 pixels, 201 denotes a pixel read first, 203 denotes a pixel read third, and 208 denotes a pixel read eighth.
[0026]
The section 20 shown in the figure is composed of 64 8 × 8 pixels, and the CMOS image sensor 2 of the present invention is divided into 80 sections for effective pixels in the column direction and 60 sections in the row direction, for a total of 4800 sections. Section 20 shows one of the 4800 sections.
[0027]
In the carrier detection mode, one specific pixel in the section 20 is read out, but between sections adjacent in the column direction, pixels are selected and read out every eight columns and every eight pixels. In the first frame read at the time of carrier detection of one frame, the pixel 201 is read, and thereafter, the pixel 202 is read in the second frame, the pixel 203 is read in the third frame, and the pixel 208 is read in the eighth frame in order. At the time of carrier detection, only the pixels with the red filter are read out, so the pixels to be read out are every other pixel, every other column, and the pixel with the green filter adjacent in the column direction and the green filter in the next column are mounted The read pixel and the pixel to which the blue filter is attached are skipped.
[0028]
By changing the pixels to be read for each frame, the resolution is improved without restricting the opening / closing time of the electronic shutter, that is, the exposure time.
[0029]
A method of receiving pulse-modulated infrared light by the CMOS image sensor 2 in the carrier detection mode will be described with reference to FIG.
In the drawing, 10 is one data cycle of infrared communication, 10a is a guide pulse, 10b and 10d are no output, 10c is symbol 0, and 10e is symbol 1.
[0030]
As shown in the figure, in the infrared communication, the data cycle 10 is 45 milliseconds. The next data cycle is the shortest and appears at 45 millisecond intervals between the guide pulse 10a and the next guide pulse 10a.
[0031]
In the infrared communication, the output continues for 2.4 milliseconds, so that the guide pulse 10a is not output. Thereafter, the output of 0.6 to 1.2 milliseconds represents the symbol 0 and the symbol 1. After "001011100" following the guide pulse 10a, a symbol 0 (10c) of 0.6 ms following the non-output 10b appears, and "0" appears, followed by a symbol 1 of 1.2 ms following the non-output 10d. (10e) appears and indicates 1. Thereafter, "10" appears, the data runs out, and there is no output until the next guide pulse appears.
[0032]
The CMOS image sensor 2 of the present invention repeats infrared light imaging at a frame rate of 1920 frames per second, that is, 0.52 millisecond intervals in the carrier detection mode. In infrared communication, the shortest pulse is a non-output of 0.6 milliseconds or a symbol 0, and the non-output and the symbol 0 are separated by photographing at an interval of 0.52 milliseconds.
[0033]
FIG. 5 shows symbol detection and data demodulation.
[0034]
In the figure, 11a, 11b, 11c, 11d and 11e are infrared light spots of the remote controller 1, 31a and 31b are data output from the remote controller 1, 5 is a network to which the microprocessor 3 is connected, and 6 is home appliances to be connected to the network. Goods.
[0035]
In the carrier detection mode, the infrared light output from the remote controller appears as a bright spot on the image captured by the CMOS image sensor 2. In the figure, the infrared light of the remote controller 1a sequentially appears as spots 11a, 11b, 11c, 11d, and 11e in five frames photographed every 0.52 seconds. The spot 11a and the spot 11c are not completely dark because any one of the symbols before and after the exposure time is applied, but are determined to be non-output because they are smaller than the threshold value. In addition, since it is higher than the threshold, the spots 11b, 11d, and 11e are determined to be outputs. The symbol 0 or the symbol 1 is distinguished depending on whether the number of outputs until the no output appears after the spot 10a determined to be no output is one or two. The non-output that appears for the first time after the spot 11a is the spot 11c, and the output in the meantime is only the spot 11b. Similarly, since the spot 11c is followed by the spot 11d and the spot 11e, the data 31b output from the remote controller 1a is demodulated into the symbol 1.
[0036]
Since the data 31b output from the remote controller 1a is data during infrared communication, it is determined that there is no output after the spot 11e and demodulated to the symbol 1. When an output appears after the spot 11e, the signal up to the spot 11e is demodulated into the symbol 1 as indicated by 31b, and the spot arriving after the spot 11e is regarded as an error, and the first spot is regarded as no output, and an error correction is attempted.
[0037]
The demodulated data includes information such as a maker code and a product code. If the data indicates another manufacturer or another product, the microprocessor 3 searches for the corresponding home appliance on the network 5 and then transmits the data to the network 5. The data is relayed to the network 5 or the data is relayed to the network 5 in expectation of receiving the corresponding home electric appliance on the network 5. As a result of the relay, the home electric appliance 6 on the network 5 is applicable, so the command is interpreted and the command is normally executed.
[0038]
When the microprocessor 3 finds a corresponding product by searching on the network 5, it obtains control system information of the product. The control system information is defined in, for example, UPnP: Universal Plug and Play proposed by Microsoft Corporation in the United States or ECHONET of the ECHONET Consortium in Japan. It switches between operation and heating operation, and controls air volume, louver direction, and temperature.
[0039]
Control of home electric appliances via the network 5 will be described with reference to FIGS.
[0040]
In FIG. 6, 1a and 1b denote remote controller positions, 12a and 12b denote infrared light spots of the remote controller 1, 32a denotes an acceleration of the infrared light spot in the y-axis direction, and 32b denotes an acceleration of the infrared light spot in the x-axis direction. , 33a, 33b are remote control screens displayed by the microprocessor, and 330a, 330b, 330c are menu items on the remote control screen.
[0041]
FIG. 6 shows selection of a menu item by a gesture.
[0042]
As shown in FIG. 5, the microprocessor 3 demodulates the data output from the remote controller 1 to obtain information such as a maker code and a product code. In FIG. 6, the microprocessor 3 acquires control system information of the product from the obtained information such as the manufacturer code and the product code. The microprocessor 3 inquires the calling state of the product via the network 5, creates a control menu, and displays it as a remote control screen 33a. The menu item 330a on the remote control screen presents three control commands as Up, the menu item 330b on the remote control screen as Down, and the menu item 330c on the remote control screen as Off.
[0043]
The user turns the remote controller 1 toward the CMOS image sensor 2 and presses or keeps pressing any button of the remote controller 2 to output infrared rays. The remote controller 1 which has been stationary at the remote controller position 1a at a certain point of time, lifts up to the remote controller position 1b by a gesture according to the menu selection of the user, and soon returns to the remote controller position 1a and stops.
[0044]
At this time, in imaging by the CMOS image sensor 2, the remote controller 1 is stationary at the infrared light spot 12 a at frame 1 but rises to the infrared light spot 12 b at frame 111 after 110 frames, and In the subsequent frame 221, the light returns to the infrared light spot 12 a again, and in the subsequent frames, it stops at this position.
[0045]
In the present invention, since the CMOS image sensor 2 can obtain an image at a high frequency of 1920 fps in the carrier detection mode, the image is obtained even between the first, 111th, and 221st frames. For the frame captured by the CMOS image sensor during this time, the microprocessor 3 tracks the spot of the infrared light output by the remote controller 1, calculates accelerations Ax and Ay in the x-axis and y-axis directions, and calculates the acceleration of the infrared light spot. An acceleration 32a in the y-axis direction and an acceleration 32b in the x-axis direction of the infrared light spot are obtained.
[0046]
The microprocessor 3 recognizes that the remote controller 1 has lifted up from the acceleration 32a of the infrared light spot in the y-axis direction and the acceleration 32b of the spot in the x-axis direction. The microprocessor 3 determines that the user wants to raise the highlight of the menu item on the remote control screen 33a, switches the menu item 330b that has already been highlighted to the menu item 330a, and shifts to the remote control screen 33b.
[0047]
In the transition from the menu screen 33a to the menu screen 33b, the infrared light output from the remote controller 1 does not irradiate or point at the menu items on the menu screen 33a or the menu screen 33a, and the user bows the remote controller 1. The highlight is moving on the remote control screen.
[0048]
In FIG. 7, 1a and 1c indicate positions of the remote controller, 12c indicates an infrared light spot of the remote controller 1, 32c indicates the acceleration of the infrared light spot in the y-axis direction, 32d indicates the acceleration of the infrared light spot in the x-axis direction, and 33c. , 33d are remote control screens displayed by the microprocessor, and 330d is a menu item on the remote control screen.
[0049]
FIG. 7 shows a pop-up of a menu item by a gesture.
[0050]
For simplicity, differences from FIG. 6 will be described. Similarly to FIG. 6, the remote controller 1 which has been stopped at the remote controller position 1a is horizontally moved to the remote controller position 1c by a gesture according to the user's menu selection, and soon returns to the remote controller position 1a and stops. During this time, the remote control 1 is in a state of outputting infrared rays.
[0051]
At this time, in the imaging by the CMOS image sensor 2, the remote controller 1 is stationary at the infrared light spot 12a at frame No. 1, but swings rightward to the infrared light spot 12C at frame number 111 after 110 frames. , 110 frames, the frame 221 returns to the infrared light spot 12a again, and in the subsequent frames, it stops at this position.
[0052]
The microprocessor 3 tracks the spot of the infrared light output from the remote controller 1, calculates accelerations Ax and Ay in the x-axis and y-axis directions, and calculates the acceleration 32c of the infrared light spot in the y-axis direction and the infrared light spot. Is obtained in the x-axis direction.
[0053]
The microprocessor 3 recognizes that the remote controller 1 has touched the right from the acceleration 32c of the infrared light spot in the y-axis direction and the acceleration 32d of the spot in the x-axis direction. The microprocessor 3 determines that the user wants to display or select a menu item below the menu item 330a for the highlighted menu item 330a on the remote control screen 33c, and the menu item 330a that has already been highlighted , The menu item 330d is popped up rightward, and the screen shifts to the remote control screen 33d.
[0054]
In the transition from the menu screen 33c to the menu screen 33d, the infrared light output from the remote controller 1 does not irradiate or point at the menu screen 33c or the menu item on the menu screen 33c, and the user moves the remote controller 1 to the right. By shaking, the lower-level menu item pops up on the remote control screen, and the highlight moves to the pop-up menu item.
[0055]
In FIG. 8, 1a and 1d are the positions of the remote controller, 12d is the infrared light spot of the remote controller 1, 32e is the acceleration of the infrared light spot in the y-axis direction, 32f is the acceleration of the infrared light spot in the x-axis direction, and 33e. , 33f are remote control screens displayed by the microprocessor.
[0056]
A selection of a menu item by a gesture will be described with reference to FIG.
[0057]
As in FIG. 6, the remote controller 1 which has been stopped at the remote controller position 1a is swung so as to draw a circle passing through the remote controller position 1d by a gesture accompanying the user's menu selection, and soon returns to the remote controller position 1a and stops. I do. During this time, the remote control 1 is in a state of outputting infrared rays.
[0058]
At this time, in the imaging by the CMOS image sensor 2, the remote controller 1 is stationary at the infrared light spot 12a at frame 1 but is at the infrared light spot 12d at frame number 111 after 110 frames. In the subsequent frame 221, the light returns to the infrared light spot 12 a again, and in the subsequent frames, it stops at this position.
[0059]
The microprocessor 3 tracks the spot of the infrared light output from the remote controller 1, calculates accelerations Ax and Ay in the x-axis and y-axis directions, and calculates the acceleration 32e of the infrared light spot in the y-axis direction and the infrared light spot. The acceleration 32f in the x-axis direction is obtained.
[0060]
The microprocessor 3 recognizes that the remote controller 1 has been swung in a circle from the acceleration 32e of the infrared light spot in the y-axis direction and the acceleration 32f of the spot in the x-axis direction. The microprocessor 3 determines that the user has selected the highlighted menu item 330d on the remote control screen 33e, highlights or blinks the already highlighted menu item 330d, and shifts to the remote control screen 33f. I do.
[0061]
In the transition from the menu screen 33e to the menu screen 33f, the infrared light output from the remote controller 1 does not irradiate or point at the menu screen 33e or the menu item on the menu screen 33e. Shake like a drawing to highlight or blink the lower-level menu item highlighted on the remote control screen.
[0062]
As shown in FIGS. 6 to 8, the microprocessor 3 updates the remote control screen by recognizing a command intended by the user by capturing a gesture of the user swinging the remote control 1 with the CMOS image sensor 2. The command thus recognized is sent to the product via the network 5 shortly to realize remote operation.
[0063]
Further, the microprocessor 3 hierarchically displays functions conventionally assigned to buttons on the remote control on the remote control screen, adds unique functions to menu items, and displays the functions according to the frequency of use. It has a so-called learning function, or dynamically changes the menu hierarchy until the function is called.
[0064]
9 and 10 show a second embodiment of the present invention.
[0065]
In the figure, 1g and 1h are the positions of the remote controller, 12g and 12h are the infrared light spots of the remote controller 1, 32g is the acceleration of the infrared light spot in the y-axis direction, 32h is the acceleration of the infrared light spot in the x-axis direction, 33g and 33h are remote control screens displayed by the microprocessor, 330e, 330f, 330g, 330h and 330i are menu items on the menu screen, and 61 is a home appliance.
[0066]
The microprocessor 3 demodulates the data output from the remote controller 1 to obtain information such as a maker code and a product code. The microprocessor 3 acquires control system information of the product from the obtained information such as the manufacturer code and the product code. The microprocessor 3 inquires the calling state of the home appliance 6 as the product via the network 5, creates a control menu, and displays it as a remote control screen 33a. The menu item 330e on the remote control screen presents three control commands as Up, the menu item 330f on the remote control screen shows Down, and the menu item 330g on the remote control screen shows Off as three control commands.
[0067]
The user turns the remote controller 1 toward the CMOS image sensor 2 and presses or keeps pressing any button of the remote controller 2 to output infrared rays. The remote controller 1 which has been stationary at the remote controller position 1g at a certain point in time is moved up to the remote controller position 1h by a gesture accompanying the menu selection of the user and stays there.
[0068]
At this time, in the imaging by the CMOS image sensor 2, the remote controller 1 is stationary at the infrared light spot 12g at the first frame, but rises to the infrared light spot 12h at the frame 111 after 110 frames, and the 110th frame. It stops at the spot 12h in the subsequent frame 221 and in this position in the subsequent frames.
[0069]
In the second embodiment of the present invention, the microprocessor 3 tracks the spot of infrared light output from the remote controller 1, calculates accelerations Ax and Ay in the x-axis and y-axis directions, and calculates y of the infrared light spot. An acceleration 32g in the axial direction and an acceleration 32h in the x-axis direction of the infrared light spot are obtained.
[0070]
The microprocessor 3 recognizes that the remote control 1 has been hiccuped and stopped from the acceleration 32g in the y-axis direction of the infrared light spot and the acceleration 32h in the x-axis direction of the spot. The microprocessor 3 determines that the user wants to switch the menu item on the remote control screen 33g from the control screen of the home electric appliance 6 to a list of other home electric appliances information, and searches for the peripheral electric home appliances. The home appliance 61 searched on the network 5 is found, and the state is inquired. The result of the inquiry is displayed as a menu item 330i on the remote control screen 33h below the menu item 330h indicating the operation state of the household electrical appliance 6, and the remote control screen shifts to 33h.
[0071]
In the transition from the menu screen 33g to the menu screen 33h, the infrared light output by the remote controller 1 does not irradiate or point at the menu items on the menu screen 33g or the menu screen 33h, and the user bows the remote controller 1. The highlight is moving on the remote control screen.
[0072]
Originally, the remote controller 1 holds the control code of the home appliance 6 or a device corresponding to the home appliance 6 and transmits only the control code assigned to the button of the remote controller 1. However, the remote controller 1 has a different control code from the home appliance 61. Cannot be controlled. In addition, since the manufacturer code and product code at the head of the data output from the remote controller 1 do not match as the home appliance 61, the home appliance 61 discards this data without interpreting the control code. That is, although the remote controller 1 and the home appliance 61 are distant from each other, as shown in the present embodiment, the microprocessor 3 intervenes to present the menu screen 33h and allow the user to select a menu item. The control shifts to the control of the home electric appliance 61.
[0073]
In this configuration, even when the microprocessor 3 is not found because the home electric appliance 6 is not connected to the network 5, the microprocessor 3 searches for surrounding home electric appliances and controllable products and creates and displays a menu screen. I do. The contents displayed at this time are menu items including the same product names and nicknames as the menu items 330h and 330i that have appeared on the menu screen 33h, and their operation states.
[0074]
FIG. 10 shows a remote controller.
[0075]
In the figure, 1 is a remote controller, 100 is an infrared light emitting unit, 101 is a power button, 102 is a numeric keypad, 103 is an arrow key, and 104 is a solar cell panel with a touch sensor.
[0076]
The remote controller 1 has a power button 101, a numeric keypad 102, and an arrow key 103 as the minimum necessary buttons for a remote controller of a normal television or VCR. The arrow keys can be replaced with 2, 4, 6, 8, or similar combinations of the numeric keys 102, and can be omitted. However, in the second embodiment of the present invention, the arrow keys are used for a volume button and fast-forward / rewind.
[0077]
In the control using the remote control screen displayed by the microprocessor 3, the user first presses the power button 101, and data including a maker code and a product code is transmitted by infrared rays. Thereafter, a menu item is selected by a gesture. As long as the user holds the remote controller 1, the remote controller 1 detects that the remote controller 1 is being held through the solar cell panel 104 with a touch sensor. During this time, unless the power button 101, the numeric keypad 102, or the arrow key 103 is pressed, the remote controller continues to transmit the reserved control code without using the home appliance that the remote controller originally controls. The transmission of the reserved control code stops when the power button 101, the numeric keypad 102, or the arrow key 103 is pressed or when the user's hand leaves the touch-sensitive solar panel 104 for a while.
[0078]
In the embodiment of the present invention, the solar cell panel 104 with a touch sensor is mounted. However, it is possible to transmit the reserved control code as expected even with a touch sensor in which the solar cell panel is omitted.
[0079]
In addition to transmitting the reserved control code, the remote controller also transmits a code for ignoring and discarding even received home appliances such as a reserved maker code and a product code, and performs a similar function by transmitting a guide signal. A similar function is achieved even if a non-output appears subsequently or an error signal is transmitted such that the guide signal is repeated. When the microprocessor 3 cannot identify the manufacturer code or the product code by intentionally embedding the error in this way, the microprocessor 3 of the present invention searches for surrounding home electric appliances and controllable products, and creates and displays a menu screen. , Realize remote control.
[0080]
【The invention's effect】
With the configuration of the CMOS image sensor and the microcomputer of the present invention, it is possible to control a household electrical appliance by swinging a remote controller rolling in a general home and performing a predetermined gesture during imaging by the CMOS image sensor.
[0081]
Since the buttons on the remote control are not used, the function you were looking for on the remote control appears in the menu display, so you can call the function without dropping your eyes on the buttons on the remote control, reducing or eliminating the number of buttons on the remote control And the liquid crystal display of the remote controller can be omitted.
[0082]
Replacement or additional home appliances can be controlled with the conventional remote controller, and the number of remote controllers can be reduced to one or one for each person, and the cost can be reduced by omitting the remote controller attached to the home appliance. It also saves the trouble of searching for a lost remote control.
[Brief description of the drawings]
FIG. 1 is a diagram showing a first embodiment.
FIG. 2 is a diagram illustrating an operation mode of the CMOS image sensor according to the first embodiment.
FIG. 3 is a diagram illustrating a reading order in a carrier detection mode of the CMOS image sensor according to the first embodiment.
FIG. 4 is a diagram showing a format and timing of infrared communication according to the first embodiment.
FIG. 5 is a diagram showing symbol detection and data demodulation in infrared communication according to the first embodiment.
FIG. 6 is a diagram illustrating selection of a menu item by a gesture according to the first embodiment.
FIG. 7 is a diagram illustrating a pop-up of a menu item by a gesture according to the first embodiment.
FIG. 8 is a diagram illustrating selection and confirmation of a menu item by a gesture according to the first embodiment.
FIG. 9 is a diagram showing a second embodiment.
FIG. 10 is a diagram showing a remote controller according to the second embodiment.
FIG. 11 is a diagram showing a conventional technique.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Remote control, 1a, 1b, 1c, 1d, 1g, 1h ... Remote control position, 10 ... Infrared light spot, 10a ... Guide pulse, 10b, 10d ... No output, 10c ... Symbol 0, 10e ... Symbol 1, 11: infrared light spot of remote control 1a, 11a, 11b, 11c, 11d, 11e: infrared light spot of remote control 1, 12: infrared light spot of remote control 1, 100: infrared light emitting section, 101: Power button, 102 numeric keypad, 103 arrow keys, 104 solar cell panel with touch sensor, 12a, 12b, 12c, 12d, 12g, 12h infrared light spot of remote controller 1, 2 CMOS image sensor, 3 ... Microprocessor, 31a, 31b: data output from remote controller 1, 32a, 32c, 32e, 32g: y-axis of infrared light spot Direction acceleration, 32b, 32d, 32f, 32h ... acceleration in the x-axis direction of the infrared light spot, 33a, 33b, 33c, 33d, 33e, 33f, 33g, 33h ... remote control screen displayed by the microprocessor, 330a, 330b , 330c, 330d, 330e, 330f, 330g, 330h, 330i: Menu items on the menu screen, 4: Bus, 41a: Visible image shooting mode setting command, 42a: Image data in the visible image shooting mode, 41b: Carrier detection mode Setting command, 42b: imaging data in the carrier detection mode, 5: a network to which the microprocessor 3 is connected, 6: home appliances connected to the network, 61: home appliances connected to the network, 7: laser pointer, 70: laser pointer, 70: Laser light drawn on the wall Tsu door, 8 ... projector, remote control screen menu items of 81, 82, 83 ... projector to display.

Claims (13)

A remote control system comprising an image sensor and a processor connected to a network,
The image sensor includes a state in which all the light receiving elements are read out one by one, a state in which the light receiving elements capturing a specific color among all the light receiving elements are read out in a column direction and a row direction,
The processor is an acceleration detection unit that detects an acceleration between imaging of the object being imaged,
Gesture identifying means for identifying a gesture from the acceleration of the object and the trajectory of the object provided by the means for detecting the acceleration, and searching or calling a device on the network to collect a state and a control command system, and to execute a control command from the state of the device. Comprising device management means for creating a reconfigured control command menu and display means for displaying the reconfigured control command menu,
The remote control system detects the trajectory and acceleration of the light emitted by the controller that emits the light of the color received by the light receiving element that captures the specific color subjected to pulse modulation, and detects the gesture drawn by the controller during the imaging. To identify,
The device management means of the remote control system switches or instructs a control command to be instructed according to the identified gesture, based on a control command and a positional relationship instructed from a control command group of a control command menu displayed on the display means. To confirm the selection of the control command
The switching of the control command or the selection of the determined control command is displayed on the display means,
The remote control system according to claim 1, wherein the remote control is performed by transmitting a selected control command to a device on the network via the network. 2. The image sensor and controller according to claim 1, wherein the specific color is red or an electromagnetic wave having a wavelength of 500 nm or more, or near infrared light or approximately 950 nm. 2. The acceleration detecting means of the processor according to claim 1, wherein the controller is the object, and a pulse-modulated light transmitted from the controller during imaging is recognized, a position during imaging is detected, and trajectory information is obtained. An acceleration detecting means for detecting the acceleration, detecting the acceleration by detecting and comparing the position information between the imagings and calculating the difference. 2. A gesture identifying means for identifying a gesture of the processor according to claim 1, wherein a time point at which the acceleration departs from a stationary state where the acceleration is substantially zero is determined as a starting point based on the acceleration and trajectory information provided by the acceleration detecting means. As an end point, a period from the start point to the end point is cut out from the trajectory information,
Gesture identifying means for comparing the extracted trajectory information with a pre-registered gesture, identifying the gesture, and transmitting the gesture information. The optical communication by a controller for emitting pulse-modulated light according to claim 1, wherein the controller optically transmits data including a maker code, a product code, and a control code subsequent to a synchronization code or a guide pulse. To send,
The controller, wherein the controller repeatedly transmits the data during a period in which the controller draws a gesture. 6. The controller of claim 5, further comprising a touch sensor for contacting a user holding the controller, wherein the touch sensor periodically detects a contact, and automatically detects when the touch sensor detects the contact. A controller that repeatedly transmits the data until the next contact is detected. 7. The controller according to claim 6, wherein, while the touch sensor detects a contact, it detects that a key to which another control command is assigned is pressed, and interrupts the operation of automatically transmitting repeatedly. A controller characterized in that data corresponding to a control command is transmitted with priority. The data according to claim 5, wherein at least one of the maker code, the product code, and the control code is a reserved code that has not been assigned to an existing product. communication. 6. The data according to claim 5, wherein the data to be transmitted subsequent to the synchronization code or the guide pulse is data in which an error is randomly added to the maker code, product code, and control code, or incomplete data in which a part is deleted. The controller and the optical communication, wherein the controller is a non-output or a synchronous code or a guide pulse. 2. The device management means according to claim 1, wherein a corresponding device on a network is searched based on data including a maker code, a product code, and a control code transmitted by the controller, and a control command menu for the found device is created. Device management means characterized by doing. 2. The device management unit according to claim 1, wherein when the data transmitted by the controller includes an error, is undefined, or is invalid, a device on the network is searched and a list of the found devices is displayed on the display unit. Device management means for collecting the operation status and control command system for each discovered device and creating a control command menu. 11. The device management means according to claim 10, wherein if the device cannot be found or if the controller selects a device list display request on the network presented in the control command menu, the device is searched for and found. Device management means for displaying a list of devices which have been identified by the display means, collecting an operation status and a control command system for each discovered device, and creating a control command menu. 4. The acceleration detecting means according to claim 3, wherein the light of the specific color has a position information and a light for a certain period when it is detected that the light of the specific color appears continuously and substantially at the same position for one to several seconds during the imaging. An acceleration detecting means which does not detect acceleration.

Images