TW201301877A - Imaging sensor based multi-dimensional remote controller with multiple input modes - Google Patents

Abstract

A method for generating a TV input command using a remote controller having an imaging sensor is presented. The method identifies the comers of a TV display screen from a graphical image captured by an imaging sensor of the remote controller. The method can then perform edge detection and segmentation of the graphical image using reference TV dimensions of the TV display screen to identify the pixel coordinates of the four corners of the TV display screen in the graphical image. The method can then map a camera center position in the pixel coordinates to virtual TV coordinates using a cross ratio algorithm, and then map a location of a cursor in the virtual TV coordinates to the coordinates of the TV display screen.

Description

Multi-dimensional remote control with multiple input modes based on image sensor

The present invention relates to a remote controller, and more particularly to a multi-input mode remote controller for a network television having an image sensor.

The recent development of Internet TV (Internet TV) not only provides traditional TV programs, but also provides interactive content such as 2D and 3D web pages, online 2D, 3D games and so on. In order to meet the needs of daily extended users for the input features of Internet TV, there is a strong need for next generation remote controls that provide improved input capabilities over existing input devices.

It is an object of the present invention to provide a network television remote control having a multi-input mode of an image sensor.

In one embodiment, the present invention provides a method of generating an input command for a television using a remote control having an image sensor. This method identifies the corner of the television display by means of a graphic image captured by the image sensor of the remote control. The method then performs edge detection and segmentation of the image using the reference television size of the television display screen to identify the pixel coordinates of the four corners of the television display in the graphical image. The method of the present method then maps one of the camera coordinates to a virtual television coordinate using a cross ratio algorithm. The position corresponding to one of the virtual television coordinates is then to the coordinates of the television display screen.

In another embodiment of the present invention, the present invention provides a remote controller including an image sensor, a multi-touch touch panel, and a microprocessor. The image sensor is used to perform image capture and depth sensing. The microprocessor is configured to process the image sensor data and receive the multi-touch touch panel data from the image sensor and the multi-touch touch panel in one of the plurality of input modes. In a gesture input mode or a cursor control mode, the microprocessor is configured to perform the following steps: (i) recognizing a graphic image captured from the image sensor of the remote controller to identify a corner of a television display screen, and (ii) Performing edge detection and segmentation of the graphics image using the reference television size of the television display screen to identify the pixel coordinates of the four corners of the television display screen in the graphics image.

In one embodiment, the present invention provides a remote control input system for a television. The system includes a remote control as described above and a television remote control support module. The television remote control support module is configured to (i) map a camera center position to a virtual television coordinate in the pixel coordinate using a cross ratio algorithm, and (ii) correspond to a virtual The position of one of the cursors in the TV coordinates to the coordinates of the screen of the television display.

The embodiments of the invention described below are illustrative and are not intended to limit the scope of the invention. It will be understood that the present invention utilizes one or more computer readable mediums, and each medium can be used to contain operational data or computer executable material. The computer executable data includes various data structures, programs or other programming modules. In addition, the computer readable medium can be random access memory (RAM), read only memory (ROM) or other capable of providing data or executing instructions. Device or part. All computer readable media are suitable for use in the present invention, but in some embodiments, the computer readable medium is tangible non transitory computer readable media.

Hardware and software structure of Internet TV and remote control

Please refer to FIG. 1. FIG. 1 is a perspective view of the remote controller. As shown in FIG. 1 , the remote controller 100 includes a 2D or 3D image sensor 102 and a touch panel 104 , wherein the image sensor 102 has pixel coordinates, and the touch panel 104 is a multi-finger touch panel or a digital digit. A digitizer is used to control the network television 110. The touchpad 104 operates by transmitting absolute coordinates or by transmitting relative coordinates of the conventional mouse mode. In addition, remote control 100 includes one or more buttons 106 for operating network television 110 or changing the mode of remote control 100.

The firmware of the remote controller 100 wirelessly transmits the data packet to the network television 110 via the wireless communication device 112, and the network television 110 performs wireless communication via the wireless communication device 114 of the network television 110. The above wireless communication method can be Bluetooth, WiFi or other suitable technologies. The network television 110 includes a remote controller support module that receives signals from the remote controller 100 and generates multi-dimensional input commands, such as controlling cursors.

The image sensor 102 of the remote controller 100 has two functions: image capture and depth sensing. Depth sensing refers to the distance between the remote control 100 and the objects in the 3D environment in pixel coordinates. Using the 2D color image captured by the image sensor 102 and the depth sensing process, the remote controller 100 can map the pixel coordinates of the camera center position to the cursor position in the TV screen coordinates.

Please refer to FIG. 2, which is a block diagram of the remote controller. As described above, the remote controller 100 includes an image sensor 102, a multi-finger touch pad 104 or a tablet, and a plurality of buttons 106 or buttons The pressure switch, wherein the image sensor 102 can be a CMOS 2D/3D sensor. As shown in FIG. 2, the image sensor 102, the multi-finger touch pad 104 and the button 106 are electrically connected to the microprocessor 200, and as shown in FIG. 1, the data of the microprocessor 200 is transmitted to the network through the wireless communication device 112. Road TV 110.

Please refer to FIG. 3. FIG. 3 is a block diagram of the remote controller support module of the firmware of the remote controller and the network television. The firmware 300 is disposed inside the remote controller 100, and the remote control support module 302 is disposed inside the network television 110. The firmware 300 receives the non-continuous data group of the multi-finger touch panel 104 and the image sensor 102, and transmits the wireless The method transmits the non-contiguous data set to the wireless communication driver 306 of the network television 110.

In some preferred embodiments, remote control 100 is a composite USB human interface device, and remote control 100 includes at least two independent logic devices. The first logic device is a multi-finger touch panel, and the second logic device is an image sensor. The television operating system 304 receives the data packets from the remote controller 100 via the wireless communication driver 306, and the data packets are sequentially transmitted to the USB tablet driver 308 and the cursor navigation module 320.

Multidimensional input function

The independent input control mode for the remote control will be explained below, please continue to refer to Figure 3. The remote controller of the present invention is a multifunctional multi-dimensional remote controller having three independent input control modes.

The first control mode is a cursor position mode, and the cursor position of the network television is controlled by combining the image sensor and the touchpad. The second control mode is a gesture input mode, and the input signal is generated by the image sensor of the remote controller when the remote controller moves in the open space. The third control mode is a multi-finger input mode, in which a multi-finger touch signal is generated by a plurality of fingers touching the touch panel.

Any of the above control modes can be applied to 2D, 3D or other multidimensional input commands. very The above input control mode can be switched by pressing one or more buttons of the remote controller.

In the cursor position mode, image sensor data generated by moving the remote controller has higher priority than touch panel data. The image sensor data can be directly pointed to by the absolute position command to navigate the cursor on the TV screen. In the cursor position mode, if the displacement rate of the image sensor is less than a preset value, for example, the remote controller is stationary, the touch panel data will be recognized and applied to the precise cursor control movement. For example, when the user points to the object on the screen through the remote control, the object selection can be made. When the above input mode is executed and the user needs to fine tune the cursor position, the user can move the cursor by touching the pad to achieve accurate cursor control without moving the remote controller.

The gesture input mode performs gesture input according to the manner in which the body moves in the open space. For example, the user holds the remote controller 100 and moves the remote controller 100 to perform a gesture. These gestures will form an input command based on the gesture database. The touchpad in gesture input mode has no function, so any contact on the touchpad is not recognized and no input commands are generated.

Gesture input mode uses only image sensor data. Similarly, the multi-finger input mode uses only touchpad data. In this mode, the touch panel data is the absolute touch position data generated by the finger gesture input of the touch panel, and all image sensor data will be ignored.

Cursor position control mode

Please continue to refer to FIG. 3, in particular, the application of the remote controller support module 302 when the remote controller is in the cursor control mode. In the cursor control mode, the television display generates a cursor image in which the position of the cursor is obtained by calculating image sensor data and touch panel data. The image sensor data generated by the image sensor pointing to the TV display can be used for rough cursor control (low precision). The coarse control of the cursor is an absolute position command.

During this process, the firmware of the remote controller performs real-time image processing for detecting the four corners of the television frame and transmitting the data to the remote controller support module 302 in the network television. In the cursor position control mode, the touch panel data generated by the finger dragging on the touch panel becomes precise cursor position control (high precision) through the relative pointing method.

As shown in FIG. 3, the remote controller support module 302 includes a cursor navigation module 320. The cursor navigation module 320 maps the pixel coordinates of the camera center position to the TV screen coordinates of the absolute cursor position. The cursor navigation module 320 further includes an interaction ratio module 312, a camera center position mapping module 314, and/or a virtual cursor mapping module 316.

The interaction ratio module 312 is configured to calculate an interaction ratio value of five points, wherein the quadrant code of the five points and the preset virtual mark correspond to the quadrant code and the pixel coordinates of the virtual television frame image. The camera center position mapping module 314 is used to map the pixel coordinates of the camera center position to the virtual television coordinates of the virtual cursor. The above process is accomplished by quadrant code, cross-proportion, and virtual TV coordinates of preset virtual tags. The virtual cursor mapping module 316 is configured to map the virtual cursor in the virtual TV frame to the cursor in the real TV frame. The cursor pointing module 310 generates a cursor position command by the output of the cursor navigation module 320, thereby assisting precise cursor control and smoothing the process. In addition, in some preferred embodiments, the support television software 302 generates a cursor command through the data packet of the multi-finger touch panel. The program module of Figure 3 will be further described in the subheading below.

Image processing of remote control and TV remote control support module

Referring again to FIG. 3, the image sensor captures the image, and the image is processed by the firmware 300 of the remote controller and/or the remote controller support module 302 of the network television. The basic steps of image processing are as follows Shown. First, the image parameters are identified through the initial steps. These parameters include the size of the television, the distance between the television display and the image sensor, the horizontal direction of the image sensor, the vertical direction and the direction of rotation, and the set of mapping parameters selected by the television remote control support module 302.

Then, the edge detection and segmentation steps are performed. In this step, the image sensor is used to detect the four corners of the TV frame, and the firmware is responsible for segmentation of the image.

Next, a continuous mapping step is performed. In this step, the television remote control support module 302 maps the coordinates of the absolute cursor position to the coordinates of the television display. Table 1 below shows the steps performed by the image sensor and the television remote control support module 302. The contents of Table 1 will be further explained.

Detect edges, segment TV frames, and identify four corners of a TV screen

The detection of the TV screen and the identification of the four corners of the TV screen can be accomplished by a depth sensing camera that simultaneously captures RGB digital images and depth images (distance data for each pixel between the camera and the object). The camera can be a commercial 3D sensing camera, such as Microsoft's Kinect®, wherein the hardware structure of the Kinect® 2D/3D image sensor is combined with a conventional 2D color image wafer and a separate 3D depth sensing chip ( Such as the Kinect® sensor). In addition, a single hardware chip that simultaneously detects images of 2D color television images and depth-sensing images is also applicable. In some embodiments, the remote control utilizes 2D image and depth sensing data to detect four corners of the television frame in the pixel plane, as shown in FIGS. 4A through 4F. 4A is a schematic diagram of a television environment in which a television display screen captures graphic images. The television environment includes a television 400, a wall 402 located behind the television 400, a recess 404 located further behind the television 400, a character 406, and a podium 408 located to the right of the television 400.

When the remote controller performs cursor navigation initialization, the remote controller transmits an instruction to the television 400 to display a single graphic image, such as a solid color screen 410. As shown in FIG. 4B, the solid color screen 410 is filled with a hatched portion in the figure, wherein the color may be blue. The remote control captures the video image and the depth data by capturing the graphic image.

For the depth information of the graphic image, please refer to FIG. 4C. FIG. 4C uses different patterns to display different depths of the object. Using image data, the remote control can search for and identify the distance between the television screen 420 and the image sensor. When the distance between the remote control and the television screen 420 is recognized, the remote controller will generate a color histogram of the depth image based on the distance. For example, Figure 4C shows the depth of the objects in the image object, and each object is labeled with a different pattern depending on its depth. As shown in FIG. 4C, the objects 420 and 422 are respectively at the same distance from the image sensor.

Next, the image of FIG. 4C is processed by binarized image, and the binarized threshold is related to the distance value between the television display screen 420. As a result, the distance values of all objects will be binarized to form a solid color. Please refer to FIG. 4D. FIG. 4D is a schematic diagram of the binarized image of FIG. 4C. It should be noted that both objects 430 and 432 are shown to fill the solid line because the two are respectively at the same distance from the image sensor.

After identifying the distance data of the TV frame, the remote controller can obtain the predicted size of the TV frame through the built-in look-up table database. Please refer to Figure 4E, predicting the size through the TV frame in the database. The controller can correctly identify the object in image 440 that is the closest to the size of the television screen. When the television object is recognized, that is, after the segmented television frame is completed, as shown in FIG. 4F, the position of each corner of the television frame is calculated as a pixel coordinate. The centroid position of the television frame can be derived from the position of the four corners 450.

An interactive scale algorithm that maps the camera's center position to the TV's empty body coordinates

When the center of the TV screen is recognized as a pixel coordinate, the interactive scale algorithm maps the camera's center position to the TV frame coordinates. The interaction ratio is a ratio in the geometry of applied mathematics. For example, referring to FIG. 5, four straight lines extend from point O, wherein points x1, x2, x3, and x4 on four straight lines are related to x1', x2', x3', and x4', and thus are as follows. Equation 1 shows that the interaction ratio of (x1, x2, x3, x4) is equal to the interaction ratio of (x1', x2', x3', x4').

The interaction ratio =[(x1-x3)/(x2-x3)]/[(x1-x4)/(x2-x4)]=[(x1'-x3')/(x2'-x3')]/[ (x1'-x4')/(x2'-x4')] Equation 1

The remote controller reflects the center position of the camera in the pixel coordinate to the position of the cursor in the TV screen coordinate through the invariance of the interactive ratio. As shown in FIG. 6, FIG. 6 is a projection transition produced by the optical center 600 projecting a line through the camera lens 602, and the line extends to the television screen 604.

In conventional computer graphics, the relationship between pixel coordinates (x, y, z) and television screen coordinates (X, Y) has been defined by projection transformation. See Equation 2.

The projection transformation matrix includes a camera internal matrix and an external matrix. The internal matrix represents internal camera parameters such as focal length. The outer matrix represents the 3D transformation and rotation between the 2D pixel scene and the 3D environment (including the TV frame). Each defined 2D position in the television screen coordinates will be captured via the lens of the digital camera and the projection conversion matrix is performed via the lens such that each pixel coordinate corresponds to the television coordinates.

Projection transformation preserves the proportion of the interaction, the proportion of the length scale, the collinearity of the points, and the order of the points. Since these projection invariants are still not changed after the projection conversion, the interactive scale maps the camera center position in the pixel coordinates to the cursor position in the TV screen coordinates. In other words, the ratio of interactions calculated from the known points in the five pixel coordinates (preset four positions or virtual markers and camera center position) is equivalent to four known points in the TV screen coordinates and unknown cursors. The proportion of the interaction of the location. As shown in FIGS. 7, 8A and 8B, the cursor position corresponding to the center of the camera can estimate its unknown position (X _cursor , Y _cursor ) according to the interaction ratio in the pixel plane.

Figure 7 is a schematic illustration of projection transitions and camera positioning to a cursor. A projection center 700 of an image sensor is depicted in FIG. The projection passes through the mirror plane 704, wherein the mirror plane 704 includes camera pixel coordinates p1, p2, p3, p4 and camera center C. The projection continues and reaches a reference plane 710, wherein the reference plane 710 includes a cursor 712 and is labeled P1, P2, P3, P4 TV screen coordinates 714.

8A and 8B are schematic diagrams showing the degree of interaction between the television screen coordinates and the camera pixel coordinates as an invariant. The image sensor 106 and the television 110 having the cursor 712 and the marks m1, m2, m3, m4 are included in FIG. 8A. Figure 8B includes captured television image 800 and camera center position, where the camera center position is labeled p5 and is located at the center of the television image. The marks p1, p2, p3, p4 correspond to the marks m1, m2, m3, m4. It is worth noting that the interactive proportional _TV = F (m1, m2, m3, m4, p5) is equal to the interactive scale _camera = F (p1, p2, p3, p4, p5).

In conventional applied mathematics, any three of the five points of the two interactive proportional equations are not collinear, and their projection transformations are as follows. Through two interaction ratios and Equations 3 and 4, the unknown cursor position (X _cursor , Y _cursor ) will be obtained.

Cross Ratio 1=(|M ₄₃₁ | ^* |M ₅₂₁ |)/(|M ₄₂₁ | ^* |M ₅₃₁ |) Cross Ratio 2=(|M ₄₂₁ | ^* |M ₅₃₂ |)/(|M ₄₃₂ | ^* | M ₅₂₁ |) Equations 3 and 4

In Equations 3 and 4, M _ijk is a matrix and the matrix Where i, j, and k represent the code of the point, respectively. ( x _i , y _i ) is the 2D coordinate of the point of the label i. The scalar value |M _ijk | is the determinant of the M _ijk matrix.

The calculation process of cursor navigation through interactive scale includes the following four steps:

Step 1: In the first step, the preset four-point original data packet is received to calculate the interaction ratio or the position of the "virtual marker" in the pixel coordinates. As shown in FIG. 9, the captured image of the television screen 900 is located in the X-axis 904 and the Y-axis 902 in the pixel coordinates, wherein the captured television frame The body image includes positions Pi (i = 1-4) of the four virtual markers 908, and a fifth point P5 906 representing the center position of the camera. Taking a VGA resolution screen as an example, P5 is located at (320, 240) in the (X, Y) coordinate system.

Step 2: In the second step, the interaction ratio is calculated through the point data of step 1. For example, using the preset point data of Pi (i=1-4) and the camera center position P5, as shown below, the interaction ratio CR1 and the interaction ratio CR2 are obtained via Equations 3 and 4. It should be noted that the conditions of the three-point collinearity should be avoided.

Pi(x, y) = position of virtual marker i P5 (x, y) = camera center position in pixel plane coordinates | M 431 | = P4xP3y + P3xP1y + P1xP4y - (P1xP3y + P3xP4y + P1yP4x) | M 421 | P4xP2y+P2xP1y+P1xP4y-(P1xP2y+P2xP4y+P1yP4x) |M 432 |=P4xP3y+P3xP2y+P2xP4y-(P2xP3y+P3xP4y+P2yP4x) |M 521 |=P5xP2y+P2xP1y+P1xP5y-(P1xP2y+P2xP5y+P1yP5x) |M 531 |=P5xP3y+P3xP1y+P1xP5y-(P1xP3y+P3xP5y+P1yP5x) |M 532 |=P5xP3y+P3xP2y+P2xP5y-(P2xP3y+P3xP5y+P2yP5x)

Step 3: Estimate the position of the cursor by the interaction ratio obtained in step 2. For example, the cursor position Cur(x5, y5) in the TV screen coordinates is calculated as follows: x5=-(E ^* (A ^* FC ^* D) / (B ^* DA ^* E) + F) / D y5 = ( A ^* FC ^* D) / (B ^* DA ^* E)

The scalar values A, B, C, D, E, and F are as follows: A=(CR1 ^* |M_TV421| ^* (Y3-Y1)-|M_TV431| ^* (Y2-Y1)) B=(CR1 ^* |M_TV421 | ^* (X1-X3)-|M_TV431| ^* (X1-X2)) C=(CR1 ^* |M_TV421| ^* (X3Y1-X1Y3)-|M_TV431| ^* (X2Y1-X1Y2)) D=(CR2 ^* |M_TV432 | ^* (Y2-Y1)-|M_TV421| ^* (Y3-Y2)) E=(CR2 ^* |M_TV432| ^* (X1-X2)-|M_TV421| ^* (X2-X3)) F=(CR2 ^* |MTV432 | ^* (X2Y1-X1Y2)-|M_TV421| ^* (X3Y2-X2Y3))

In the above equation, CR1 and CR2 are obtained from step 2. The position of the virtual marker Vi = (Xi, Yi) is the preset value in the TV coordinates. The determinant |M_TV ijk| is obtained by the position Vi=(Xi, Yi) of the virtual mark in the TV coordinates.

Step 4: When the above three steps are completed, it will return and repeat step 1.

Avoid computational overflow of interactive proportional positioning

The difficulty in using the interactive scale in projection geometry is that when the three points are collinear, the interaction ratio will become zero or infinite. Figure 10 is a schematic diagram of the calculated overflow generated by three points. As shown in FIG. 10, the four corners 1002 of the television frame 1000 are preset points, which are the same as the virtual logos, and are used to calculate the interaction ratio. When the cursor 712 is located on any of the four corners 1002, a three-point collinear condition will occur.

11A and FIG. 11B are another schematic diagrams in which different virtual markers 1100 are combined to form a three-point collinear line. As shown in Figure 11A, the television screen coordinates are dispersed into the quadrant. As shown in FIG. 11B, the pixel coordinates correspond to the camera center 1110 to FIG. 11A.

To avoid collinear conditions, the cursor navigation module identifies the quadrant in which the cursor is located. Figures 12A through 12D show the appropriate locations of the virtual markers 1100 to effectively calculate the interaction scale based on the current position of the cursor in the television coordinates. The preset four points used to calculate the interaction ratio are as follows.

Figures 12A through 12D show a first virtual marker 1100 in which the first virtual marker 1100 is located in a diagonal quadrant of the quadrant in which the cursor is located. Next, the second virtual mark 1100 is located on a horizontal edge of the first virtual mark, and the second virtual mark is half a horizontal line away from the first virtual mark. The third virtual mark 1100 is located on a vertical side line where the first virtual mark is located, and the third virtual mark is one-quarter of a vertical line from the first virtual mark. The fourth virtual marker 1100 is located on the opposite vertical On the sideline, and its height is a quarter of a vertical edge.

In other words, if the cursor is in quadrant I and is shifting in the direction of quadrant I or quadrant II of the television screen, then no collinearity is created with virtual marker 1100 shown in FIG. 12A or 12B. Similarly, if the cursor is in quadrant II and is shifting in the direction of quadrant I or quadrant II of the television screen, then no collinearity is created with virtual marker 1100 shown in FIG. 12B. If the cursor is in quadrant III and is shifting in the direction of quadrant III or quadrant IV of the television screen, then no collinearity is created with virtual marker 1100 shown in Figure 12C. In addition, if the cursor is in quadrant IV and is being displaced in the direction of quadrant III or quadrant IV of the television screen, it does not create a collinear situation with the virtual marker 1100 shown in FIG. 12D.

13A to 13D are schematic diagrams of initial navigation and subsequent navigation for smoothly calculating the interaction ratio. As shown in FIG. 13A, when the cursor is navigated for the first time, the image sensor of the remote controller recognizes the captured television frame 1302, wherein the television frame 1302 includes the camera center 1304. The remote controller transmits the message data of the identification quadrant to the TV support program. As shown in FIG. 13B, the television support program receives the message material but does not update the location of the upstream 1312 of the television screen 1310.

As shown in FIG. 13C, when the quadrant and the preset four virtual markers 1306 appear in the pixel coordinates, the remote controller transmits the data group to the TV remote control support module, wherein the data group includes the current quadrant code, the positions of the four corners, and the use. To calculate the pixel coordinates of the five points of the pen. Then, the TV remote control support module calculates the interaction ratio in the pixel plane on behalf of the remote controller, and calculates the preset four virtual markers corresponding to the TV screen coordinates according to the quadrant code transmitted by the remote controller.

Virtual TV frame concept

The purpose of interactive proportional positioning is to make the camera center position in the pixel coordinates correspond to the real Cursor for TV screen coordinates. However, the above method is not suitable for the case where the remote controller is far away from the TV frame body, because the distance captured by the TV frame is too small, and the preset four points cannot be used to calculate the interaction ratio.

Therefore, the concept of the virtual TV frame can overcome the situation where the remote controller is far away from the TV frame. Figure 14 is a schematic diagram of the operation of the virtual television frame. The main concept is to define a virtual TV frame 1302, wherein the virtual TV frame 1302 must be large enough to ensure the accuracy of the calculation of the interaction ratio and to navigate the cursor. Therefore, FIG. 14 shows that a virtual TV frame 1302 is an extension of the TV frame, and the virtual TV frame 1302 is larger than the actual TV frame 110. The cursor navigation module calculates the interaction ratio through the virtual TV frame 1302, and maps the camera center to the virtual TV frame coordinates. The size of the virtual TV frame 1302 can be set by the user, and any size larger than the actual TV frame 110 can be applied.

FIG. 15 is a schematic diagram of captured images of the actual television frame 110 and the user-defined virtual television frame 1302. As shown in FIG. 15, in the pixel coordinates, the length and width of the virtual television frame 1302 is approximately three times that of the actual television frame 110.

FIG. 16A is a schematic diagram showing the calculation of the interaction ratio through the captured image of the virtual television frame. In FIG. 16A, the camera captures the image of the physical television frame 110 on the pixel plane 1600. The centroid position of the physical television frame 110 can be calculated. Next, as shown in FIG. 16A, the virtual television frame 1302 surrounds the centroid of the physical television frame 110. Next, the cursor navigation module identifies the quadrant of the virtual television frame 1302, and the camera center is located in the quadrant. Regarding the position of the preset four points 1004 (virtual mark) in the virtual television frame 1302, please refer to FIGS. 12A to 12D.

Refer to Figures 16B through 16D for the process of mapping camera center 1602 to virtual television coordinates. Figure 16B plots the calculation of the interaction ratio in virtual television coordinates 1302. Figure 16C is a schematic diagram showing the cursor 1610 and the map of the camera center mapped to the virtual television coordinates. Figure 16D depicts a cursor 1610 and a schematic diagram of the virtual cursor mapping to the real television coordinates and the real cursor 1622. When the above-described intermediate mapping process completes the calculation of the virtual cursor V _cur (Vx, Vy), then as shown in FIG. 16D, the virtual cursor position is mapped to the real virtual cursor position Cur(x, y) of the real television 110, Its calculations are shown in Equations 5 and 6 below.

x=Vx/width extension ratio y=Vy/height extension ratio equations 5 and 6

Estimate the size of the TV frame through the forecast database

In some embodiments, as shown in FIG. 4E, the look-up database is used to assist the remote controller in determining the television frame in the image. The look-up table includes the data of the size of the TV frame. The size of the TV frame (width and height in the pixel coordinates) is the actual measurement result obtained by the 2D/3D image sensor at different 3D positions and directions.

When the 3D camera obtains the distance (for example, depth) between the remote controller and the TV frame in the backward direction of the cursor, the image sensor also captures the captured image. Through the above image, the remote controller will transmit the pixel coordinates of the four corners of the camera image to the TV remote control support module.

In some embodiments, the cursor navigation module in the TV remote control support module searches the look-up table and derives a reference size of the television frame by the distance from the television. As shown in Figure 17, the lookup table will correspond to different tables depending on the distance.

However, in some embodiments, since various horizontal angles or vertical angles are formed between the remote controller and the television screen 110, the television frame captured by the image sensor of the remote controller is not labeled. Quasi-rectangular. Referring to FIG. 18 and FIG. 19, FIG. 18 is a schematic diagram showing various horizontal angles formed by the remote controller and the television screen 110, and FIG. 19 is a schematic diagram showing various vertical angles of the remote controller and the television screen 110. As shown in FIGS. 18 and 19, various captured images 1804a-e and 1900a-e captured by the remote controllers 100a-j. It is worth mentioning that the TV length 1802 in Fig. 18 is an invariant, and the TV width 1902 in Fig. 19 is an invariant.

The cursor navigation module compares the lengths of the edges of the images in FIGS. 20A to 20D by using the measured dimensions when the horizontal angle and the vertical angle are both zero, and pushes back the angle and direction of the remote controller by comparing the results. Next, the cursor navigation module searches for the closest reference size, and the reference size is transmitted to the remote controller to assist the remote controller in determining the television frame in the image, as shown in FIG. 4E.

However, in some instances, the reference size may be inaccurate if the position and orientation of the remote control are significantly changed. If the remote controller cannot segment the TV frame image smoothly, the remote controller transmits a request to the TV remote control support module for causing the television to redisplay a camera image, such as the aforementioned blue screen, to correct the reference size.

Handling Z angle

The change in Z angle comes from the rotation of the image sensor perpendicular to the TV screen. The Z angle change can be used to perform TV commands or TV volume control. In general, the change in the Z angle is not used for cursor navigation because it is not necessary for changes in the position of the cursor. Therefore, if the amount of change in the Z angle is greater than a threshold, such as 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60 or even greater than 60 degrees, the cursor navigation module changes the cursor icon A non-cursor icon customized for the user.

However, if the Z angle change is less than the threshold, the cursor navigation calculation captures the centroid of the TV frame image, and in the calculation of the interaction ratio, the Z angle of the virtual TV frame is considered to be equal to zero. Please refer to Referring to Figure 21, the Z-angle can be estimated by capturing the tangent of the longer side of the image from the X-axis and the TV frame. 21 is a schematic diagram of a captured image 2100, wherein the captured image 2100 includes a camera center position 2104 and a television frame 2102 labeled with a Z angle 2106.

Simultaneous operation of coarse-precision mode cursor instructions

Referring to Figure 22, Figure 22 plots the simultaneous coarse-precision mode input generation on the X-Y plane. As described above, the absolute position of the cursor in the television screen coordinates can be obtained by projecting the pixel plane projection to the interactive ratio of the plane of the television frame. The user-customized partial mapping area 2200 surrounding the current cursor position 2202 is used to perform precise control of the touchpad 104. The cursor can be continuously moved by the partial mapping area 2200, and when the cursor 2202 is moved, the local mapping area also synchronously follows the trajectory movement of the cursor.

The user moves the remote control 100 to navigate the cursor 2202, while the partial mapping area 2200 is the coarse control of the cursor pointing. Next, the cursor 2202 is touched or dragged with a finger to make the cursor 2202 for precise control. When performing precise control, the user can move the cursor 220 to any position through the partial mapping area 2200. FIG. 23A shows that the remote controller 100 performs coarse cursor control, and FIG. 23B shows that the precise cursor control is continued using the touch panel 106.

This concept also applies to other coarse input control devices, such as inertial sensors in remote controls. With the touchpad, the gyro sensor or acceleration sensor is also suitable for coarse control of the cursor pointing, while the touchpad 106 provides precise control of the vernier pointing. A small trackball, a small rocker or other device with a class function can replace the touchpad 106 to achieve a simultaneous coarse-accurate control method.

Referring back to Figures 24A to 24C, the size of the partial mapping area 2200 can be based on the television 110 and the remote control. Depending on the distance between the devices 100. If the distance is farther, the accuracy of the cursor control needs to be lower.

Figures 25A through 25D show three user-selectable cursor control modes that provide (i) relative cursor orientation for only the tablet, (ii) simultaneous coarse/precise cursor control, and (iii) only the imagery of the remote control The tester data is roughly controlled.

Figures 26A through 26D show how to avoid the conversion of coarse and precise controls resulting in poor cursor navigation. As shown in FIG. 26A, the drag action is just finished and the cursor 2202 is located at the upper left corner of the partial map area 2200. When the user wants to navigate the cursor through the movement of the remote controller 100, as shown in FIG. 26B and FIG. 26C, the cursor 2202 will stay at the same position. When the displacement of the remote controller 100 stops or is less than the movement threshold, the user can re-drag the finger on the touchpad to initialize the precise cursor control.

Cursor navigation when the distance between the TV and the remote control is very short

If the distance between the TV and the remote control is extremely short, the image sensor may not be able to capture the complete TV frame and cannot detect the four corners of the TV screen frame. In this example, the firmware of the remote control automatically switches the cursor navigation of the image sensor to the cursor navigation of the touch panel, and transmits a signal that is “too close to the TV” to the remote control support module of the host. The host's TV program will display a note message to inform the user of the distance issue and the availability of the tablet's cursor navigation in this case.

3D input mode

27A to 27D show that an additional two free angles (DOF) are generated by the remote controller 100. Referring to FIGS. 27A and 27B, the remote controller 100 includes a Z-axis conversion command. Referring to FIGS. 27C and 27D, the remote controller 100 includes a rotation command of the Z axis. TV remote control support module It is used to change the size of the captured TV frame objects 2710a-2710b, thereby generating an enlargement/reduction instruction. The calculation of the Z angle change has been mentioned above and is applicable to user specific instructions, such as controlling the volume level.

When a 3D image application is used, the user moves the 3D cursor in the 3D environment to select the 3D object. For example, as shown in FIG. 28A, the user moves the remote controller 100 in the front-rear direction to generate a coarse control command, and/or the user moves the remote controller 100 in the left-right direction to generate an X-conversion command of the 3D cursor. The finger touch signal is also used to generate precise control of the 3D cursor, wherein the user-defined partial mapping area 2200 is located in the X-Z plane.

As shown in FIG. 28b, the user can also move the remote controller 100 to generate a Y-conversion command or a Z-conversion command of the 3D cursor to perform a coarse control command. Furthermore, the finger touch signal can produce precise control in which the user-customized mapping area 22000 is located in the Y-Z plane.

Gesture input mode in free space

In some embodiments, the user generates a 2D or 3D gesture input through the mobile remote controller 100 in free space while in contact with the touchpad 106. As described above, the button of FIG. 1 is installed on the body of the remote controller for switching three control modes: a 2D cursor mode, a gesture mode of the mobile remote controller body, and a multi-finger gesture mode.

Figures 29A and 29B show schematic diagrams of 2D gesture input in free space. In the gesture mode, if the user touches the surface of the touchpad 106, a bubble icon 2900 will be displayed. When the user moves the remote control in free space, the bubble track will be displayed on the TV display. The gesture recognition application in the television recognizes a single stroke character 2902, such as a separate Arabic numeral. If the gesture recognition system successfully recognizes the character, the program will display a solid color track 2904 for Inform the user that this is a successful outcome. Gesture input features can also be extended to 3D gestures. As shown in FIGS. 30A and 30B, how to move the X-Z plane by the remote controller 100 in the 3D space, thereby generating 3D bubble tracks 3000, 3002.

Multi-finger input mode

In some embodiments of the multi-finger input mode, the touchpad data is treated as the only input command. In order to avoid unintended and/or unwelcome inputs due to the shaking of the remote control body, the image sensor data will be ignored as a condition.

31A through 31C depict the formation of a plurality of two-finger touches on the surface of the touchpad 106 by the finger 3100, and thereby generate an additional four DOF input commands. However, some users may want to use the tablet's multi-finger touch gesture and the mobile remote's body at the same time, so the firmware and host TV program allows the user to choose between two types of input data. Figure 32 shows the simultaneous integration of the image sensor and the touchpad to produce a spiral input command 3200.

As shown in FIG. 32, the user moves the remote controller in the front and rear direction to generate a coarse Z conversion command. At the same time, the user can generate a Z-axis rotation command by touching the finger gesture on the board 106. In combination with the above two gestures, and converting the circular trajectory into a 3D rotation instruction in the 3D image environment, the firmware in the remote controller 100 and the television interface software in the host generate a Z-axis rotation command.

The above are only the preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Therefore, any equivalent changes or modifications made without departing from the spirit of the present invention should be included in the present invention. Within the scope of the patent application.

100‧‧‧Remote control

1000‧‧‧TV frame

102‧‧‧Image sensor

1002‧‧‧ corner

104‧‧‧Touch panel

1004‧‧‧virtual mark

106‧‧‧ button

1100‧‧‧virtual mark

110‧‧‧Internet TV

1110‧‧‧virtual mark

112‧‧‧Wireless communication device

1302‧‧‧TV frame

114‧‧‧Wireless communication device

1304‧‧‧ Camera Center

200‧‧‧Microprocessor

1306‧‧‧virtual mark

300‧‧‧ Firmware

1310‧‧‧TV screen

302‧‧‧Remote Control Support Module

1312‧‧‧ cursor

304‧‧‧TV operating system

1600‧‧‧pixel plane

306‧‧‧Wireless communication driver

1602‧‧‧ Camera Center

308‧‧‧USB tablet driver

1610‧‧‧ cursor

310‧‧‧ cursor pointing module

1802‧‧‧TV length

312‧‧‧Interactive proportional module

1902‧‧‧TV width

314‧‧‧ Camera Center Position Mapping Module

2100‧‧‧ images

316‧‧‧Virtual cursor mapping module

2102‧‧‧TV frame

320‧‧‧ cursor navigation module

2104‧‧‧ Camera Center Location

400‧‧‧TV

2106‧‧‧Z angle

402‧‧‧ wall

2200‧‧‧Partial mapping area

404‧‧‧

2202‧‧‧ cursor

406‧‧‧ characters

2900‧‧‧ bubble icon

408‧‧‧Podium

2902‧‧‧Single stroke character

410‧‧‧ solid color screen

2904‧‧‧ Solid color track

420‧‧‧TV screen

3000‧‧‧ bubble track

422‧‧‧ objects

3002‧‧‧ bubble track

430‧‧‧ objects

3100‧‧‧ fingers

432‧‧‧ objects

3200‧‧‧Spiral input instructions

440‧‧ images

100a-100j‧‧‧Remote control

450‧‧‧ corner

1804a-1804e‧‧‧ Capture images

700‧‧‧Projection Center

1900a-1900e‧‧‧ capture images

704‧‧‧Mirror plane

2710a-2710d‧‧‧TV frame objects

710‧‧‧ reference plane

C‧‧‧ Camera Center

712‧‧‧ cursor

M1-m5‧‧‧ mark

900‧‧‧TV screen

P1-P5‧‧‧ TV screen coordinates

902‧‧‧Y axis

P1-p5‧‧‧ camera pixel coordinates

904‧‧‧X-axis

X1-x4‧‧‧ points on four straight lines

906‧‧‧ fifth point

X1-X4‧‧‧ points on four straight lines

908‧‧‧virtual mark

O‧‧‧a little

714‧‧‧TV screen coordinates

1622‧‧‧ true cursor

600‧‧‧Optical Center

604‧‧‧TV screen

602‧‧‧ camera lens

800‧‧‧TV images

Figure 1: A perspective view of a remote control and a network TV.

Figure 2: Block diagram of the firmware of the remote control.

Figure 3: Block diagram of the remote control support module for the remote control firmware and network TV.

4A to 4F are schematic diagrams showing the process of detecting edges, segmenting and recognizing corners.

Figure 5: A two-dimensional diagram of a four-line intersection with two planes.

Figure 6: is a three-dimensional diagram of a line crossing the second plane.

Figure 7: Schematic diagram of projection conversion and mapping to the cursor at the center of the camera.

8A and 8B are schematic diagrams showing the invariants of the ratio of the interaction between the television screen coordinates and the camera pixel coordinates.

Figure 9 is a schematic diagram of pixel coordinates with virtual markers mapped to television coordinates.

Figure 10: Schematic diagram of calculating the overflow of the three points of the virtual mark.

11A and 11B are schematic diagrams showing the virtual mark on the display screen and the captured image of the three-point collinear line.

Figures 12A through 12D are schematic diagrams showing the position of the virtual marker based on the quadrant of the cursor.

13A to 13D are diagrams showing a cursor generation process including successful execution of an interactive scale calculation.

Figure 14: is a perspective view of the remote control, the network point and the virtual TV frame.

Figure 15 is a schematic diagram of the captured image of the TV frame and the virtual TV frame.

Figure 16A is a representative diagram of capturing an image and aligning it with a virtual television frame.

16B to 16B D: are schematic diagrams showing the steps of calculating the position of the cursor.

Figure 17: is a schematic diagram of the table.

Figure 18: Front view of the graphic image of the TV display based on remote controllers with different horizontal angles Figure.

Figure 19 is a side view of a graphical image of a television display taken from a remote control with different vertical angles.

20A to 20D are television displays that are captured at different angles.

Figure 21 is a schematic illustration of a television display including a Z angle.

Figure 22 is a perspective view of a television display applied to a remote control and a remote control having coarse-precision control features.

Figure 23A is a schematic diagram showing the transition of a partially mapped area on a display screen.

Figure 23B is a schematic illustration of cursor movement for a partially mapped area on a display screen.

Figures 24A through 24C are perspective views of the dimensions of the remote control and its partially mapped regions at different distances from the television.

25A to 25C are schematic diagrams showing three independent cursor control modes.

26A to 26D are schematic diagrams of processes for avoiding smooth navigation of the cursor.

27A and 27B are schematic diagrams showing an additional free-angle input gesture and its display image for the Z-axis movement.

27C and 27D are schematic diagrams showing an additional free-angle input gesture and its display image for the Z-axis rotation.

28A and 28B are schematic diagrams showing the movement of a cursor in three dimensions.

29A to 29C are schematic views of a two-dimensional gesture in an open space.

30A to 30B are schematic views of three-dimensional gestures in an open space.

31A to 31C are schematic views of a multi-touch gesture of a multi-finger touch panel.

Figure 32 is a schematic illustration of a three-dimensional gesture through an image sensor and a touchpad.

700‧‧‧Projection Center

704‧‧‧Mirror plane

710‧‧‧ reference plane

712‧‧‧ cursor

P1-P5‧‧‧ TV screen coordinates

P1-p5‧‧‧ camera pixel coordinates

714‧‧‧TV screen coordinates

Claims (20)

A method for generating a television input command by using a remote controller having an image sensor, wherein the method includes the following steps: the image sensor of the remote controller captures a graphic image of a television display screen and transmits the image The remote controller identifies a plurality of corners of the television display screen; performing edge detection and segmentation of the graphic image by using one of the TV display screens to identify the pixel in the four corners of the television display screen of the graphic image a coordinate; a remote control support module and an interactive scale algorithm of the television, the center position of the camera located in the pixel coordinate is mapped to a virtual television coordinate; and the virtual support module is used to One of the cursors in the television coordinates is mapped to the coordinates of the television display screen. The method of claim 1, further comprising the step of: identifying a corresponding data of the television reference size through a look-up table. The method of claim 2, wherein the look-up table is included in a three-dimensional position and direction, the correspondence between the television reference size and the controller to the television display screen. The method of claim 1, further comprising the step of: transmitting the depth information of the four corners in the graphic image of the television display screen and the remote control support module of the pixel coordinate to the television. The method of claim 1, further comprising the steps of: identifying a quadrant of the camera center position of a television display screen in the graphic image; calculating a virtual marker position to avoid the interaction ratio Collinear positioning in the algorithm; The virtual marker location is used in the interactive scale algorithm. The method of claim 1, further comprising the steps of: identifying whether the distance between the remote controller and the television display screen exceeds a predetermined distance; defining a center of gravity of the television display screen of the graphic image Deriving a virtual television frame; calculating the virtual mark location by the virtual television frame; and using the virtual mark position in the interactive scale algorithm. The method of claim 1, further comprising the steps of: generating a cursor navigation command and displaying the cursor in the television display screen. The method of claim 1, further comprising the steps of: repeating edge detection and segmentation of the graphic image and mapping one of the cursors in the virtual television coordinate to the television display screen. coordinate. The method of claim 8, further comprising the step of: stopping the edge detection and segmentation of the graphic image and the virtual television coordinates when the amount of change of a Z angle is greater than a predetermined value One of the cursors in the position is mapped to the coordinates of the television display screen. The method of claim 1, further comprising the steps of: defining a partial mapping area around the cursor position of the television display screen; and inputting data based on the touch panel of the remote controller Move the cursor position in the mapped area. The method of claim 10, further comprising the step of maintaining a relative position of the cursor in the partial mapping area when the image sensor data changes the local mapping area. The method of claim 10, further comprising the step of: changing the local mapping area according to a change in distance between the remote controller and the television display screen; size. The method of claim 1, further comprising the step of: modifying the position of the cursor in the television coordinates by inputting data of one of the remote controllers. The method of claim 1, further comprising the step of: discriminating a free space gesture generated by moving the controller. The method of claim 1, further comprising the step of switching from the remote control input mode to using only one of the touchpads of the remote controller without using data from the image sensor. A remote controller includes: an image sensor for capturing a graphic image and sensing a spatial depth; a multi-touch panel; and a microprocessor for processing the image sensor based on one of the plurality of input modes Or the data of the multi-touch panel; wherein in the at least one input mode, the microprocessor recognizes a plurality of corner positions of a television display screen in one of the graphics images captured by the image sensor of the remote controller Edge detection and segmentation of the graphic image by one of the television reference screens of the television display screen to identify the pixel coordinates of the four corners of the television display screen in the graphic image. The remote controller of claim 16, wherein in the at least one input mode, the microprocessor further identifies a quadrant located at a center position of the camera of a television display screen in the graphic image, and calculates a The virtual marker position avoids the occurrence of collinear positioning in the interactive scale algorithm. The remote controller of claim 16, wherein in a multi-touch mode, the microprocessor processes only data from the multi-touch panel. A remote control input system for a television, comprising: a remote controller comprising: an image sensor for capturing a graphic image and sensing a spatial depth; a multi-touch panel; and a microprocessor for identifying the image sensor by the remote controller Extracting a plurality of corner positions of a television display screen in one of the graphic images, and performing edge detection and segmentation of the graphic image by using a television reference size of the television display screen to identify the television display in the graphic image a pixel coordinate of the four corners of the screen; and a remote control support module that maps a camera center position in the pixel coordinate to a virtual television coordinate based on an interactive scale algorithm, and the virtual television coordinates are One of the cursors is mapped to the coordinates of the television display screen. The remote control input system of claim 19, wherein the microprocessor further recognizes a quadrant located at a center position of the camera of a television display screen in the graphic image, and calculates a virtual mark position to avoid The collinear positioning situation occurs in the interactive proportional algorithm.