JP2006099228A - Data processing apparatus and program - Google Patents

Abstract

An operation instruction close to that of a conventional pointing device is issued based on movement of a subject imaged by an imaging means.
The movement of a finger in a direction (X-axis direction or Y-axis direction) perpendicular to the optical axis direction of an imaging unit is detected, and the display position of the pointer on the display screen is moved based on this movement. Also, when the movement of the finger in the optical axis direction (Z-axis direction) of the image pickup unit 14 is detected and the finger approaches the image pickup unit 14 and enters the specified range, it is designated by the display position of the pointer on the display screen. Detects the selected item selection operation.
[Selection] Figure 5

Description

  The present invention relates to a data processing apparatus and program for performing a pointer operation on a display screen using a camera.

  Conventionally, a pointer for performing an operation instruction on a display screen is always displayed on the screen, and the pointer is moved by operating a ball mouse, an optical mouse, a trackball, a touch pad, or the like. On the other hand, many small devices such as mobile phones and PDAs (Personal Digital Assistants) do not have a pointer-specific input device such as a mouse, and the display ratio of the pointer in the display screen is difficult to see. Therefore, the focus position on the screen is usually moved by key operation of the main body.

  Further, after the focus position is determined, it is necessary to perform an item selection operation in order to execute a function corresponding to the item displayed at the position. In the case of a normal mouse, the selection operation is performed by clicking or double-clicking, and when the operation is performed using the arrow keys, the selection operation is performed by pressing the enter key. When the touch panel is equipped on the screen, the selection operation can be performed by directly touching the target position.

In recent years, many portable terminals and the like are equipped with a small camera. For example, a mobile communication terminal is proposed in which a pointer movement and a click operation are performed according to a gesture of a user's fingertip photographed with a camera (see, for example, Patent Document 1).
JP 2002-290529 A

  However, the pointer operation instruction by the gesture using the camera as in the above prior art is not familiar to the user who is used to the conventional pointer movement operation on a plane such as a mouse and the selection operation by pressing the attached button. . Therefore, further development has been desired so that an operation instruction closer to a mouse or the like that is a conventional pointing device can be performed.

  An object of the present invention is to provide a data processing apparatus and a program that enable an operation instruction close to that of a conventional pointing device based on movement of a subject imaged by an imaging unit.

  In order to solve the above-described problem, in the data processing device according to the first aspect of the present invention, there is provided an image capturing unit that captures an image of a subject, and a direction perpendicular to the optical axis direction of the image capturing unit captured by the image capturing unit. A first movement detecting means for detecting movement in the image, a second movement detecting means for detecting movement of the subject imaged by the imaging means in the optical axis direction of the imaging means, and the first movement detecting means. Display control means for moving the display position of the pointer on the display screen based on movement of the detected subject in a direction perpendicular to the optical axis direction of the imaging means, and subject detected by the second movement detection means Selection operation detecting means for detecting the selection operation of an item designated by the display position of the pointer on the display screen based on the movement of the imaging means in the optical axis direction. And wherein the door.

  According to a fourth aspect of the present invention, a first movement detection function for detecting a movement of a subject imaged by the imaging means in a direction perpendicular to the optical axis direction of the imaging means is captured by the computer and the imaging means. A second movement detection function for detecting movement of the image pickup means in the optical axis direction of the image pickup means, and movement of the object detected in the first movement detection function in a direction perpendicular to the optical axis direction of the image pickup means. Based on the display control function for moving the display position of the pointer on the display screen and the movement of the subject detected in the second movement detection function in the optical axis direction of the imaging means on the display screen. This is a program for realizing a selection operation detecting function for detecting an operation for selecting an item designated by a display position of a pointer.

  According to the first and fourth aspects of the present invention, the display position of the pointer on the display screen is moved based on the movement of the subject imaging unit in the direction perpendicular to the optical axis direction, and the optical axis of the subject imaging unit is detected. Since the selection operation of the item specified by the display position of the pointer on the display screen is detected based on the movement in the direction, the operation instruction close to the conventional pointing device is based on the movement of the subject imaged by the imaging means It can be performed.

  Hereinafter, a mobile phone 10 as a data processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.

  FIG. 1A is an external perspective view of the mobile phone 10. As shown in FIG. 1A, an upper part provided with a display unit 12 and a lower part provided with an operation unit 13 are joined by a hinge 17 provided with an imaging unit 14, and the upper part about the hinge 17 as an axis. The lower part can be folded. FIG. 1B is an external view showing a folded state of the mobile phone 10. The mobile phone 10 includes a display unit 12 for confirming information and an operation unit 13 for performing simple operation instructions. In the present embodiment, the foldable mobile phone having the above-described configuration is taken as an example, but is not particularly limited.

  Next, the internal configuration of the mobile phone 10 will be described with reference to FIG. As shown in FIG. 2, the mobile phone 10 includes a control unit 11, a display unit 12, an operation unit 13, an imaging unit 14, a storage unit 15, and a detection unit 16, and these units are connected to each other via a bus 18. Yes.

  The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like (not shown). In the CPU, various controls stored in the ROM using a predetermined area of the RAM as a work area. According to the program, a control signal is sent to each of the above parts to control the overall operation of the mobile phone 10.

  The control unit 11 displays a pointer at a predetermined position on the display screen of the display unit 12 when the presence of a finger as a subject is detected from the image captured by the imaging unit 14. Here, the pointer refers to a display form that indicates a specific item, position, etc. in the display screen, and includes a so-called cursor, a reverse display of a display item, and the like. Further, the control unit 11 moves the display position of the pointer on the display screen of the display unit 12 based on the movement of the finger in the direction perpendicular to the optical axis direction of the imaging unit 14. Further, the control unit 11 detects an operation for selecting an item designated by the display position of the pointer on the display screen of the display unit 12 based on the movement of the finger in the optical axis direction of the imaging unit 14.

  The display unit 12 includes a liquid crystal display and the like, and displays an image on the display screen based on display data input from the control unit 11.

  The operation unit 13 includes various keys such as a numeric key, a character key, and a function key for inputting an operation instruction to the mobile phone 10, and outputs a press signal by the key operation to the control unit 11. The user inputs various operation instructions to the mobile phone 10 by operating the operation unit 13 while referring to the screen displayed on the display unit 12.

  The imaging unit 14 includes an optical lens made of glass, plastic, and the like, and an imaging pixel surface in which a large number of imaging elements such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor) are arranged on a plane. . When an instruction to perform shooting is input from the control unit 11, the imaging unit 14 converts the subject image input via the optical lens and formed on the imaging pixel surface into an electrical signal, and generates image data. Output to the control unit 11.

  The storage unit 15 includes a recording medium in which various control processing programs executed by the control unit 11 and data used by these programs are stored in advance. This recording medium is composed of a magnetic / optical recording medium, a nonvolatile memory, or the like, and stores an OS (Operating System), various application programs, various condition settings or data. Note that a part or all of the programs or data stored in the storage unit 15 may be received from other devices via a communication network and stored.

  In addition, the storage unit 15 stores images indicating reference positions A, B, and C that serve as a reference for finger movement, which will be described later. Further, the storage unit 15 stores the occupancy rate of the finger portion with respect to the entire image and the distance from the finger to the imaging unit 14 in association with each other. Specifically, the storage unit 15 may store a conversion table in which the occupation rate and the distance are associated with each other, and the occupation rate for calculating the distance from the finger to the imaging unit 14 is a variable. The calculated formula may be stored.

  The detection unit 16 includes a detection DSP (Digital Signal Processor), and detects a finger and a finger movement from an image captured by the imaging unit 14 according to an instruction from the control unit 11. As the finger moving direction, as shown in FIG. 3, the Z axis is taken in the optical axis direction of the imaging unit 14, and the X axis and the Y axis are taken in the direction perpendicular to the Z axis.

The detection unit 16 detects the presence of a finger from the image captured by the imaging unit 14.
FIG. 4A is a schematic diagram of a captured image 21 obtained by n × m imaging elements in the imaging unit 14. In each image element, a pixel column x consisting of n columns and a pixel row y consisting of m rows are arranged in a lattice matrix. FIG. 4B is a diagram illustrating the user's fingertip 23 in the captured image 22. The shaded portion of the captured image 22 is a pixel that is detected to be a skin color based on hue, lightness, and saturation. When the shape of the pixel portion detected to be skin color is a shape indicating the characteristics of the finger, the occupancy rate of the finger portion with respect to the entire image is calculated, and the occupancy rate exceeds a specified value (for example, 10%) In addition, the presence of a finger is detected. In order to prevent frequent changes in the judgment of whether or not a finger is present due to camera shake or the influence from the periphery of the captured image, the change in the occupation ratio is within a predetermined range (for example, 10% ± 1%). It is preferable to detect the presence of a finger when the operation continues for a predetermined time (for example, 1 second). Although the detection here is performed from the detection of the skin color and the determination of the shape, the color and shape may be set in any way and are not particularly limited.

The detection unit 16 detects the movement direction and amount of movement of the finger in the directions perpendicular to the optical axis direction of the imaging unit 14 (X-axis direction and Y-axis direction in FIG. 3) from the image captured by the imaging unit 14.
FIG. 5 (a) shows reference positions A and B that serve as references for finger movement. A downward movement is indicated by the finger moving from the reference position A to the reference position B. Conversely, the upward movement is indicated by the finger moving from the reference position B to the reference position A. FIG. 5 (b) shows reference positions A and C which are the reference for finger movement. When the finger moves from the reference position A to the reference position C, a rightward movement is indicated. Conversely, the movement of the finger from the reference position C to the reference position A indicates a leftward movement.

  In the reference position setting mode, when the user actually performs an operation, the shape, position, and movement of the image at the reference position are captured, and images of the reference positions A, B, and C suitable for each user are stored. Also good.

The detection unit 16 detects the movement of the finger in the optical axis direction (Z-axis direction in FIG. 3) of the imaging unit 14 from the image captured by the imaging unit 14.
When the finger approaches the imaging unit 14, the occupation ratio of the finger portion with respect to the entire image increases, and conversely, when the finger moves away, the occupation ratio decreases. The detection unit 16 calculates the distance from the finger to the imaging unit 14 based on this occupation rate. The detection unit 16 may acquire a distance corresponding to the occupation rate based on the conversion table stored in the storage unit 15, or by a calculation formula using the occupation rate stored in the storage unit 15 as a variable. The distance may be calculated.

  Next, the operation of the mobile phone 10 will be described with reference to the flowcharts shown in FIGS. 6, 8, 10, and 11. The operation process here is a process executed by the control unit 11 when performing a screen operation on the display unit 12 of the mobile phone 10. This flowchart is for explaining a program for causing a computer constituting the mobile phone 10 to realize each function.

  As shown in FIG. 6, first, an image is taken by the imaging unit 14, and image data is captured (step S1). The image data is stored in the storage unit 15.

  Next, it is determined whether or not a finger is present in the image (step S2). If a finger is present in the image (step S2; YES), it is determined whether or not the point control mode in which point control is performed by moving the finger of the user is set (step S3). When the point control mode is not set (step S3; NO), the point control mode is set (step S4), and a pointer is displayed at a reference position on the display screen of the display unit 12 (for example, the center of the screen). (Step S5). FIG. 7A shows an example of the display screen of the display unit 12 before the presence of a finger is detected. The pointer is not displayed until the presence of a finger is detected. FIG. 7B shows an example in which the presence of a finger is detected and the pointer 31 is displayed on the display screen of the display unit 12.

  In step S3, when the point control mode is set (step S3; YES), the process proceeds to step S6.

  Next, image analysis is performed (step S6), and the image captured by the imaging unit 14 is compared with the images indicating the reference positions A, B, and C stored in the storage unit 15, so that the finger is moved. It is determined whether or not the reference position is A, B, C (steps S7, S8, S9).

  If the finger is at the reference position A (step S7; YES), the process proceeds to step S21 in FIG. When the finger is at the reference position B (step S7; NO, step S8; YES), the process proceeds to step S51 in FIG. If the finger is at the reference position C (step S7; NO, step S8; NO, step S9; YES), the process proceeds to step S71 in FIG.

  If the finger is not in any of the reference positions A, B, and C (step S7; NO, step S8; NO, step S9; NO), the process returns to step S1.

  In step S2, when no finger is present in the image (step S2; NO), the point control mode is canceled (step S10), and the pointer displayed on the display screen of the display unit 12 is deleted (step S10). Step S11). If the screen operation on the display unit 12 is not terminated (step S12; NO), the process returns to step S1. When the screen operation on the display unit 12 is to be ended (step S12; YES), the operation process is ended. The end of the screen operation indicates the end of the pointer movement process with the finger and the selection process of the pointer, and is ended in response to an operation for instructing the end by an operation switch (not shown) of the operation unit 13. Further, in response to an operation for instructing the start by the operation switch, the process shown in the flow of FIG. 6 is started, and a pointer moving process and a pointer selecting process are performed.

Next, with reference to FIG. 8, a process when it is determined in step S7 in FIG. 6 that the finger is at the reference position A will be described.
As shown in FIG. 8, an image is taken by the imaging unit 14, and image data is captured (step S21).

  Next, image analysis is performed (step S22), and the shape of the image and the occupancy rate of the finger portion with respect to the entire image are stored in the storage unit 15 as a history. Then, it is determined whether or not the movement of the user's finger in the direction perpendicular to the optical axis direction of the imaging unit 14, that is, the X-axis direction or the Y-axis direction shown in FIG. 3 is detected (step S23). When the movement of the user's finger in the X-axis direction or the Y-axis direction is detected (step S23; YES), the detected movement direction is stored in the register f1 in the control unit 11 (step S24).

  When the movement direction is the direction from the reference position A to the reference position B (step S25; YES), as shown in FIG. 9A, a downward arrow 32 indicating the movement direction is displayed on the display screen of the display unit 12. Displayed (step S26), the pointer 31 on the display screen is moved downward according to the amount of finger movement (step S27). When the pointer 31 moves to the bottom on the display screen, the entire screen is scrolled up.

  When the moving direction is the direction from the reference position A to the reference position C (step S25; NO, step S28; YES), a right-pointing arrow is displayed on the display screen of the display unit 12 (step S29), and the finger is moved. The pointer on the display screen is moved to the right according to the amount (step S30). When the pointer moves to the rightmost position on the display screen, the entire screen is scrolled to the left.

  If the movement direction is from the reference position B to the reference position A, or from the reference position C to the reference position A (step S25; NO, step S28; NO, step S31; YES), these fingers The display position of the pointer is not moved along with this movement, and the process proceeds to step S32. When it is determined in step S7 in FIG. 6 that the finger is at the reference position A, the movement in the reverse direction (the direction from the reference position B to the reference position A, or the direction from the reference position C to the reference position A). By disregarding, the display position of the pointer can be continuously moved in the forward direction.

Next, the distance Z ₀ from the imaging unit 14 to the finger in the optical axis direction (Z-axis direction) of the imaging unit 14 is acquired based on the occupancy rate of the finger portion with respect to the entire image (step S32), and the process returns to step S21. .

  If the movement of the user's finger in the X-axis direction or the Y-axis direction is not detected in step S23 (step S23; NO), the movement of the finger in the optical axis direction (Z-axis direction) of the imaging unit 14 is detected. Is done. First, it is determined whether or not there is a change in the occupancy ratio of the finger portion with respect to the entire image (step S33). If there is a change in the occupancy rate (step S33; YES), it is determined whether the occupancy rate is equal to or less than a specified value (step S34). When the occupation ratio is equal to or less than the specified value (step S34; YES), it is considered that the user has lifted his / her finger from the shooting range of the imaging unit 14, and the process proceeds to step S40.

When the occupation ratio is larger than the specified value (step S34; NO), the distance from the imaging unit 14 to the finger is calculated based on the occupation ratio (step S35). Then, it is determined whether or not this distance is within a specified range (step S36). For example, the distance from the current imaging unit 14 to the finger, using the distance Z ₀ obtained in step S32, whether or not within the range of ± 20% of the Z _0/2 around the Z _0/2 is To be judged.

  If the distance is within the specified range as a result of bringing the finger close to the imaging unit 14 (step S36; YES), pointer selection processing is performed (step S37), and the process proceeds to step S40. The pointer selection process is a process for selecting an item designated by the display position of the pointer. FIG. 9B shows a display example when the pointer selection process is performed when the pointer 31 is at the position indicating the item 7. The item 33 portion 33 on the display screen is highlighted to indicate that it is selected.

  If the moving direction is neither the direction from the reference position B to the reference position A or the direction from the reference position C to the reference position A in Step S31 (Step S31; NO), or if the occupation rate does not change in Step S33 ( (Step S33; NO) or when the distance is not within the specified range in Step S36 (Step S36; NO), the count-up is started (Step S38). In a state where the moving direction is neither the direction from the reference position B to the reference position A, the direction from the reference position C to the reference position A, a state where there is no change in the occupation ratio, or a state where the distance is not within the specified range, When the count value M is equal to or greater than the predetermined value (step S39; YES), the process proceeds to step S40. The process that moves from step S39 to step S40 is when the user interrupts the operation with the finger and the above-described state has elapsed for a predetermined period. If movement of the finger is detected before the count value M reaches the predetermined value (step S39; NO), the process returns to step S21.

  In step S40, if a downward or rightward arrow indicating the moving direction is displayed, the display is deleted (step S40). Then, the counter (count value M) is cleared (step S41), and the process proceeds to step S10 in FIG.

Next, with reference to FIG. 10, a process when it is determined in step S8 of FIG. 6 that the finger is at the reference position B will be described.
As shown in FIG. 10, first, an upward arrow is displayed on the display screen of the display unit 12 (step S51). Subsequently, an image is captured by the imaging unit 14 and image data is captured (step S52).

  Next, image analysis is performed (step S53), and the shape of the image and the occupancy rate of the finger portion with respect to the entire image are stored in the storage unit 15 as a history. Then, it is determined whether or not the movement of the user's finger in a direction perpendicular to the optical axis direction of the imaging unit 14 (X-axis direction or Y-axis direction) is detected (step S54). When the movement of the user's finger in the X-axis direction or the Y-axis direction is detected (step S54; YES), the detected movement direction is stored in the register f1 in the control unit 11 (step S55).

  When the movement direction is the direction from the reference position B to the reference position A (step S56; YES), the pointer on the display screen is moved upward according to the movement amount of the finger (step S57). If the pointer moves to the top of the display screen, the entire screen is scrolled down.

  When the moving direction is the direction from the reference position A to the reference position B (step S56; NO, step S58; YES), the pointer display position is not moved along with the movement of the finger, and the process proceeds to step S59. .

Next, the distance Z ₀ from the imaging unit 14 to the finger in the optical axis direction (Z-axis direction) of the imaging unit 14 is acquired based on the occupancy rate of the finger portion with respect to the entire image (step S59), and the process returns to step S52. .

  If the movement of the user's finger in the X-axis direction or the Y-axis direction is not detected in step S54 (step S54; NO), the movement of the finger in the optical axis direction (Z-axis direction) of the imaging unit 14 is detected. Is done. First, it is determined whether or not there is a change in the occupancy ratio of the finger portion with respect to the entire image (step S60). If there is a change in the occupancy (step S60; YES), it is determined whether the occupancy is equal to or less than a specified value (step S61). When the occupation ratio is equal to or less than the specified value (step S61; YES), it is considered that the user has lifted his / her finger from the shooting range of the imaging unit 14, and the process proceeds to step S67.

  When the occupation rate is larger than the specified value (step S61; NO), the distance from the imaging unit 14 to the finger is calculated based on the occupation rate (step S62). Then, it is determined whether this distance is within a specified range (step S63).

  If the distance is within the specified range (step S63; YES), pointer selection processing is performed (step S64), and the process proceeds to step S67.

  If the moving direction is not the direction from the reference position A to the reference position B in step S58 (step S58; NO), if there is no change in the occupation ratio in step S60 (step S60; NO), or the distance is specified in step S63. If it is not within the range (step S63; NO), the count-up is started (step S65). When the count value M is greater than or equal to a predetermined value when the moving direction is not the direction from the reference position A to the reference position B, the occupancy rate is not changed, or the distance is not within the specified range ( Step S66; YES), the process proceeds to Step S67. The process that proceeds from step S66 to step S67 is when the user interrupts the operation with the finger and the above-described state has elapsed for a predetermined period. If the movement of the finger is detected before the count value M reaches the predetermined value (step S66; NO), the process returns to step S52.

  In step S67, the display of the upward arrow indicating the moving direction is deleted (step S67). Then, the counter (count value M) is cleared (step S68), and the process proceeds to step S10 in FIG.

Next, with reference to FIG. 11, a process when it is determined that the finger is at the reference position C in step S9 in FIG. 6 will be described.
As shown in FIG. 11, first, a leftward arrow is displayed on the display screen of the display unit 12 (step S71). Subsequently, an image is captured by the imaging unit 14 and image data is captured (step S72).

  Next, image analysis is performed (step S73), and the shape of the image and the occupancy rate of the finger portion with respect to the entire image are stored in the storage unit 15 as a history. Then, it is determined whether or not the movement of the user's finger in a direction (X-axis direction or Y-axis direction) perpendicular to the optical axis direction of the imaging unit 14 is detected (step S74). When the movement of the user's finger in the X-axis direction or the Y-axis direction is detected (step S74; YES), the detected movement direction is stored in the register f1 in the control unit 11 (step S75).

  When the movement direction is the direction from the reference position C to the reference position A (step S76; YES), the pointer on the display screen is moved to the left according to the movement amount of the finger (step S77). If the pointer moves to the leftmost position on the display screen, the entire screen is scrolled to the right.

  When the movement direction is the direction from the reference position A to the reference position C (step S76; NO, step S78; YES), the pointer display position is not moved along with the movement of the finger, and the process proceeds to step S79. .

Next, the distance Z ₀ from the imaging unit 14 to the finger in the optical axis direction (Z-axis direction) of the imaging unit 14 is acquired based on the occupancy ratio of the finger portion with respect to the entire image (step S79), and the process returns to step S72. .

  If the movement of the user's finger in the X-axis direction or the Y-axis direction is not detected in step S74 (step S74; NO), the detection of the finger movement in the optical axis direction (Z-axis direction) of the imaging unit 14 is detected. Is done. First, it is determined whether or not there is a change in the occupancy ratio of the finger portion with respect to the entire image (step S80). If there is a change in the occupancy rate (step S80; YES), it is determined whether the occupancy rate is equal to or less than a specified value (step S81). When the occupation ratio is equal to or less than the specified value (step S81; YES), it is considered that the user has lifted his / her finger from the shooting range of the imaging unit 14, and the process proceeds to step S87.

  When the occupation ratio is larger than the specified value (step S81; NO), the distance from the imaging unit 14 to the finger is calculated based on the occupation ratio (step S82). Then, it is determined whether this distance is within a specified range (step S83).

  If the distance is within the specified range (step S83; YES), pointer selection processing is performed (step S84), and the process proceeds to step S87.

  If the moving direction is not the direction from the reference position A to the reference position C in step S78 (step S78; NO), if the occupation rate does not change in step S80 (step S80; NO), or the distance is specified in step S83. If not within the range (step S83; NO), the count-up is started (step S85). When the count value M is equal to or greater than a predetermined value when the moving direction is not the direction from the reference position A to the reference position C, the occupancy rate is not changed, or the distance is not within the specified range ( Step S86; YES), the process proceeds to Step S87. The process that moves from step S86 to step S87 is when the user interrupts the operation with the finger and the above-described state has elapsed for a predetermined period. If movement of the finger is detected before the count value M reaches the predetermined value (step S86; NO), the process returns to step S72.

  In step S87, the left arrow indicating the moving direction is deleted (step S87). Then, the counter (count value M) is cleared (step S88), and the process proceeds to step S10 in FIG.

  As described above, according to the mobile phone 10 of the present embodiment, the display position of the pointer on the display screen is moved based on the movement of the finger in the direction perpendicular to the optical axis direction of the imaging unit 14, and the imaging unit Since the selection operation of the item designated by the display position of the pointer on the display screen is detected based on the movement of the finger in the direction of the optical axis 14, the conventional operation is performed based on the movement of the finger imaged by the imaging unit 14. The operation instruction close to the pointing device can be performed. Further, since a special pointing device is not required, the present invention can be realized by software processing without greatly changing the conventional mobile phone.

  In addition, when the presence of a finger is detected from an image captured by the imaging unit 14, a pointer can be displayed at a predetermined position on the display screen. Therefore, since the pointer is not displayed when the finger is not detected, an unintended pointer is not displayed on the display screen, and the screen configuration can be easily viewed.

  In the present embodiment, the cellular phone 10 is taken as an example of the data processing device, but a PDA with a digital camera, a PC (Personal Computer) with a digital camera, a digital camera, or the like may be used, and is not particularly limited.

  In the present embodiment, the example using the reference positions A, B, and C as shown in FIGS. 5A and 5B has been described. However, when the moving range of the display position is narrow, the reference position The position may be set near the center of the front of the imaging unit 14, the movement of the subject from this position may be detected, and the display position of the pointer may be moved vertically and horizontally.

  In the present embodiment, the pointer is not displayed in the normal mode, and the pointer is displayed on the display screen when the point control mode is set. However, the pointer is displayed in the normal mode and the point control is performed. At the start of the mode, the pointer is displayed differently from the normal display (for example, the pointer blinks or is displayed larger than the normal display), and the process moves to the pointer movement process and the pointer selection process. You may make it alert | report to a user.

  Further, in the present embodiment, in response to an operation for instructing start by an operation switch (not shown) of the operation unit 13, the processing shown in the flow of FIG. 6 is started to perform pointer movement processing and pointer selection processing. Further, in response to the operation for instructing the end by the operation switch (step S12; YES), the processing is ended. However, this operation switch is not always necessary, and the operation switch is not necessary at step S2 in FIG. In the state where the presence is not detected, processing other than the pointer movement processing and the pointer selection processing is performed. When the presence of a finger is detected in step S2, the processing automatically shifts to the pointer movement processing and the pointer selection processing. You may do it. Alternatively, the camera portion of the imaging unit 14 may be rotated to a specific position to automatically shift to pointer movement processing and pointer selection processing.

In the present embodiment, the distance from the imaging unit 14 to the finger is within a range of Z _0/2 ± 20% with reference to the distance Z ₀ acquired when movement in the X-axis direction and the Y-axis direction is detected. When it enters, it is determined as a pointer selection operation, but the distance from the imaging unit 14 to the finger enters the specified range (for example, Z _0/2 ± 20%) twice within a predetermined time Alternatively, it may be detected as a pointer selection operation. This operation is equivalent to a conventional double click with the mouse, and can provide an easy-to-understand operation to the user as a pointer selection operation. Further, depending on the number of times the distance from the imaging unit 14 to the finger has entered the specified range within a predetermined time, for example, one time may be distinguished as a click operation, and two times may be distinguished as a double click operation. It can also be set to perform other selection operations.
Further, when the distance from the imaging unit 14 until the finger becomes shorter than the distance Z ₀ obtained when moving the detection of X-axis and Y-axis directions, but was detected as the pointer selection operation, the distance Z ₀ If it becomes longer, it may be detected as a pointer selection operation. In this case, too, the distance is within the specified range within a predetermined time or the selection operation is performed according to the number of times. You may make it set.

  Further, in the present embodiment, a finger that exists within a specified range of distance from the imaging unit 14 is detected, but the finger is moved so as to cover the imaging unit 14 and its moving direction is detected. It is good.

  Further, in the present embodiment, the thumb is described as an example of the subject imaged by the imaging unit 14, but the present invention is not limited to this, and another object such as a finger or a pen may be used. Further, the position of the imaging unit 14 in the data processing apparatus such as the mobile phone 10 is not limited to the illustrated example, and may be any position that is easy for the user to use.

  In the present embodiment, the finger movement direction in the direction perpendicular to the optical axis direction of the imaging unit 14 is detected separately for the X-axis direction and the Y-axis direction. The directions may be combined into a vector and the pointer may be moved in the combined direction.

<Modification>
As a modified example, instead of displaying the reference position determined by the pointer in step S5 in FIG. 6 as the initial position at this position, the pointer is displayed at this position with the position opposite to the moving direction as the initial position on the display screen. It is good to do. For example, after the downward arrow is displayed in step S26 of FIG. 8, the pointer is displayed at this position with the upper center position on the display screen as the initial position. Further, after a right arrow is displayed in step S29 of FIG. 8, the pointer is displayed at this position with the center left position on the display screen as the initial position. Further, after an upward arrow is displayed in step S51 of FIG. 10, the pointer is displayed at this position with the lower center position on the display screen as the initial position. Further, after the left arrow is displayed in step S71 of FIG. 11, the pointer is displayed at this position with the center right position on the display screen as the initial position.

(A) is an external appearance perspective view of the mobile phone 10 in embodiment of this invention, (b) is an external view which shows the folded state of the mobile phone 10. FIG. It is a block diagram which shows the internal structure of the mobile telephone 10 in this invention. It is a figure for demonstrating the moving direction of a finger | toe. (A) is the schematic diagram of the captured image in the mobile telephone 10, (b) is the figure which showed the user's fingertip in the captured image. (A) is a figure which shows the reference positions A and B used as the reference | standard of finger movement, (b) is a figure which shows the reference positions A and C used as the reference | standard of finger movement. 4 is a flowchart for explaining an operation process of the mobile phone 10. (A) is an example of the display screen of the display part 12 before presence of a finger | toe is detected, (b) is an example of the display screen of the display part 12 at the time of a pointer display. 4 is a flowchart for explaining an operation process of the mobile phone 10. (A) is an example of the display screen of the display unit 12 when the pointer is moved, and (b) is an example of the display screen of the display unit 12 when the pointer is selected. 4 is a flowchart for explaining an operation process of the mobile phone 10. 4 is a flowchart for explaining an operation process of the mobile phone 10.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Mobile telephone 11 Control part 12 Display part 13 Operation part 14 Imaging part 15 Storage part 16 Detection part 17 Hinge 18 Bus

Claims (4)

Imaging means for imaging a subject;
First movement detection means for detecting movement of a subject imaged by the imaging means in a direction perpendicular to the optical axis direction of the imaging means;
Second movement detecting means for detecting movement of the subject imaged by the imaging means in the optical axis direction of the imaging means;
Display control means for moving the display position of the pointer on the display screen based on movement of the subject detected by the first movement detection means in a direction perpendicular to the optical axis direction of the imaging means;
Selection operation detection that detects a selection operation of an item designated at the display position of the pointer on the display screen based on the movement of the imaging unit in the optical axis direction of the subject detected by the second movement detection unit Means,
A data processing apparatus comprising: The data processing apparatus according to claim 1,
Subject detection means for detecting the presence of the subject from an image captured by the imaging means;
The data processing apparatus, wherein the display control means displays a pointer at a predetermined position on the display screen when the presence of the subject is detected by the subject detection means. The data processing apparatus according to claim 1 or 2,
Storage means for storing an image indicating a reference position of the subject;
The first movement detection unit compares the image captured by the imaging unit with an image indicating the reference position of the subject stored in the storage unit, so that the light of the imaging unit of the subject is detected. A data processing apparatus for detecting movement in a direction perpendicular to an axial direction. On the computer,
A first movement detection function for detecting movement of a subject imaged by the imaging means in a direction perpendicular to the optical axis direction of the imaging means;
A second movement detection function for detecting movement of the subject imaged by the imaging means in the optical axis direction of the imaging means;
A display control function for moving the display position of the pointer on the display screen based on the movement of the subject detected in the first movement detection function in a direction perpendicular to the optical axis direction of the imaging means;
Selection operation detection that detects a selection operation of an item designated by the display position of the pointer on the display screen based on the movement of the subject detected in the second movement detection function in the optical axis direction of the imaging unit Function and
A program to realize

Images