JP2010118018A - Position identifying device, position identification program, computer-readable storage medium, and position identification method - Google Patents

Abstract

An object of the present invention is to make it possible to specify a contact position regardless of the color of a contact object that contacts a display surface of a planar member while reducing the size of the apparatus.
A sensor built-in liquid crystal panel (301) including a plurality of pixel circuits (31) and a plurality of photosensors (6) provided for each pixel circuit (31) and each of the pixel circuits (31) are connected to R pixel circuits (31r-B). Each time the image display processing unit 18 that sequentially develops color in the pixel circuit 31b and the image display processing unit 18 perform color development in the R pixel circuit 31r to the B pixel circuit 31b, an image of the pen P that is in contact with the sensor built-in liquid crystal panel 301 is displayed. Based on each of the images of the pen P scanned by the scan processing unit 12 that scans using each of the optical sensors 6 and the R pixel circuit 31r to B pixel circuit 31b for color development by the scan processing unit 12, respectively. A contact position determination processing unit 14 </ b> A for specifying a contact position on the P-sensor-equipped liquid crystal panel 301 is provided.
[Selection] Figure 1

Description

  The present invention relates to a position specifying device, a position specifying program, a computer-readable recording medium, and a position specifying method, and more specifically, a position specifying device and a position specifying program for specifying a contact position of a contact object such as a pen. The present invention also relates to a computer-readable recording medium on which the position specifying program is recorded, and a position specifying method.

  As is well known, image display devices having a liquid crystal display as an image display unit are widely used in various devices such as mobile phones and PDAs (Personal Digital Assistants).

  In particular, a PDA has been provided with a touch sensor (analog resistance film type, capacitance type) for a long time, enabling touch input to input information by directly bringing a finger or the like into contact with a liquid crystal display. In addition, in a mobile phone and other devices, a sensor built-in type liquid crystal display device including a touch sensor is expected to spread.

  On the other hand, for example, an electronic blackboard having a drawing display function recognizes an instruction position such as a pen touching the display screen, and displays a predetermined color in a predetermined area including the instruction position on the display screen. You can draw on the screen.

  Combining such an electronic blackboard function with an image display function such as a liquid crystal display can further improve user convenience.

  As a conventional example of a drawing display device that electronically draws on a display screen using a pen or the like, for example, Patent Literature 1 discloses a technique related to an input color determination method on an electronic blackboard.

  In this electronic blackboard input color determination method, the color detector in the pen holder determines which of the red, blue, and black felt pens remains in the pen holder provided on the electronic blackboard. Thus, it is possible to determine the color of the felt pen currently in use that does not exist in the pen holder and to draw with that color.

  As another example, Patent Document 2 discloses a technique related to a display method in CAD (computer-aided design).

  In this CAD display method, a pen number corresponding to a specific color can be input, and based on the input pen number, a color corresponding to the pen number can be drawn on the display screen. ing.

  As yet another example, Patent Document 3 discloses a technique related to a message print method.

  In this message printing method, a printed photograph is printed out with a handwritten message written in the photo data by writing a message using a light pen on the display screen on which the photo data is displayed. . The message color is selected by selecting an icon or the like displayed on the display screen with a light pen.

  As yet another example, Patent Document 4 discloses a technique related to a coordinate detection device.

  This coordinate detection apparatus draws along the movement of the pen with the detected pen thickness.

  As yet another example, Patent Document 5 discloses a technique related to an input device using an input pen.

  In the input device using the input pen, a display unit is provided at the input pen tip, and display according to the input mode is performed on the display unit.

  As yet another example, Patent Document 6 discloses a technique related to an ultra-thin input / output integrated information processing apparatus.

  In this ultra-thin input / output integrated information processing device, R (Red), G (Green), and B (Blue) color filters are provided in front of the image sensor and TFT liquid crystal to read images and recognize characters. Can be done.

  As still another example, Patent Document 7 discloses a technique related to a flat display device.

  In this flat display device, the light pen 171 uses a blue LED 601, and the LED 601 is turned on when the light exit port 602 of the pen tip comes into contact with the display panel 148 or approaches within a certain distance.

The light emitted from the LED 601 is reflected by the inner surface of the pen shaft 603 and guided to the exit 602. The power switch SW of the LED 601 is an on / off unit attached to the pen tip 611, a proximity sensor. It is controlled by the detection part of the infrared sensor.
JP 61-213918 (published September 22, 1986) Japanese Patent Laid-Open No. 02-121072 (published May 8, 1990) Japanese Patent Laid-Open No. 10-268440 (released on October 9, 1998) Japanese Patent Laid-Open No. 7-200216 (published August 4, 1995) JP 9-54652 A (published February 25, 1997) Japanese Patent Laid-Open No. 4-282609 (released on October 7, 1992) JP 2006-243850 A (published September 14, 2006) Japanese Patent Laid-Open No. 1-188069 (published July 27, 1989) JP-A-4-313722 (published on November 5, 1992)

  By the way, as described above, in the ultra-thin input / output integrated information processing apparatus disclosed in Patent Document 6, by arranging the image sensors for each of the sub-pixels corresponding to the R, G, and B pixels, Image display by color TFT liquid crystal and reading of a color image of an image reading object placed on the display screen are possible.

  However, this ultra-thin input / output integrated information processing apparatus only discloses that all the pixels of the color TFT liquid crystal transmit light when reading a color image of an image reading object. .

  In addition, the ultra-thin input / output integrated information processing apparatus disclosed in Patent Document 6 has a problem in that a pen tip having the same color as the color of the display screen may not be detected as an image.

  For example, when the color of the pen tip is red, the pen tip absorbs red light, so that the red light is not reflected from the pen tip. Therefore, when the unit pixel around the contact portion of the pen tip happens to be colored in red, it is impossible to determine whether or not the pen has touched.

  Further, none of Patent Documents 1 to 5 and 7 disclose or suggest such problems.

  Here, although it is a technical field regarding a scanner, there is a color scanner disclosed in Patent Documents 8 and 9 as another example of a method of reading an image.

  FIG. 15C is a schematic diagram conceptually showing the operation of this conventional color scanner. As shown in FIG. 15C, the color scanners disclosed in Patent Documents 8 and 9 sequentially irradiate a color image with R, G, and B light, and the image sensor captures the image. Yes.

  However, since the technologies disclosed in Patent Documents 8 and 9 are technologies that belong to the technical field of scanners, the viewpoints of performing color detection or position detection using pixels for image display are both described and Not suggested.

  Further, the color scanners disclosed in Patent Documents 8 and 9 use another optical system element such as a lens, and the viewpoint of miniaturization of the apparatus is not considered at all.

  The present invention has been made in view of the above-described conventional problems, and its object is to provide a color of a contact object that comes in contact with the display surface of an input / output integrated display (planar member) useful for downsizing the apparatus. It is an object of the present invention to provide a position specifying device, a position specifying program, a computer-readable recording medium on which the position specifying program is recorded, and a position specifying method.

  In order to solve the above-described problem, the position specifying device of the present invention includes a planar member including a plurality of unit pixels and a plurality of imaging sensors provided for each unit pixel, and each of the unit pixels. A first color, a second color, and a third color that satisfy the principle of equal color, and a color control unit that sequentially develops the unit pixels for the first color and the second color, respectively. Each time the color is developed with the third color, the first color and the second color are obtained by the image pickup control means for picking up an image of the contact object in contact with the planar member using the image pickup sensors and the image pickup control means. And a position specifying means for specifying a contact position of the contact object on the planar member based on each of the images of the contact object picked up each time the color and the third color are developed. It is said.

  In order to solve the above-described problem, the position specifying method of the present invention uses a planar member including a plurality of unit pixels and a plurality of imaging sensors provided for each unit pixel. In the method, each of the unit pixels includes: a coloring control step of sequentially developing each of the unit pixels in a first color, a second color, and a third color that satisfy a principle of equal colors; and the coloring control step. An imaging control step of imaging an image of a contact object in contact with the planar member using each of the imaging sensors each time the first color, the second color, and the third color are developed, and the imaging control A position for identifying a contact position of the contact object on the planar member based on each of the images of the contact object imaged each time the first color, the second color, and the third color are developed in the step. Including specific steps It is a symptom.

  According to the configuration or the method, the planar member includes a plurality of unit pixels and a plurality of imaging sensors provided at least one for each unit pixel.

  Therefore, a case where an image is displayed on a planar member (for example, on a predetermined display surface by operating a plurality of unit pixels of the planar member is considered. It can be displayed on the “surface”.

  In addition, the light emitted from the unit pixels of the planar member is irradiated with light on the display surface, and the reflected light is detected by the imaging sensors of the planar member. By doing so, it is possible to capture an image of the contact object on the display surface.

  According to the above configuration, since it is not necessary to configure the image display unit and the image capturing unit as separate devices, the input / output integrated device can be downsized.

  Incidentally, as described above, the ultra-thin input / output integrated information processing apparatus disclosed in Patent Document 6 has a problem in that a pen tip having the same color as the color of the display surface may not be detected as an image.

  For example, when the color of the pen tip is red, the pen tip absorbs red light, so that the red light is not reflected from the pen tip. Therefore, when the unit pixel around the contact portion of the pen tip is colored red, it is impossible to determine whether the pen has touched.

  From the viewpoint of improving the accuracy of position detection, three colors satisfying the principle of color matching such as R, G, and B, as in the ultra-thin input / output integrated information processing apparatus disclosed in Patent Document 6, are set to 1 Rather than performing the position detection by analyzing one piece of image data obtained by picking up the color every time and imaging the state in which these three colors are mixed, the three colors satisfying the same color principle such as R, G and B, that is, Prepare three sets of image data that were picked up using each image sensor each time the first, second, and third colors were developed in sequence, and each time the first, second, and third colors were developed. Thus, it is preferable to analyze these three imaging data.

  This is because the first color, the second color, and the second color are compared with the case where the position detection is performed by analyzing one imaged data obtained by imaging the state in which the first color, the second color, and the third color satisfying the principle of equal colors are mixed. This is because it is much easier to analyze and the detection accuracy is higher when the position detection is performed by analyzing the three imaging data in a state where the three colors are separated from each other.

  Therefore, the inventor of the present application sequentially irradiates the contact object with light of the first color, the second color, and the third color that satisfy the principle of color matching, and obtains three pieces of imaging data by reflected light of each light. Furthermore, if the color of the contact portion of the contact object or the contact position of the contact object is detected based on these three imaging data, the ultra-thin input / output integrated information disclosed in the above-mentioned Patent Document 6 is obtained. The inventor found that the problems of the processing apparatus could be solved.

  Therefore, in the configuration or method, based on such knowledge, in the color development control unit or the color development control step, each of the unit pixels is a first color, a second color, and a third color that satisfy the principle of color matching, Color is developed sequentially.

  Further, in the imaging control means or the imaging control step, each time each of the unit pixels is colored with the first color, the second color, and the third color in the color development control means or in the color development control step, the surface An image of a contact object that comes into contact with the member is picked up using each of the image sensors.

  Further, according to the configuration or the method, in the position specifying unit or the position specifying step, the image is picked up every time the first color, the second color, and the third color are developed by the imaging control unit or in the imaging control step. Based on each of the images of the contact object, the contact position of the contact object on the planar member is specified.

  Here, the “equal color principle” is a scientific term in chromatics, and is the principle that all colors can be expressed in three colors.

  In order to detect the color of the contact object, light having three different colors may be irradiated to the contact object and the reflected light may be detected based on the principle of color matching.

  Furthermore, if the position where the color of the contact object is detected is specified, the contact position of the contact object can be specified.

  Therefore, if the first color, the second color, and the third color are used to identify the position where the color of the contact object is detected, the color of the contact portion of the contact object is the same as the first color. Even in the case of a color, it is possible to specify the contact position based on image data obtained by imaging the remaining second and third colors.

  In other words, the contact position can be specified regardless of the color of the contact object that contacts the display surface of the planar member.

  Note that “coloring with the first color, the second color, and the third color” means that the first color is developed first, then the second color is developed, and then the third color is developed. .

  As mentioned above, the contact position can be specified irrespective of the color of the contact object which contacts the display surface of the input-output integrated display (planar member) useful for size reduction of an apparatus.

  Here, another problem of the ultra-thin input / output integrated information processing apparatus described in Patent Document 6 will be described with reference to FIGS. 15 (a) and 15 (b).

  FIGS. 15A and 15B are conceptual diagrams for explaining the relationship in size between the image sensor and the color filter when a color filter is provided between the light source and the image sensor. The sensor is placed under the color filter).

  Hereinafter, a case where three subpixels exist in one pixel pixel will be described.

  FIG. 15A conceptually shows the size relationship between the image sensor and the color filter in the ultra-thin input / output integrated information processing apparatus disclosed in Patent Document 6, and includes three image sensors. A case where three primary color filters of R, G, and B are provided in each subpixel is shown.

  As described above, in the method of providing an image sensor for each sub-pixel, fine processing is required and there is a problem in cost, and the area of the image sensor for each color of R, G, and B is reduced and sensitivity is lowered. There is a problem that it will.

  On the other hand, for example, FIG. 15B conceptually shows a case where the size of each of the three primary color filters of R, G, and B is the same as the size of one pixel, and an image sensor is provided for each pixel. .

  In this case, light for three subpixels can be used and the size of the image sensor can be made relatively large, so that the sensitivity can be improved by about three times, but the resolution is reduced to 1/3. There are also problems.

  In order to solve such a problem, in the position specifying device of the present invention, in addition to the above-described configuration, it is preferable that the number of unit pixels and the number of imaging sensors is one-to-one and the same number.

  From the viewpoint of detection accuracy of position detection, it is preferable to arrange an image sensor for each color pixel. However, according to the configuration, as described above, the detection accuracy of position detection is compared even if the number of image sensors is small. Therefore, it is possible to prevent a decrease in detection sensitivity while ensuring the current sensor area (number of unit pixels: number of imaging sensors = 1: 1) and position detection accuracy.

  Further, according to the above configuration, in order to increase the detection sensitivity, it is not necessary to develop one color using all the light emitted from the three color pixels in the unit pixel, so that the resolution is deteriorated. Can also be prevented.

  According to the above configuration and method, the sensitivity is reduced and the resolution is ensured while ensuring the size of the sensor area necessary for detecting the position of the contact object on the input / output integrated display (planar member) useful for downsizing of the apparatus. Can be prevented.

  Here, based on FIG. 16C, another problem of the configuration for developing the first color, the second color, and the third color will be described.

  In the case of the color scanners disclosed in Patent Documents 8 and 9 described above, image data of a document is captured with the document or the like placed on the scan screen, so that R, G, and B light is emitted on the scan screen. Even if it is sequentially irradiated, the light is hidden by the original document and is hardly visible to the user.

  However, in the case of detecting the contact position of the contact object that is in contact with the display surface of the input / output integrated display (planar member), the user is viewing the state in which the image is displayed on the display surface. In view of detecting the contact position of the contact object, when the first color, the second color, and the third color are sequentially developed, another problem arises that the display flickers.

  Therefore, in order to solve such a problem, the position specifying device according to the present invention, in addition to the above-described configuration, performs the third color after the coloring control unit starts the operation of coloring the first color. It is preferable to provide period control means for providing a color development pause period between the color development operation period until the color development operation is completed and the next color development operation period.

  According to the above configuration, since the period control means provides the color development pause period between the color development operation period and the next color development operation period, the display of the planar member is less flicker than when no color development pause period is provided. Can be reduced.

  The “coloring pause period” may be a period in which each unit pixel is not colored at all, or may be a period in which normal image display for displaying an application image or the like is performed.

  Further, the “coloring suspension period” is preferably a time longer than the time that the flickering of the display on the planar member is not noticed by the user, thereby reducing the flickering of the display of the planar member. it can.

  Furthermore, it is preferable that the “coloring pause period” is not longer than a time that the specific delay of the contact position by the position specifying unit is not noticed by the user, whereby the position specification is performed at an appropriate timing. Can do.

  In the position specifying device of the present invention, in addition to the above-described configuration, it is preferable that the period control unit adjusts the period between the coloring operation period and the coloring suspension period.

  According to the above configuration, the period control means adjusts the period between the color development operation period and the color development pause period.

  Therefore, the length of each of the coloring operation period and the coloring suspension period can be increased or decreased.

  In addition to the above configuration, the position specifying device of the present invention further includes contact determination means for determining whether or not the contact object is in contact with the planar member, and the period control means includes the contact When the determination unit determines that the contact object is in contact with the planar member, it is preferable to shorten the length of the color development suspension period.

  According to the said structure, a contact determination means determines whether the said contact thing is contacting the said planar member.

  The period control means shortens the length of the color development suspension period when the contact determination means determines that the contact object is in contact with the planar member.

  Therefore, when the contact object is in contact, the frequency of the coloring operation is increased, so that the accuracy of specifying the contact position can be increased.

  In addition, since the frequency of the coloring operation is increased only when the contact object is in contact, the power consumption can be reduced.

  Further, in the position specifying device of the present invention, in addition to the above configuration, the period control unit is configured to perform the color development pause period when the contact determination unit determines that the contact object is not in contact with the planar member. It is preferable to lengthen the length.

  According to the configuration, the period control unit increases the length of the color development suspension period when the contact determination unit determines that the contact object is not in contact with the planar member. Yes.

  Therefore, when the contact object is not in contact, the frequency of the coloring operation is lowered, so that the useless coloring operation can be reduced and the power consumption can be suppressed.

  In addition to the above configuration, the position specifying device of the present invention may further include the position specifying unit when the second contact position is specified after the first contact position is specified by the position specifying unit. Interpolation line specifying means for specifying the specified interpolation line for interpolating between the first contact position and the second contact position is provided, the first contact position, the second contact position, and the It is preferable that a position specifying result display control means for displaying the interpolation line is provided.

  According to the above configuration, the interpolation line specifying means specifies the contact position specifying means when the second contact position is specified after the first contact position is specified by the contact position specifying means. An interpolation line for interpolating between the first contact position and the second contact position is specified.

  The position specifying result display control means displays the first contact position, the second contact position, and the interpolation line.

  Therefore, the first contact position and the second contact position can be smoothly connected and displayed on the display surface.

  Further, in the position specifying device of the present invention, in addition to the above configuration, the first color is one of four colors consisting of red, green, blue and white, and the second color is the four colors. Any one of the remaining three colors that are not the first color, and the third color is any one of the remaining two colors that are not the second color among the three colors. preferable.

  According to the above configuration, the present invention can be applied to an image display device that uses a group of R, G, and B pixels generally used in an image display device as a unit pixel.

  In order to color the unit pixel in white, all of the R pixel, G pixel, and B pixel may be colored.

  Each means, each function, each step, and each process in the position specifying device and the position specifying method may be realized by a computer. In this case, the computer is operated as each means, There is also provided a position specifying program for realizing the position specifying device and the position specifying method on a computer by realizing a function or causing a computer to execute each step and each process, and a computer-readable recording medium on which the position specifying program is recorded Falls within the scope of the present invention.

  As described above, the position specifying device of the present invention includes a planar member including a plurality of unit pixels and a plurality of imaging sensors provided for each unit pixel, and each of the unit pixels. Color development control means for sequentially producing colors in the first color, the second color, and the third color that satisfy the color principle, and the color development control means, each of the unit pixels is designated as the first color, the second color, and the third color. An imaging control unit that captures an image of a contact object that is in contact with the planar member using each imaging sensor each time the color is developed, and the first color, the second color, and the second color by the imaging control unit. And a position specifying means for specifying a contact position of the contact object on the planar member based on each of the images of the contact object picked up every time the three colors are developed.

  Further, as described above, the position specifying method of the present invention is a position specifying method using a planar member including a plurality of unit pixels and a plurality of imaging sensors provided for each unit pixel. Then, in each of the unit pixels, the first color, the second color, and the third color satisfying the principle of equal color are sequentially developed, and in the color development control step, each of the unit pixels is In each of the first color, the second color, and the third color, an imaging control step of imaging an image of a contact object that contacts the planar member using the imaging sensors, and the imaging control step, A position specifying step of specifying a contact position of the contact object on the planar member based on each of the images of the contact object captured each time the first color, the second color, and the third color are developed; It is a method that includes

  Therefore, it is possible to specify the contact position regardless of the color of the contact object that contacts the display surface of the input / output integrated display (planar member) useful for downsizing the device.

  An embodiment of the present invention will be described below with reference to FIGS.

[Embodiment 1]
(Overview of data display / sensor device and overview of its functions)
First, based on FIG. 1, an outline configuration and functions of a data display / sensor device (position specifying device) 100 according to an embodiment of the present invention will be described.

  FIG. 1 is a schematic diagram showing an overview configuration and a summary of functions of a data display / sensor device 100 according to an embodiment of the present invention.

  As shown in FIG. 1, the general configuration of the data display / sensor device 100 according to the present embodiment includes a housing 300A and a housing 300B that can be folded in two and coupled by a hinge H. A sensor built-in liquid crystal panel (planar member) 301A and a sensor built-in liquid crystal panel (planar member) 301B (hereinafter collectively referred to as a sensor built-in liquid crystal panel (planar shape) are respectively formed on two inner surfaces facing each other in a folded state of the housing 300B. Member) 301) are arranged so as to face each other.

  The planar member such as the sensor built-in liquid crystal panel 301 is not limited to the sensor built-in liquid crystal panel of the present embodiment. For example, as a display, in addition to a liquid crystal display, an electrophoretic display, a twist ball type In addition to displays, reflective displays using fine prism films, displays using light modulation elements such as digital mirror devices, organic EL (Light Emitting) light-emitting elements, inorganic EL light-emitting elements, LEDs (Light Emitting) Examples include displays using elements with variable emission luminance, such as diodes, field emission displays (FEDs), and plasma displays. These inputs and outputs are integrated with image sensors (image sensors, optical sensors). Where integrated displays were developed in the future Even so, it will be apparent from the following description that the present invention can be applied to these input / output integrated displays.

  As an outline of the functions of the data display / sensor device 100, a plurality of pixel circuits (unit pixels) 31 (see FIG. 8) included in each sensor built-in liquid crystal panel 301 and at least one for each pixel circuit 31 are provided. By driving a plurality of optical sensors (imaging sensors) 6 (see FIGS. 2A and 2B), the pen is in contact with one of the two liquid crystal panels 301A and 301B with a built-in sensor. (Contact) It is possible to detect (specify) the contact position of the pen tip of P.

  Here, the unit pixel is described in parentheses in the parenthesis of the pixel circuit 31. However, as shown in FIG. 2A, the “unit pixel” is strictly in the case of a liquid crystal display, for example. This is a set including at least the color filter 53r, the color filter 53g, the color filter 53b, the liquid crystal layer 52, and the backlight 307. In the following description, the pixel circuit 31 is referred to as a “unit pixel” for the sake of convenience. Will be described. Note that the part describing the pixel circuit 31 itself means the pixel circuit 31 itself. The same is true for the notation in parentheses of the R pixel circuit (color pixel, R pixel) 31r, the G pixel circuit (color pixel, G pixel) 31g, and the B pixel circuit (color pixel, B pixel) 31b.

  As described above, the sensor built-in liquid crystal panel 301 includes the plurality of pixel circuits 31 and the plurality of photosensors 6 provided at least one for each pixel circuit 31.

  Therefore, by operating the plurality of pixel circuits 31 of the sensor built-in liquid crystal panel 301, an image can be displayed on a predetermined display surface (hereinafter simply referred to as “display surface”) on the sensor built-in liquid crystal panel 301. is there.

  In addition, the plurality of pixel circuits 31 of the sensor built-in liquid crystal panel 301 use light generated when light is emitted (emits light) to shine light on the pen P on the display surface, and the reflected light is applied to the plurality of sensor built-in liquid crystal panels 301. The image of the pen P on the display surface can be scanned by scanning (imaging) with the optical sensor 6.

  Note that the numbers of the plurality of photosensors 6 and the plurality of pixel circuits 31 may be arbitrary as long as the image display function and the position detection function can be ensured.

  For example, the number of each of the plurality of photosensors 6 and the plurality of pixel circuits 31 may be the same number on a one-to-one basis, or on the one-to-two basis and the latter may be twice as many as the former.

  Further, if the sensor area required for position detection can be secured, the number of each of the plurality of photosensors 6 and the plurality of pixel circuits 31 is 1: 3, and the latter is three times the number of the former. There may be. By adopting such a configuration, in the case where the pixel circuit 31 has, for example, three color pixels, it is possible to secure independence for each color detected by the optical sensor 6 of light generated by each color pixel. That is, the optical sensor 6 may include at least one for each of the plurality of pixel circuits 31.

  Here, as described above, the method of providing an image sensor for each sub-pixel as in the input / output integrated information processing apparatus disclosed in Patent Document 6 requires fine processing and has a problem in cost. At the same time, there is a problem that the area of the image sensor for each color of R, G, and B is reduced and the sensitivity is lowered.

  On the other hand, when the size of each of the R, G, and B primary color filters is the same as the size of one pixel and an image sensor is provided for each pixel, light of three sub-pixels can be used, and the size of the image sensor Since the sensitivity can be made relatively large, the sensitivity can be improved by about three times, but there is also a problem that the resolution is reduced to 1/3.

  In order to solve such a problem, in the data display / sensor device 100 of this embodiment, the numbers of the pixel circuits 31 and the photosensors 6 are one-to-one (see FIG. 2B). It is explained as a thing.

  From the viewpoint of detection accuracy of position detection, it is preferable to arrange the photosensor 6 for each of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b. However, according to the above configuration, the number of photosensors 6 is small. However, since the detection accuracy of position detection is relatively high, the detection sensitivity is lowered while ensuring the current sensor area (number of pixel circuits 31: number of photosensors 6: 1) and position detection detection accuracy. Can be prevented.

  In addition, according to the above configuration, in order to increase the detection sensitivity, one color is obtained using all the light emitted from the three R pixel circuits 31r, G pixel circuits 31g, and B pixel circuits 31b in the pixel circuit 31. Since it is not necessary to cause color development, it is possible to prevent deterioration in resolution.

  According to the above configuration, the sensitivity is reduced and the resolution is deteriorated while securing the size of the sensor area of the optical sensor 6 necessary for detecting the position of the pen P on the sensor built-in liquid crystal panel 301 useful for downsizing the apparatus. Can be prevented.

  Each of the plurality of photosensors 6 and the plurality of pixel circuits 31 is preferably arranged in a plane in the surface direction of the sensor built-in liquid crystal panel 301 (in-plane direction of the display image). For example, a matrix-like arrangement can be exemplified.

  The positional relationship between the plurality of photosensors 6 and the plurality of pixel circuits 31 is, for example, the case where the numbers are one-to-one and the same number, and the pixel circuit 31 includes three color pixels as described below. Is preferably close to any of the three color pixels (see, for example, FIG. 2B).

  Further, as will be described in detail later, the data display / sensor device 100 employs a configuration in which each pixel circuit 31 periodically repeats a coloring operation of sequentially developing colors in R, G, and B.

  Further, as a result, the light generated by the coloring operation immediately after the pen tip of the pen P comes into contact is irradiated onto the pen P, and the reflected light is scanned by the plurality of optical sensors 6, thereby changing the pen tip color of the pen P. Regardless, the position where the color of the pen tip is detected, that is, the contact position of the pen tip is detected.

  Further, the data display / sensor device 100 can follow the contact position of the pen P on the sensor built-in liquid crystal panel 301 by repeating this coloring operation, specify the contact position locus L, and draw the locus L. It has become.

  According to the above configuration, since the image display unit and the image capturing unit do not need to be configured as separate devices, the data display / sensor device 100 can be downsized.

  In addition, the color or contact position of the contact object in contact with the display surface of the sensor built-in liquid crystal panel 301 can be detected while preventing a decrease in sensitivity and a decrease in resolution.

  First, an outline of the sensor built-in liquid crystal panel 301 included in the data display / sensor device 100 will be described below.

(Outline of LCD panel with built-in sensor)
The sensor built-in liquid crystal panel 301 provided in the data display / sensor device 100 is a liquid crystal panel capable of detecting an image of a contact object in addition to displaying data. Here, the image detection of the contact object is, for example, detection of a position pointed (touched) by the user with a finger or a pen, or reading (scanning) of an image of a printed material. The device used for display is not limited to a liquid crystal panel, and may be an organic EL (Electro Luminescence) panel or the like.

  The structure of the sensor built-in liquid crystal panel 301 will be described with reference to FIGS. 2 (a) and 2 (b). FIG. 2A is a diagram schematically showing a cross section of the sensor built-in liquid crystal panel 301. The sensor built-in liquid crystal panel 301 described here is an example, and any structure can be used as long as the display surface and the reading surface are shared.

  As shown in the figure, the sensor built-in liquid crystal panel 301 includes an active matrix substrate 51A disposed on the back surface side and a counter substrate 51B disposed on the front surface side, and has a structure in which a liquid crystal layer 52 is sandwiched between these substrates. is doing. The active matrix substrate 51A is provided with a pixel electrode 56, a data signal line 57, an optical sensor circuit 32 (not shown), an alignment film 58, a polarizing plate 59, and the like. The counter substrate 51B is provided with color filters 53r (red), 53g (green), 53b (blue), a light shielding film 54, a counter electrode 55, an alignment film 58, a polarizing plate 59, and the like. In addition, a backlight 307 is provided on the back surface of the sensor built-in liquid crystal panel 301.

  The photosensor 6 included in the photosensor circuit 32 is provided in the vicinity of the pixel electrode 56 provided with the blue color filter 53b, but is not limited to this configuration. It may be provided in the vicinity of the pixel electrode 56 provided with the red color filter 53r, or may be provided in the vicinity of the pixel electrode 56 provided with the green color filter 53g.

  Next, another example of the arrangement method of the optical sensor 6 will be described with reference to FIG.

  FIG. 2B is a schematic diagram showing one pixel of another example of the sensor built-in liquid crystal panel 301 provided in the data display / sensor device 100.

  As shown in the figure, in this example, the optical sensor 6 is arranged in a straight line so that the optical sensor 6 is close to any of the color filter 53r, the color filter 53g, and the color filter 53b, and the color filter 53g and the color filter. It is arranged on the upper side with respect to the paper surface of 53b. By adopting such an arrangement, the size of the optical sensor 6 can be increased, and light can be detected with high sensitivity from any of the color filter 53r, the color filter 53g, and the color filter 53b.

  Next, with reference to FIGS. 3A and 3B, two types of methods for detecting the position where the user touches the sensor built-in liquid crystal panel 301 with a finger or a pen will be described.

  FIG. 3A is a schematic diagram showing how the position touched by the user is detected by detecting the reflected image. When the light 63 is emitted from the backlight 307, the optical sensor circuit 32 including the optical sensor 6 detects the light 63 reflected by the contact object 64 such as a finger. Thereby, the reflected image of the contact object 64 can be detected. Thus, the sensor built-in liquid crystal panel 301 can detect the touched position by detecting the reflected image.

  FIG. 3B is a schematic diagram showing how the position touched by the user is detected by detecting a shadow image. As shown in FIG. 3B, the optical sensor circuit 32 including the optical sensor 6 detects external light 61 transmitted through the counter substrate 51B and the like. However, when there is a contact object 62 such as a pen, since the incident of the external light 61 is hindered, the amount of light detected by the optical sensor circuit 32 is reduced. Thereby, the image of the contact object 62 can be detected. As described above, the sensor built-in liquid crystal panel 301 can also detect the touched position by detecting the shadow image.

  As described above, the optical sensor 6 may detect reflected light (shadow image) of the light emitted from the backlight 307 or may detect a shadow image caused by external light. Further, both the two types of detection methods may be used together to detect both a shadow image and a reflection image.

(Data display / sensor configuration)
Next, the configuration of the main part of the data display / sensor device 100 will be described with reference to FIG. FIG. 4 is a block diagram showing a main configuration of the data display / sensor device 100. As shown in FIG. As illustrated, the data display / sensor device 100 includes one or more display / light sensor units 300, a circuit control unit 600, a data processing unit 700, a main control unit 800, a storage unit 901, a primary storage unit 902, and an operation unit. 903, an external communication unit 907, an audio output unit 908, and an audio input unit 909. Here, although the data display / sensor device 100 is described as including a single display / light sensor unit 300, a plurality of data display / sensor devices 100 may be included.

  The display / light sensor unit 300 is a so-called liquid crystal display device with a built-in light sensor. The display / light sensor unit 300 includes a sensor built-in liquid crystal panel 301, a backlight 307, and a peripheral circuit 309 for driving them.

  The sensor built-in liquid crystal panel 301 includes a plurality of pixel circuits 31 and photosensor circuits 32 arranged in a matrix. The detailed configuration of the sensor built-in liquid crystal panel 301 will be described later.

  The peripheral circuit 309 includes a liquid crystal panel drive circuit 304, an optical sensor drive circuit 305, a signal conversion circuit 306, and a backlight drive circuit 308.

  The liquid crystal panel driving circuit 304 outputs a control signal (G) and a data signal (S) in accordance with the timing control signal (TC1) and the data signal (D) from the display control unit 601 of the circuit control unit 600, and the pixel circuit 31. It is a circuit which drives. Details of the driving method of the pixel circuit 31 will be described later.

  The optical sensor driving circuit 305 is a circuit that drives the optical sensor circuit 32 by applying a voltage to the signal line (R) in accordance with a timing control signal (TC2) from the sensor control unit 602 of the circuit control unit 600. Details of the driving method of the optical sensor circuit 32 will be described later.

  The signal conversion circuit 306 is a circuit that converts the sensor output signal (SS) output from the optical sensor circuit 32 into a digital signal (DS) and transmits the converted signal to the sensor control unit 602.

  The backlight 307 includes a plurality of white LEDs (Light Emitting Diodes) and is disposed on the back surface of the sensor built-in liquid crystal panel 301. When a power supply voltage is applied from the backlight drive circuit 308, the backlight 307 is turned on and irradiates the sensor built-in liquid crystal panel 301 with light. Note that the backlight 307 is not limited to white LEDs, and may include LEDs of other colors. The backlight 307 may include, for example, a cold cathode fluorescent lamp (CCFL) instead of the LED.

  When the control signal (BK) from the backlight control unit 603 of the circuit control unit 600 is at a high level, the backlight drive circuit 308 applies a power supply voltage to the backlight 307, and conversely, the backlight control unit When the control signal from 603 is at a low level, no power supply voltage is applied to the backlight 307.

  Next, the circuit control unit 600 will be described. The circuit control unit 600 has a function as a device driver that controls the peripheral circuit 309 of the display / light sensor unit 300. The circuit control unit 600 includes a display control unit 601, a sensor control unit 602, a backlight control unit 603, and a display data storage unit 604.

  The display control unit 601 receives display data from the display data processing unit 701 of the data processing unit 700, and performs timing control on the liquid crystal panel driving circuit 304 of the display / light sensor unit 300 in accordance with an instruction from the display data processing unit 701. A signal (TC1) and a data signal (D) are transmitted, and the received display data is displayed on the sensor built-in liquid crystal panel 301.

  The display control unit 601 temporarily stores the display data received from the display data processing unit 701 in the display data storage unit 604. Then, a data signal (D) is generated based on the primary stored display data. The display data storage unit 604 is, for example, a video random access memory (VRAM).

  The sensor control unit 602 transmits a timing control signal (TC2) to the optical sensor driving circuit 305 of the display / optical sensor unit 300 in accordance with an instruction from the sensor data processing unit 703 of the data processing unit 700, and the sensor built-in liquid crystal panel 301. Run the scan with.

  In addition, the sensor control unit 602 receives a digital signal (DS) from the signal conversion circuit 306. Then, image data is generated based on the digital signal (DS) corresponding to the sensor output signal (SS) output from all the optical sensor circuits 32 included in the sensor built-in liquid crystal panel 301. That is, the image data read in the entire reading area of the sensor built-in liquid crystal panel 301 is generated. Then, the generated image data is transmitted to the sensor data processing unit 703.

  The backlight control unit 603 transmits a control signal (BK) to the backlight drive circuit 308 of the display / light sensor unit 300 in accordance with instructions from the display data processing unit 701 and the sensor data processing unit 703 to drive the backlight 307. Let

  When the data display / sensor device 100 includes a plurality of display / light sensor units 300, the display control unit 601 determines which display / light sensor unit 300 displays the display data from the data processing unit 700. When an instruction is received, the liquid crystal panel drive circuit 304 of the display / light sensor unit 300 is controlled according to the instruction. When the sensor control unit 602 receives an instruction from the data processing unit 700 regarding which display / light sensor unit 300 is to scan the contact object, the sensor control unit 602 performs light of the display / light sensor unit 300 according to the instruction. The sensor drive circuit 305 is controlled and a digital signal (DS) is received from the signal conversion circuit 306 of the display / light sensor unit 300 according to the instruction.

  Next, the data processing unit 700 will be described. The data processing unit 700 has a function as middleware that gives an instruction to the circuit control unit 600 based on a “command” received from the main control unit 800. Details of the command will be described later.

  The data processing unit 700 includes a display data processing unit 701 and a sensor data processing unit 703. When the data processing unit 700 receives a command from the main control unit 800, at least one of the display data processing unit 701 and the sensor data processing unit 703 depends on the value of each field (described later) included in the received command. One works.

  The display data processing unit 701 receives display data from the main control unit 800, and gives instructions to the display control unit 601 and the backlight control unit 603 according to the command received by the data processing unit 700, and displays the received display data. The image is displayed on the sensor built-in liquid crystal panel 301. The operation of the display data processing unit 701 according to the command will be described later.

  The sensor data processing unit 703 gives an instruction to the sensor control unit 602 and the backlight control unit 603 according to the command received by the data processing unit 700.

  The sensor data processing unit 703 receives image data from the sensor control unit 602 and stores the image data in the image data buffer 704 as it is. Then, in accordance with the command received by the data processing unit 700, the sensor data processing unit 703 performs “whole image data”, “partial image data (partial image coordinate data) based on the image data stored in the image data buffer 704. At least one of “including coordinate data” and “coordinate data” is transmitted to the main control unit 800. The whole image data, partial image data, and coordinate data will be described later. The operation of the sensor data processing unit 703 according to the command will be described later.

  Next, the main control unit 800 executes an application program. The main control unit 800 reads the program stored in the storage unit 901 into a primary storage unit 902 configured by, for example, a RAM (Random Access Memory) and executes the program.

  An application program executed by the main control unit 800 causes the data processing unit 700 to display display data on the sensor built-in liquid crystal panel 301 and to scan a contact object on the sensor built-in liquid crystal panel 301. Command and display data. When “data type” is designated as a command, at least one of whole image data, partial image data, and coordinate data is received from the data processing unit 700 as a response to the command.

  The circuit control unit 600, the data processing unit 700, and the main control unit 800 can be configured by a CPU (Central Processing Unit), a memory, and the like, respectively. The data processing unit 700 may be configured by a circuit such as an ASIC (application specific integrate circuit).

  Next, the storage unit 901 stores programs and data executed by the main control unit 800 as shown in the figure. The program executed by the main control unit 800 may be separated into an application-specific program and a general-purpose program that can be shared by each application.

  Next, the operation unit 903 receives an input operation of the user of the data display / sensor device 100. The operation unit 903 includes input devices such as a switch, a remote controller, a mouse, and a keyboard, for example. Then, the operation unit 903 generates a control signal corresponding to the user's input operation of the data display / sensor device 100, and transmits the generated control signal to the main control unit 800.

  As an example of the switch, a hardware switch such as a power switch 905 that switches power on and off and a user switch 906 to which a predetermined function is assigned in advance is assumed.

  In addition, the data display / sensor device 100 includes an external communication unit 907 for communicating with an external device by wireless / wired communication, an audio output unit 908 such as a speaker for outputting audio, and a microphone for inputting an audio signal. A voice input unit 909 such as the above may be provided as appropriate.

(Command details)
Next, details of commands transmitted from the main control unit 800 to the data processing unit 700 will be described with reference to FIGS. 5 and 6. FIG. 5 is a diagram schematically illustrating an example of a command frame structure. FIG. 6 is a diagram for explaining an example of values that can be specified for each field included in the command and an outline thereof.

  As shown in FIG. 5, the commands are “header”, “data acquisition timing”, “data type”, “scan method”, “scan image gradation”, “scan resolution”, “scan panel”, “display panel”. "And" Reserve "fields. In each field, for example, values shown in FIG. 6 can be designated.

  The “header” field is a field indicating the start of a frame. As long as it is possible to identify the “header” field, the value of the “header” field may be any value.

  Next, the “data acquisition timing” field is a field for designating a timing at which data should be transmitted to the main control unit 800. In the “data acquisition timing” field, for example, values “00” (sense), “01” (event), and “10” (all) can be specified.

  Here, “sense” specifies that the latest data is to be transmitted immediately. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “sense”, the latest data specified in the “data type” field is immediately updated to the main control unit 800. Send to.

  The “event” designates transmission at a timing when a change occurs in the image data received from the sensor control unit 602. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “event”, the image that receives the data specified in the “data type” field from the sensor control unit 602. The data is transmitted to the main control unit 800 at a timing when a change larger than a predetermined threshold occurs.

  “All” designates data transmission at a predetermined cycle. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “all”, the data designated in the “data type” field is transferred to the main control unit 800 at a predetermined cycle. Send to. The predetermined period coincides with the period in which the optical sensor circuit 32 performs scanning.

  Next, the “data type” field is a field for designating the type of data acquired from the sensor data processing unit 703. In the “data type” field, for example, values of “001” (coordinates), “010” (partial image), and “100” (entire image) can be specified. Furthermore, by adding these values, “coordinates” and “partial image” / “whole image” can be specified simultaneously. For example, when “coordinate” and “partial image” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose value of the “data type” field is “whole image”, the sensor data processing unit 703 transmits the image data itself stored in the image data buffer 704 to the main control unit 800. The image data itself stored in the image data buffer 704 is referred to as “whole image data”.

  In addition, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 selects a portion where a change larger than a predetermined threshold has occurred from the image data received from the sensor control unit 602. A region to be included is extracted, and image data of the extracted region is transmitted to the main control unit 800. Here, the image data is referred to as “partial image data”. When a plurality of partial image data are extracted, the sensor data processing unit 703 transmits the extracted partial image data to the main control unit 800.

  Further, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 detects representative coordinates in the partial image data as the coordinates of the point to which the contact object approaches. Then, coordinate data indicating the position of the representative coordinate in the partial image data is transmitted to the main control unit 800. The representative coordinates include, for example, the coordinates of the center of the partial image data, the coordinates of the center of gravity of the partial image data, and the like.

  Next, when receiving a command whose value of the “data type” field is “coordinate”, the sensor data processing unit 703 transmits coordinate data indicating the position of the representative coordinate in the entire image data to the main control unit 800. When a plurality of partial image data are extracted, the sensor data processing unit 703 detects representative coordinates in the entire image data of the extracted partial image data, and the coordinate data indicating the representative coordinates is detected. Each is transmitted to the main control unit 800 (multi-point detection).

  Specific examples of the whole image data, the partial image data, and the coordinate data will be described later with reference to schematic diagrams.

  Next, the “scan method” field is a field for designating whether or not the backlight 307 is turned on at the time of executing the scan. In the “scan method” field, for example, values of “00” (reflection), “01” (transmission), and “10” (reflection / transmission) can be designated.

  “Reflection” specifies that scanning is performed with the backlight 307 turned on. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “reflection”, the sensor data processing unit 703 performs sensor control so that the optical sensor driving circuit 305 and the backlight driving circuit 308 operate in synchronization. An instruction is given to the unit 602 and the backlight control unit 603.

  “Transmission” specifies that scanning is performed with the backlight 307 turned off. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “transparent”, the sensor control unit 602 operates the optical sensor driving circuit 305 and does not operate the backlight driving circuit 308. Instructions to the backlight control unit 603. Note that “reflection / transmission” specifies that scanning is performed using both “reflection” and “transmission”.

  Next, the “scanned image gradation” field is a field for designating gradations of the partial image data and the entire image data. In the “scanned image gradation” field, for example, values of “00” (binary) and “01” (multivalue) can be designated.

  When the sensor data processing unit 703 receives a command whose “scan image gradation” field value is “binary”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as monochrome data. .

  When the sensor data processing unit 703 receives a command whose “scanned image gradation” field value is “multivalued”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as multitone data. To do.

  Next, the “scan resolution” field is a field for designating the resolution of the partial image data and the entire image data. In the “resolution” field, for example, values of “0” (high) and “1” (low) can be designated.

  Here, “high” designates a high resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “high”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 with high resolution. For example, it is desirable to designate “high” for image data (image data such as a fingerprint) to be subjected to image processing such as image recognition.

  “Low” designates a low resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “low”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 at a low resolution. For example, it is desirable to designate “low” for image data (such as touched finger or hand image data) that only needs to be recognized.

  Next, in the “scan panel” field, when the data display / sensor device 100 includes a plurality of display / light sensor units 300, which display / light sensor unit 300 performs scanning of the contact object. This field is specified. In the “scan panel” field, for example, values “001” (first display / light sensor unit 300) and “010” (second display / light sensor unit 300) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when both the first and second display / light sensor units 300 are specified at the same time, “011” can be specified.

  Here, the sensor data processing unit 703 controls the photo sensor driving circuit 305 and the backlight driving circuit 308 of the designated display / light sensor unit 300 according to the value of the “scan panel” field of the received command. An instruction is given to the sensor control unit 602 and the backlight control unit 603.

  Next, the “display panel” field specifies which display / light sensor unit 300 displays the display data when the data display / sensor device 100 includes a plurality of display / light sensor units 300. It is a field to be. In the “scanned image gradation” field, for example, values of “001” (first display / light sensor unit 300) and “010” (second display / light sensor unit 300) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when both the first and second display / light sensor units 300 are specified at the same time, “011” can be specified.

  Here, for example, when the display data processing unit 701 receives a command whose value of the “display panel” field is the display / light sensor unit 300, the display / light sensor unit 300 displays the display / light sensor unit 300 to display the display data. An instruction is given to the display control unit 601 and the backlight control unit 603 so as to control the liquid crystal panel driving circuit 304 and the backlight driving circuit 308 of the optical sensor unit 300.

  Next, the “reserved” field is a field that is appropriately specified when it is necessary to further specify information other than information that can be specified in the above-described fields.

  Note that an application executed by the main control unit 800 does not need to use all the above-described fields when transmitting a command, and an invalid value (such as a NULL value) may be set for a field that is not used. .

  When the user wants to acquire coordinate data of a position touched with a finger or pen, a command specifying “coordinate” in the “data type” field is transmitted to the data processing unit 700. Therefore, it is desirable to specify “all” in the “data acquisition timing” field of the command to acquire coordinate data. Further, since it is only necessary to acquire coordinate data of the touched position, the scanning accuracy may not be high. Therefore, “low” may be specified as the value of the “resolution” field of the command.

  Further, when “coordinate” is specified in the “data type” field of the command, for example, when the user touches the sensor built-in liquid crystal panel 301 with a plurality of fingers or pens at the same time, the coordinate data of the touched position is used. Can be acquired (multi-point detection).

  When acquiring image data of a contact object such as a document, a command specifying “whole image” in the “data type” field is transmitted to the data processing unit 700. Since it is common to perform a scan in a stationary state, it is not necessary to periodically perform the scan. Therefore, in this case, it is desirable to designate “sense” or “event” in the “data acquisition timing” field. When scanning a contact object such as a document, it is desirable that the scanning accuracy be high so that the user can easily read the characters. Therefore, it is desirable to designate “high” in the “resolution” field.

(Whole image data / Partial image data / Coordinate data)
Next, the whole image data, the partial image data, and the coordinate data will be described with reference to FIGS. 7A and 7B. The image data shown in FIG. 7A is image data obtained as a result of scanning the entire sensor built-in liquid crystal panel 301 when no contact object is placed on the sensor built-in liquid crystal panel 301. The image data shown in FIG. 7B is image data obtained as a result of scanning the entire sensor-equipped liquid crystal panel 301 when the user touches the sensor-equipped liquid crystal panel 301 with a finger.

  When the user touches the sensor built-in liquid crystal panel 301 with a finger, the amount of light received by the photosensor circuit 32 in the vicinity of the touch changes, so that the voltage output from the photosensor circuit 32 changes, and as a result, In the image data generated by the sensor control unit 602, the brightness of the pixel value of the portion touched by the user changes.

  In the image data shown in FIG. 7B, the brightness of the pixel value of the portion corresponding to the user's finger is higher than that in the image data shown in FIG. In the image data shown in FIG. 7B, the smallest rectangular area (area PP) that includes all pixel values whose lightness changes more than a predetermined threshold is “partial image data”.

  The image data indicated by the area AP is “whole image data”.

  Also, the coordinate data in the whole image data (area AP) of the representative coordinates Z of the partial image data (area PP) is (Xa, Ya), and the coordinate data in the partial image data (area PP) is (Xp, Yp). It is.

(Configuration of sensor built-in liquid crystal panel)
Next, the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309 of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 8 is a block diagram showing the main part of the display / light sensor unit 300, particularly the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309.

  The sensor built-in liquid crystal panel 301 includes a pixel circuit 31 for setting light transmittance (brightness) and an optical sensor circuit 32 that outputs a voltage corresponding to the intensity of light received by the sensor. The pixel circuit 31 is used as a general term for the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b corresponding to the red, green, and blue color filters, respectively.

  The pixel circuits 31 are arranged on the sensor built-in liquid crystal panel 301 in the column direction (vertical direction) and 3n in the row direction (horizontal direction). A set of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b is continuously arranged in the row direction (lateral direction). This set forms one pixel.

  In order to set the light transmittance of the pixel circuit 31, first, the high level voltage (TFT 33 is turned on) to the scanning signal line Gi connected to the gate terminal of the TFT (Thin Film Transistor) 33 included in the pixel circuit 31. Voltage). Thereafter, a predetermined voltage is applied to the data signal line SRj connected to the source terminal of the TFT 33 of the R pixel circuit 31r. Similarly, the light transmittance is also set for the G pixel circuit 31g and the B pixel circuit 31b. Then, by setting these light transmittances, an image is displayed on the sensor built-in liquid crystal panel 301.

  Next, the photosensor circuit 32 is arranged for each pixel. One pixel may be arranged in the vicinity of each of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b.

  In order for the optical sensor circuit 32 to output a voltage corresponding to the light intensity, first, the sensor readout line RWi connected to one electrode of the capacitor 35 and the anode terminal of the photodiode 36 are connected. A predetermined voltage is applied to the sensor reset line RSi. In this state, when light is incident on the photodiode 36, a current corresponding to the amount of incident light flows through the photodiode 36. Then, according to the current, the voltage at the connection point (hereinafter referred to as connection node V) between the other electrode of the capacitor 35 and the cathode terminal of the photodiode 36 decreases. When the power supply voltage VDD is applied to the voltage application line SDj connected to the drain terminal of the sensor preamplifier 37, the voltage at the connection node V is amplified and output from the source terminal of the sensor preamplifier 37 to the sensing data output line SPj. Based on the output voltage, the amount of light received by the optical sensor circuit 32 can be calculated.

  Next, the liquid crystal panel drive circuit 304, the optical sensor drive circuit 305, and the sensor output amplifier 44, which are peripheral circuits of the sensor built-in liquid crystal panel 301, will be described.

  The liquid crystal panel drive circuit 304 is a circuit for driving the pixel circuit 31, and includes a scanning signal line drive circuit 3041 and a data signal line drive circuit 3042.

  The scanning signal line driving circuit 3041 sequentially selects one scanning signal line from the scanning signal lines G1 to Gm for each line time based on the timing control signal TC1 received from the display control unit 601, and A high level voltage is applied to the selected scanning signal line, and a low level voltage is applied to the other scanning signal lines.

  Based on the display data D (DR, DG, and DB) received from the display controller 601, the data signal line driver circuit 3042 generates a predetermined voltage corresponding to the display data for one row for each line time. The data signal lines SR1 to SRn, SG1 to SGn, and SB1 to SBn are applied (line sequential method). Note that the data signal line driver circuit 3042 may be driven by a dot sequential method.

  The optical sensor driving circuit 305 is a circuit for driving the optical sensor circuit 32. Based on the timing control signal TC2 received from the sensor control unit 602, the optical sensor driving circuit 305 selects a predetermined sensor readout signal line from the sensor readout signal lines RW1 to RWm for each line time. A read voltage is applied, and a voltage other than a predetermined read voltage is applied to the other sensor read signal lines. Similarly, based on the timing control signal TC2, a predetermined reset voltage is applied to the sensor reset signal line selected from the sensor reset signal lines RS1 to RSm for each line time, and the others. A voltage other than a predetermined reset voltage is applied to the sensor reset signal line.

  The sensing data output signal lines SP1 to SPn are grouped into p groups (p is an integer of 1 to n), and the sensing data output signal lines belonging to each group are connected via a switch 47 that is sequentially turned on in time division. And connected to the sensor output amplifier 44. The sensor output amplifier 44 amplifies the voltage from the group of sensing data output signal lines connected by the switch 47 and outputs the amplified voltage to the signal conversion circuit 306 as sensor output signals SS (SS1 to SSp).

(Configuration of main control unit and data stored in storage unit)
Next, the configuration of the main control unit 800 and various data stored in the storage unit 901 will be described with reference to FIG.

  FIG. 9 is a block diagram showing details of the main control unit 800 and the storage unit 901 in the main configuration of the data display / sensor device 100.

  As illustrated, the main control unit 800 includes a scan processing unit (imaging control unit) 12, a contact position determination processing unit (contact position specifying unit) 14A, and an image display processing unit (coloring control unit, position specifying result display control unit). 18 is included.

  The storage unit 901 includes three storage areas: a scan attribute information storage unit 22, an image data storage unit 23, and a display attribute information storage unit 24.

  The scan attribute information storage unit 22 includes two storage areas: a contact position specifying command storage unit 222 and a position specifying result display command storage unit 223.

  The contact position specifying command storage unit 222 stores values of fields of “contact position specifying command” described below in advance.

  The position specifying result display command storage unit 223 stores in advance values of each field of a “position specifying result display command” described below.

  On the other hand, the image data storage unit 23 includes two storage areas of a contact position specifying data storage unit 231 and a contact position specifying result storage unit 232.

  The contact position specifying data storage unit 231 is configured such that the image display processing unit 18 described below causes the first color, the second color, and the third color satisfying the same color principle to be colored in order, and the scan processing unit 12 This is a storage area for storing scan data scanned using each optical sensor 6 every time the first color, the second color, and the third color are developed.

  The contact position specifying result storage unit 232 is a storage area for storing detection results such as the contact position specified by the contact position determination processing unit 14A.

  Further, the display attribute information storage unit 24 includes two recording areas of a primary color image creation data storage unit 241 and an image display timing storage unit 242.

  The primary color image creation data storage unit 241 stores in advance R image display data, G image display data, and B image display data described below.

  In addition, the image display timing storage unit 242 adjusts the start timing and end timing of the coloring operation period and the coloring suspension period described below, and adjusts the length and frequency of each period. Data is stored in advance.

  As shown in FIG. 9, the image display processing unit 18 includes a primary color image creation unit (color development control unit) 181, a contact position identification result display image creation unit (position identification result display control unit) 182, and a normal image creation unit 183. It is out.

  Incidentally, as described above, the ultra-thin input / output integrated information processing apparatus disclosed in Patent Document 6 has a problem in that a pen tip having the same color as the color of the display surface may not be detected as an image.

  In addition, with a conventional color scanner, when the original is not placed or the display surface is exposed to air, scanning results in black image data, but in order to avoid such image data, It is conceivable to provide a diffusion plate on the display surface of the sensor built-in liquid crystal panel 301.

  By providing this diffusing plate, it is possible to convert the black image data into pure white image data.

  However, when this diffusion plate is provided on the display surface of the sensor built-in liquid crystal panel 301, the reflected light generated when the light emitted from the backlight is irradiated onto the diffusion plate is pure white.

  Therefore, for example, when scanning is performed by turning on all light of R, G, and B by providing a diffusion plate on a liquid crystal panel such as the ultra-thin input / output integrated information processing apparatus disclosed in Patent Document 6, When the color of the pen tip is white, the contact position is not detected at all.

  From the viewpoint of improving the accuracy of position detection, the first color, the second color, and the third color are developed at a time, and one scan data obtained by scanning the mixed state of these three colors is analyzed. Rather than performing position detection, the first, second, and third colors that satisfy the same color principle are developed, and each time the first, second, and third colors are developed. It is preferable to prepare three pieces of scan data scanned using each optical sensor 6 and analyze these three pieces of scan data.

  This is because the first color, the second color, and the second color are detected rather than performing the position detection by analyzing one scan data obtained by scanning the mixed state of the first color, the second color, and the third color that satisfy the principle of equal colors. This is because it is much easier to analyze and the detection accuracy is higher if the position detection is performed by analyzing the three scan data in a state where the three colors are separated from each other.

  Therefore, the inventor of the present application sequentially irradiates the pen P with the light of the first color, the second color, and the third color that satisfy the principle of equal colors, and obtains three scan data by reflected light of each light. Further, if the color of the contact portion of the pen P or the contact position of the pen P is detected based on the three scan data, the ultra-thin input / output integrated information disclosed in the above-mentioned Patent Document 6 is used. The present inventors have found out that the problem that the processing apparatus has and the problem when the diffusion plate is provided can be solved.

  Therefore, based on such knowledge, the primary color image creation unit (color generation control unit) 181 transmits the primary color image data to the display / light sensor unit 300 via the data processing unit 700 and the circuit control unit 600, and the sensor built-in liquid crystal. Each of the pixel circuits 31 included in the panel 301 is sequentially developed with a first color, a second color, and a third color that satisfy the principle of equal colors.

  Specifically, for example, when the first color is developed, the entire primary color image data created by the primary color image creation unit 181 may be data for only the first color. Further, when the second color is developed, the entire primary color image data may be data of only the second color, and when the third color is developed, the entire primary color image data may be the data of only the third color.

  Note that “coloring with the first color, the second color, and the third color” means that the first color is developed first, then the second color is developed, and then the third color is developed. .

  The “equal color principle” is a scientific term in chromatics, and is the principle that all colors can be expressed in three colors.

  Further, specific examples of color schemes that satisfy the principle of the equal color principle include three primary colors of colors, three primary colors of light, and three colors selected along every ring on a 12-color circle ( Secondary color etc.).

  In order to detect the color of the contact object, it is only necessary to irradiate the contact object with light having three different colors and detect the reflected light based on the principle of color matching.

  Moreover, it is preferable that the switching speed from the specific color of the first color, the second color, and the third color to another color is as high as possible from the viewpoint of detection sensitivity.

  Further, as a configuration for causing the pixel circuit 31 to develop the first color, the second color, and the third color, for example, a collection of color pixels that can develop each of the first color, the second color, and the third color is a pixel. The circuit 31 may be used.

  Further, as the color pixels, as in the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b of this embodiment, a combination of three primary color filters and liquid crystal is used, and light from the backlight 307 is transmitted to the three primary color filters. For example, a color pixel that emits (colors) light without using a backlight, such as a color pixel using an LED element. It may be.

  In addition, as an example of the first color, the second color, and the third color that satisfy the principle of equal colors, the first color is selected from the three colors of red (R), green (G), and blue (B). When the second color is any one of the three colors that are not the first color and the third color is the remaining one of the two colors that is not the second color Can be illustrated.

  As another example of the first color, the second color, and the third color that satisfy the principle of equal colors, the first color is any one of red, green, and blue, and three colors that are white. The second color is any one of the three colors that are not the first color, and the third color is the remaining one of the two colors that is not the second color. The case can be illustrated.

  Thus, an image display in which a pixel circuit 31 is a group of R pixels (R pixel circuit 31r), G pixels (G pixel circuit 31g), and B pixels (B pixel circuit 31b) that are generally used in an image display device. The present invention can be applied to an apparatus.

  In order to color the pixel circuit 31 in white, all of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b may be colored. Specifically, the entire primary color image data created by the primary color image creation unit 181 may be white data.

  Hereinafter, the case where the data display / sensor device 100 sequentially colors the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b will be described. The R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b, respectively. The primary color image data for developing the color is referred to as “R image display data”, “G image display data”, and “B image display data”.

  The normal image creating unit 183 creates “normal image display data” for performing normal image display based on application data such as moving images.

  Further, the contact position specifying result display image creating unit 182 follows the contact position of the pen P touching the sensor built-in liquid crystal panel 301, and displays the “contact position specifying result display” for displaying the locus L due to the change of the contact position. Image ".

  In the case of the color scanners disclosed in Patent Documents 8 and 9 described above, image data of a document is captured with the document or the like placed on the scan screen, so that R, G, and B light is emitted on the scan screen. Even if it is sequentially irradiated, the light is hidden by the original document and is hardly visible to the user.

  However, as in the data display / sensor device 100 of the present embodiment, when the contact position of the pen tip of the pen P that is in contact with the display surface of the sensor built-in liquid crystal panel 301 is detected, an image is displayed on the display surface. In order to detect the contact position of the pen tip of the pen P while the user is looking at the current state, flickering occurs when the first color, the second color, and the third color are sequentially developed. Another problem arises.

  Therefore, in order to solve such a problem, the image display timing control unit 184 determines that the “B image display” is performed after the primary color image creation unit (period control unit) 181 starts creating “R image display data”. The color development suspension period is provided between the period until the transmission of “data” is completed (color development operation period) and the next color development operation period.

  According to the above configuration, the image display timing control unit 184 provides the color development pause period between the color development operation period and the next color development operation period, so that the sensor built-in liquid crystal panel 301 is compared with the case where no color development pause period is provided. The display flicker of the display surface can be reduced.

  The “coloring pause period” may be a period in which each pixel circuit 31 is not colored at all, or may be a period in which normal image display for displaying an application image or the like is performed.

  In the present embodiment, the normal image creation unit 183 creates application image data or the like (normal image display data), and transmits the normal image display data to the display / light sensor unit 300 via the data processing unit 700. A period during which a normal image display operation is performed will be described as a “coloring suspension period”.

  That is, in the data display / sensor device 100, the primary color image creation unit 181 and the normal image creation unit 183 periodically set the pixel circuit 31 to the first color, the second color, and the second color under the control of the image display timing control unit 184. A series of operations (coloring operation) for sequentially developing colors in three colors and a coloring pause period in which the coloring operation is stopped and a normal application image is displayed are alternately repeated.

  Further, the image display timing control unit 184 adjusts the period between the coloring operation period and the coloring suspension period.

  As a result, the lengths of the coloring operation period and the coloring suspension period can be lengthened or shortened.

  Note that the color development suspension period is preferably set to a time longer than the time when the flickering of the display on the sensor built-in liquid crystal panel 301 is not noticed by the user. Thereby, the flicker of the display of a planar member can be reduced.

  Further, it is preferable that the color development suspension period is set to a time that is not long enough for the user to be concerned about the specific delay of the contact position of the pen P by the contact position specifying unit (position specifying means) 141. Thereby, position specification can be performed at an appropriate timing.

  Next, the scan processing unit 12 includes a scan execution unit (imaging control unit) 131, a scan data holding unit 134, and a scan attribute information setting unit 135. The scan execution unit 131 further includes a scan execution instruction unit (imaging unit). Control means) 132 and a scan data storage unit (imaging control means) 133.

  The scan processing unit (imaging control means) 12 contacts the sensor built-in liquid crystal panel 301 each time the image display processing unit 18 causes each of the pixel circuits 3 to develop the first color, the second color, and the third color. The image of the pen P to be scanned is scanned using each optical sensor 6.

  The scan execution instructing unit 132 transmits various commands to the data processing unit 700, and the main control unit 800 for scan data (for example, whole image data, partial image data, representative coordinates, etc.) corresponding to the scan attribute included in the command. To send to.

  The scan data storage unit 133 performs control for storing the scan data transmitted to the main control unit 800 in accordance with an instruction from the scan execution instruction unit 132 in the scan data holding unit 134 or the image data storage unit 23. .

  The scan data holding unit 134 temporarily stores scan data. The image data storage unit 23 may temporarily store the scan data or may store it constantly.

  The scan attribute information setting unit 135 reads various command data recorded in the scan attribute information storage unit 22 and sets the scan attribute of the data display / sensor device 100.

  Next, the contact position determination processing unit (contact position specifying unit, contact determination unit) 14A includes a contact position specifying unit (contact position specifying unit, contact determination unit) 141, a contact determination unit (contact determination unit) 142, and an interpolation processing unit. (Interpolation line specifying means) 143 and a contact position specifying result storage unit 232 are included.

  The contact position specifying unit 141 uses the pen P based on each of the images of the pen P scanned by the optical sensor 6 every time the primary color image creating unit 181 develops the first color, the second color, and the third color. The position (contact position) where the pen tip color of the pen P is detected on the sensor built-in liquid crystal panel 301 is specified (detected). That is, if the position where the pen tip color of the pen P is detected is specified, this position can be specified as the contact position of the pen P.

  Therefore, if the first color, the second color, and the third color are developed to specify the position where the pen tip color of the pen P is detected, the pen tip color of the pen P is assumed to be the first color. Even in the case of the same color as the color, it is possible to specify the contact position based on the scan data obtained by scanning the remaining second and third colors.

  In other words, the contact position can be specified regardless of the color of the pen P (or the pen tip) that contacts the display surface of the sensor built-in liquid crystal panel 301.

  As mentioned above, the contact position can be specified irrespective of the color of the contact thing which contacts on the display surface of the liquid crystal panel 301 with a sensor useful for size reduction of an apparatus.

  The contact determination unit (contact determination means) 142 determines whether or not the pen tip of the pen P is in contact with the sensor built-in liquid crystal panel 301.

  Here, the determination as to whether or not the pen tip of the pen P is in contact with the sensor built-in liquid crystal panel 301 is made, for example, by determining that the contact determination unit 142 has detected the contact position of the pen P. As a trigger, it may be determined that the pen P has touched the sensor built-in liquid crystal panel 301.

  Conversely, the contact determination unit 142 triggers that the contact position specifying unit 141 does not detect the contact position of the pen P at all for a predetermined time as a trigger (the pen up). What is necessary is just to judge.

  In FIG. 9, the contact position specifying unit 141 and the contact determination unit 142 are separate functional blocks. However, the contact position specifying unit 141 may be configured to have both functions of the contact determination unit 142.

  The image display timing control unit 184 shortens the length of the color development suspension period when the contact determination unit 142 (or the contact determination unit 142) determines that the pen P is in contact with the sensor built-in liquid crystal panel 301. It has become.

  Thereby, when the pen P is in contact, the frequency of the coloring operation is increased, so that the accuracy of specifying the contact position can be increased.

  Further, since the frequency of the coloring operation is increased only when the pen P is in contact, the power consumption can be reduced.

  In addition, the image display timing control unit 184 increases the length of the color suspension period when the contact determination unit 142 (or the contact determination unit 142 determines that the pen P is not in contact with the sensor built-in liquid crystal panel 301). It has become.

  As a result, when the pen P is not in contact, the frequency of the coloring operation is lowered, so that the useless coloring operation can be reduced and the power consumption can be suppressed.

  Further, the interpolation processing unit (interpolation line specifying unit) 143 is configured so that when the second contact position is specified after the contact position specifying unit 141 specifies the first contact position, the contact position specifying unit 141 The specified interpolation line for interpolating between the first contact position and the second contact position is specified.

  Further, the contact position specifying result display image creating unit 182 creates a contact position specifying result display image for displaying the first contact position, the second contact position, and the interpolation line. .

  Accordingly, the first contact position and the second contact position can be smoothly connected and displayed on the display surface.

  Although details are not described in the present embodiment, as interpolation methods applicable to the present invention, zero-order interpolation (nearest neighbor interpolation, nearest neighbor interpolation), linear interpolation (linear interpolation, first-order interpolation), and cubic interpolation ( For example, cubic interpolation).

  Further, when three or more contact positions are specified, an appropriate interpolation function may be interpolated. For example, as an example of an interpolation method using an interpolation function, Lagrange interpolation, spline interpolation, interpolation using a Sinc function, Lanczos interpolation (Lanzosh interpolation) and the like can be exemplified.

(Relationship between various commands and data display / sensor device operation)
Here, the relationship between various commands in the data display / sensor device 100 and the operation of the data display / sensor device 100 will be described.

  In the present embodiment, the name of the command related to the contact position specifying process of the data display / sensor device 100 is referred to as a “contact position specifying command” and will be described as being stored in the contact position specifying command storage unit 222 in advance.

  Further, the name of the command related to the contact position specifying result display processing of the data display / sensor device 100 is referred to as “position specifying result display command”, and is stored in the position specifying result display command storage unit 223 in advance.

  The “contact position specifying command” is a command for mainly setting the scan attribute of the scan processing unit 12 for the contact position specifying process.

  The “position specifying result display command” is a command for mainly setting the scan attribute of the scan processing unit 12 for the touch position specifying result display process.

  First, in the “contact position specifying command” and “position specifying result display command”, the value of the “data acquisition timing” field shown in FIG. 6 is designated as “01” (event).

  This is because, as described above, the “event” specifies that the data processing unit 700 transmits the scan data to the main control unit 800 at the timing when the image data received from the sensor control unit 602 has changed. Therefore, the scan data is sent to the main control unit 800 at the timing when the change occurs in the image data regardless of the above-described coloring operation period and coloring suspension period.

  Thereby, it is possible to identify the contact position regardless of the color of the contact object during the coloring operation period. In the color development pause period, scan data may not be transmitted to the main control unit 800 if the color of the contact object and the color around the contact portion of the display surface happen to be the same color. In other cases, scan data is transmitted to the main control unit 800.

  For example, in the coloring operation period, when the color of the contact object is the same as the first color, every time the contact object contacts the sensor built-in liquid crystal panel 301, the color is developed with at least the second color and the third color. Since the image data of the captured scan data should change, the scan data is transmitted to the main control unit 800 at this timing.

  The contact position of the contact object can be specified from the partial image data or the representative coordinates included in the scan data at this time.

  Also, even during the color development suspension period, if the scan data is transmitted to the main controller 800, the contact position of the contact object can be specified.

  Further, as described above, the image display timing control unit 184 adjusts the period and so on, and controls the image display timing so that the coloring operation period and the coloring suspension period are alternately repeated. Therefore, even if the contact position of the contact object cannot be specified during the color development pause period, the contact position of the contact object can be detected in the color development operation period immediately after that.

  As mentioned above, the contact position can be specified irrespective of the color of the contact thing which contacts on the display surface of the liquid crystal panel 301 with a sensor useful for size reduction of an apparatus.

  Next, in both “contact position specifying command” and “position specifying result display command”, at least “010” (partial image) is specified as the value of the “data type” field.

  This is because, as described above, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the change from the image data received from the sensor control unit 602 is larger than a predetermined threshold value. This is because it is possible to extract an area including a portion where the occurrence of the occurrence occurs and transmit image data (partial image data and representative coordinates) of the extracted area to the main control unit 800.

  When a plurality of partial image data are extracted, the sensor data processing unit 703 transmits the extracted partial image data to the main control unit 800.

  Furthermore, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 represents the representative coordinates in the partial image data as the coordinates of the point where the pen tip of the pen P contacts. Is detected, and coordinate data indicating the position of the representative coordinate in the partial image data is transmitted to the main control unit 800.

  The scan data storage unit 133 stores the “partial image data” and the “representative coordinates” in the contact position specifying data storage unit 231 of the image data storage unit 23.

  The representative coordinates include, for example, the coordinates of the center of the partial image data, the coordinates of the center of gravity of the partial image data, and the like.

  Next, in both “contact position specifying command” and “position specifying result display command”, “00” (reflection) is designated as the value of the “scan method” field. This is for irradiating the pen tip of the pen P with light that the pixel circuit 31 develops and detecting the reflected light.

  Next, in both “contact position specifying command” and “position specifying result display command”, “01” (multi-value) is designated as the value of the “scanned image gradation” field. This is because the position where the color of the pen tip of the pen P is detected is set as the contact position, and therefore it is necessary to detect with a multi-value gradation.

  In both the “contact position specifying command” and the “position specifying result display command”, either “0” (high) or “1” (low) may be specified as the value of the “scan resolution” field. . This is because a resolution that can specify the contact position of the pen P is sufficient. The value of the “scan resolution” field is preferably set to “0” (high) from the viewpoint of position specifying accuracy.

  Next, the details of the relationship between the operation of the data display / sensor device 100 and the “contact position specifying command” and “position specifying result display command” will be described.

  First, the scan attribute information setting unit 135 reads the “contact position specifying command” stored in advance from the contact position specifying command storage unit 222 and sets the field value of “data acquisition timing” to “event”. Shall.

  Here, when the pen tip of the pen P comes into contact with the sensor built-in liquid crystal panel 301, as described above, the “data acquisition timing” is set to “event”. If the sensor data processing unit 703 recognizes that the image data received from the sensor control unit 602 has changed due to the pen tip of the pen P coming into contact with the sensor built-in liquid crystal panel 301, the main control unit 800 obtains the change. Send the scanned data.

  Note that if the contact determination unit 142 (or the contact position specifying unit 141) determines that the pen tip of the pen P has touched the sensor built-in liquid crystal panel 301 with the scan data transmitted to the main control unit 800 as a trigger. good.

  The scan data storage unit 133 stores the obtained scan data in the contact position specifying data storage unit 231 as contact position specifying data.

  Thereafter, the scan data storage unit 133 notifies the contact position specifying unit 141 that the scan data has been stored in the contact position specifying data storage unit 231, and the contact position specifying unit 141 receives the contact position specifying data. Based on the scan data recorded in the storage unit 231, the contact position of the contact object is specified. In the present embodiment, the coordinate data (Xa, Ya) in the whole image data (area AP) of the representative coordinate Z of the partial image data (area PP) described above is specified.

  Thereafter, the contact position specifying unit 141 notifies the interpolation processing unit 143 that the contact position of the contact object has been specified.

  When the interpolation processing unit 143 receives the above notification and the contact position specified by the contact position specifying unit 141 is one, the interpolation processing unit 143 selects the contact position as the contact position specifying result holding unit 144 or the contact position specifying result storage unit. 232, and notifies the contact position specifying result display image creation unit 182 to that effect.

  On the other hand, the interpolation processing unit 143 specifies the first position specified by the contact position specifying unit 141 when the second contact position is specified after the first contact position is specified by the contact position specifying unit 141. And the interpolation line for interpolating between the second contact position and the second contact position are specified, and all data relating to the first contact position, the second contact position, and the interpolation line are stored in the contact position specifying result holding unit 144 or the contact The information is stored in the position specifying result storage unit 232 and a notification to that effect is sent to the contact position specifying result display image creating unit 182.

  Further, the contact position specifying result display image creating unit 182 receives the notification and displays the contact position, or the first contact position, the second contact position, and the interpolation line. A specific result display image is created.

  Note that the color for displaying the contact position, the first contact position, the second contact position, and the interpolation line may be set in advance.

  Accordingly, the first contact position and the second contact position can be smoothly connected and displayed on the display surface.

(Data display / sensor operation)
Next, based on FIG.10 and FIG.11, the detail of operation | movement of the data display / sensor apparatus 100 is demonstrated.

  FIG. 10 is a timing chart of the contact determination and contact position specifying process of the data display / sensor device 100.

First, as shown in FIG. 10, there are provided a coloring operation period T _{ON in} which R, G, and B are sequentially colored, and a coloring pause period T _{OFF in} which the coloring operation is not performed. Further, the coloring operation period _TON is repeated every 1/10 (second) in FIG.

Here, when the display color of the display surface of the color of the tip of the pen P and the sensor-equipped liquid crystal panel 301 are the same, the color operation period T _ON, but the contact position of the pen P is detected, During the color development pause period T _OFF , the contact position of the pen P is not detected.

In the coloring operation period _TON , R image display data, G image display data, and B image display data are sequentially generated.

Here, R pixel lighting period T _R is the time from being displayed R image display data to the display is completed, G pixel lighting period T _G is the display from the display of the G image data for display There time until completion, B pixel lighting period T _B is the time from B image display data is displayed until the display is completed.

At this time, the length of the coloring operation period T _ON is expressed by the following equation.

Coloring operation period T _ON = R pixel lighting period T _R + G pixel lighting period T _G + B pixel lighting period T _B
The length of the coloring operation period T _ON is, from the viewpoint of reducing the flickering of the screen, as short as possible is preferable.

  Next, based on the flowchart shown in FIG. 11, each flow of a contact position specific process is demonstrated. FIG. 11 is a flowchart showing the operation of the data display / sensor device 100 (contact position specifying step).

  First, the user turns on the power switch 905 of the operation unit 903 of the data display / sensor device 100 and becomes “START”.

  In step S1 (hereinafter simply referred to as “S1”), when the display surface of the sensor built-in liquid crystal panel 301 is in an operating state (screen ON state), the scan attribute information setting unit 135 scans the scan attribute information. The value of each field of “contact position specifying command” is read from the contact position specifying command storage section 222 of the storage section 22 and the scan processing section 12 is driven.

  Here, the scan execution unit 131 of the scan processing unit 12 transmits a “contact position specifying command” to the data processing unit 700 to “NO”, and the process proceeds to S2.

  In S <b> 2, the image display timing control unit 184 drives the normal image creation unit 183 during a predetermined color development pause period to create normal image display data such as an application image, and the created normal image display data is stored as data. The image is transmitted to the display / light sensor unit 300 via the processing unit 700 and the sensor control unit 602, and an application image or the like is displayed on the sensor built-in liquid crystal panel 301.

  At this time, the sensor control unit 602 drives the optical sensor driving circuit 305 with the timing signal TC 2, and the optical sensor driving circuit 305 drives the optical sensor 6 via the optical sensor circuit 32. Scan data such as all image data and partial image data scanned by the optical sensor 6 is transmitted to the sensor data processing unit 703 via the signal conversion circuit 306 and the sensor control unit 602.

  The scan data transmitted to the data processing unit 700 is temporarily stored in the image data buffer 704. The scan of the scan data and the storage of the scan data in the image data buffer 704 are always repeated with a predetermined timing signal TC2.

  Next, the process proceeds to S3, and when the predetermined color development suspension period ends, the image display timing control unit 184 drives the primary color image creation unit 181, and the primary color image creation unit 181 records in the primary color image creation data storage unit 241 in advance. R image display data, G image display data, and R image display data are created based on the primary color image creation data, and the display / light sensor unit is provided via the data processing unit 700 and the sensor control unit 602. 300, the R image, the G image, and the R image are sequentially displayed (colored) on the sensor built-in liquid crystal panel 301, and the process proceeds to S4.

  In S4, since the value of the “data acquisition timing” field of the scan execution unit 131 is set to “event”, the scan execution unit 131 sequentially outputs the R image, the G image, and the R image to the sensor built-in liquid crystal panel 301. If the pen P is in contact with the display surface of the sensor built-in liquid crystal panel 301 before being displayed, at least one of the display images of the R image, the G image, and the R image has predetermined image data. Scan data including at least partial image data is acquired from the data stored in the image data buffer 704 of the data processing unit 700 at a timing when a change equal to or greater than the threshold occurs, and the scan data storage unit 133 contacts the acquired scan data. The data is stored in the position specifying data storage unit 231 and the process proceeds to S5.

  If the pen P is not in contact with the display surface of the sensor built-in liquid crystal panel 301 before the R image, the G image, and the R image are sequentially displayed on the sensor built-in liquid crystal panel 301, the scan execution unit 131 Do not get scan data.

  In S5, the contact position specifying unit 141 reads the scan data with the scan data stored in the contact position specifying data storage unit 231 as a trigger, specifies the contact position of the pen P, and proceeds to S6. In S5, the contact position specifying unit 141 does not operate when the scan data is not stored in the contact position specifying data storage unit 231.

  When the contact position is specified by the contact position specifying unit 141 in S <b> 6, the contact position specifying unit 141 stores the contact position specifying result in the contact position specifying result holding unit 144 and the contact position specifying result storage unit 232. And "YES", the process proceeds to S7. In S6, if the contact position is not specified by the contact position specifying unit 141, the process returns to S1.

  In S <b> 7, the contact position specification result display image creation unit 182 starts the operation with the contact position specification result stored in the contact position specification result holding unit 144 and the contact position specification result storage unit 232 as a trigger. If only one contact position is specified, the process proceeds to S10.

  If two or more contact positions are specified in S7, the contact position specifying result display image creation unit 182 drives the interpolation processing unit 143 and proceeds to S8.

  In S8, the interpolation processing unit 143 specifies an interpolation line that interpolates between the two contact positions that are sequentially specified, and proceeds to S9.

  In S9, the contact position specification result display image creation unit 182 causes the pixel circuit 31 existing on the interpolation line specified by the two or more contact positions and the interpolation processing unit 143 to develop a predetermined color with a predetermined color. And proceed to S11.

  In S10, the contact position specifying result display image creating unit 182 creates a contact position specifying result display image for coloring the pixel circuit 31 at the specified touch position with a predetermined color, and the process proceeds to S11.

  In S11, the contact position specification result display image creation unit 182 transmits the contact position specification result display image created in S9 or S10 to the display / light sensor unit 300 via the data processing unit 700 and the circuit control unit 600. Then, the contact position specifying result display image is displayed on the display surface of the sensor built-in liquid crystal panel 301 and the process proceeds to S12.

  In S12, the contact determination unit 142 determines that the pen tip of the pen P has touched the display surface of the sensor built-in liquid crystal panel 301 using the contact position specified by the contact position specifying unit 141 as a trigger. Is sent to the image display timing control unit 184.

  When the image display timing control unit 184 receives the above notification and the contact determination unit 142 determines that the pen P has touched, whether or not the length of the predetermined color suspension period has already been changed. Confirm.

  If the image display timing control unit 184 confirms that the length of the predetermined color development suspension period has already been changed, “YES” is set, and the process directly returns to S1.

  In S12, when the image display timing control unit 184 confirms that the length of the predetermined coloring pause period has not been changed, the image display timing control unit 184 sets the length of the coloring pause period to a predetermined length. Change the length to short and return to S1.

  Each of the above steps is repeated until the display surface of the sensor built-in liquid crystal panel 301 is finally out of operation (screen OFF state) in S1.

  As described above, the data display / sensor device 100 that can detect the contact position of the pen tip of the pen P in contact with the display surface of the sensor built-in liquid crystal panel 301 useful for downsizing the device regardless of the color of the pen tip. Can be provided.

[Embodiment 2]
The following will describe another embodiment of the present invention with reference to FIGS. Configurations other than those described in the present embodiment are the same as those in the first embodiment. For convenience of explanation, members having the same functions as those shown in the drawings of the first embodiment are given the same reference numerals, and explanation thereof is omitted.

  In the present embodiment, a case will be described in which the functions of the display control unit 601 and the sensor control unit 602 in FIG. 4 are configured by hardware logic.

  FIG. 12 is a block diagram showing an example when the main configuration of the data display / sensor device 100 is implemented by hardware.

  As shown in FIG. 12, the main configuration of the present embodiment (referred to as data display / sensor device A) includes an optical sensor 6, a shift register (coloring control means) 401, an inverter (coloring control means) 402r, an XNOR circuit (coloring). Control means) 402g, inverter (coloring control means) 402b, AND circuit (imaging control means) 403, shift clock generator (coloring control means) 404, transistor (imaging control means) 405r, transistor (imaging control means) 405g and transistor (Imaging control means) 405b, transistor with an inverter (imaging control means) 406r, transistor with an inverter (imaging control means) 406g, transistor with an inverter (imaging control means) 406b, R pixel circuit 31r, G pixel circuit 31g and B pixel circuit 31b, coordinate calculation unit (contact position Constant means, the contact determination means) 14B, and a structure comprising an image memory 407R, the image memory 407g, and an image memory 407b.

  FIG. 12 shows a case where the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31 are all lit (white display). In the following drawings, the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31 are turned on / off in parentheses.

  The shift register (color generation control means) 401 sequentially outputs the information “11000000000000” one by one by driving the shift clock generator 404.

  Note that the drive period of the shift clock generation unit 404 is controlled by the image display timing control unit 184. As described above, this controls the operation timing between the coloring operation period and the coloring suspension period.

  FIG. 12 shows that “0” (LOW level) is output to the inverter 402r and the XNOR circuit 402g, and “1” (HIGH level) is output to the XNOR circuit 402g and the inverter 402b. .

  Since “0” is input, the inverter 402r outputs “1”. As a result, the R pixel circuit 31r is turned on.

  The XNOR circuit 402g outputs “1” because “00” is input. Thereby, the G pixel circuit 31g is turned on.

  The inverter 402b outputs “1” because “0” is input. Thereby, the B pixel circuit 31b is turned on.

  Thereby, the optical sensor 6 receives reflected light generated by irradiating the pen P with light generated by the color (light emission) of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31.

  Here, the AND circuit 403 outputs “1” because “111” is input. However, since “1” is input to all the gates of the transistors 405 r to 405 b, the transistors 405 r to 405 The gate at 405b is opened.

  However, in the transistor with inverter 406r to the transistor 406b with inverter, “1” from the AND circuit 403 is inverted by the inverter and “0” is input to the gate of the transistor. 407r to image storage 407b do not flow.

  That is, the state of FIG. 12 shows a normal image display state (white screen) in which the contact position specifying process is not performed.

  FIG. 13A is a diagram showing a state in which the R pixel is lit when the main configuration of the data display / sensor device A is implemented by hardware, and FIG. 13B is a diagram in which the G pixel is lit. FIG. 13C is a diagram illustrating a state in which the B pixel is lit.

  First, as shown in FIG. 13A, the shift register 401 is driven, and “0” is input to the inverter 402r. Therefore, the inverter 402r outputs “1”. As a result, the R pixel circuit 31r is turned on.

  Since “01” is input to the XNOR circuit 402g, the XNOR circuit 402g outputs “0”. Thereby, the G pixel circuit 31g is not lit.

  Further, since “1” is input to the inverter 402b, the inverter 402b outputs “0”. Thereby, the B pixel circuit 31b is not lit.

  At this time, the optical sensor 6 receives reflected light generated by irradiating the pen P with light generated by the R pixel circuit 31r to develop color (emit light).

  Here, since “100” is input to the AND circuit 403, the AND circuit 403 outputs “0”.

  Here, since “0” is input from the AND circuit 403 to the inverters 402r to 402b, all “1” s are input to the gates of the transistors 406r to 406b with inverters. The gate of the transistor 406b with the inverter 406b is opened.

  On the other hand, in the transistors 405r to 405b, since only “1” is input to the gates of the transistors 405r, only the gates of the transistors 405r are opened.

  Thereby, the detection information based on the detection signal of the optical sensor 6 is stored in the image storage 407r. Thereafter, the shift clock generation unit 404 drives the shift register 401 and proceeds to the state of FIG.

  In FIG. 13B, since the shift register 401 is driven and “1” is input to the inverter 402r, the inverter 402r outputs “0”. As a result, the R pixel circuit 31r is not lit.

  In addition, since “11” is input to the XNOR circuit 402g, the XNOR circuit 402g outputs “1”. As a result, the G pixel circuit 31g is turned on.

  Further, since “1” is input to the inverter 402b, the inverter 402b outputs “0”. Thereby, the B pixel circuit 31b is not lit.

  At this time, the optical sensor 6 receives reflected light generated by irradiating the pen P with light generated by the G pixel circuit 31g to develop color (emit light).

  Here, since “010” is input to the AND circuit 403, the AND circuit 403 outputs “0”.

  Here, since “0” is input from the AND circuit 403 to the inverters 402r to 402b, all “1” s are input to the gates of the transistors 406r to 406b with inverters. The gate of the transistor 406b with the inverter 406b is opened.

  On the other hand, in the transistors 405r to 405b, since only “1” is input to the gate of only the transistor 405g, only the transistor 405g is opened.

  Thereby, the detection information based on the detection signal of the optical sensor 6 is stored in the image storage 407g.

  Similarly, in FIG. 13C, since the shift register 401 is driven and “1” is input to the inverter 402r, the inverter 402r outputs “0”. As a result, the R pixel circuit 31r is not lit.

  Further, since “10” is input to the XNOR circuit 402g, the XNOR circuit 402g outputs “0”. Thereby, the G pixel circuit 31g is not lit.

  Since “0” is input to the inverter 402b, the inverter 402b outputs “1”. Thereby, the B pixel circuit 31b is turned on.

  At this time, the optical sensor 6 receives reflected light generated by irradiating the pen P with light generated by the color (light emission) of the B pixel circuit 31b.

  Here, since “001” is input to the AND circuit 403, the AND circuit 403 outputs “0”.

  Here, since “0” is input from the AND circuit 403 to the inverters 402r to 402b, all “1” s are input to the gates of the transistors 406r to 406b with inverters. The gate of the transistor 406b with the inverter 406b is opened.

  On the other hand, in the transistors 405r to 405b, since only “1” is input to the gate of only the transistor 405b, only the gate of the transistor 405b is opened.

  Thereby, the detection information based on the detection signal of the optical sensor 6 is stored in the image storage 407b.

  The coordinate calculation unit 14B triggers the detection information to be recorded in all of the image storage 407r to the image storage 407b by the series of operations from FIG. 13A to FIG. 13C described above. The contact position of the pen tip of P is analyzed, and coordinate information is output. As for the contact determination, the coordinate calculation unit 14B may determine that the pen P has touched the sensor built-in liquid crystal panel 301 using the output of the coordinate information as a trigger.

  From the above, the contact position of the pen tip of the pen P that touches the display surface of the input / output integrated display (liquid crystal panel 301 with a built-in sensor) useful for downsizing the device is detected regardless of the color of the pen tip. A data display / sensor device A can be provided.

[Embodiment 3]
The following will describe still another embodiment of the present invention with reference to FIG. Configurations other than those described in the present embodiment are the same as those in the first and second embodiments. For convenience of explanation, members having the same functions as those shown in the drawings of Embodiments 1 and 2 are given the same reference numerals, and explanation thereof is omitted.

  FIG. 14 is a block diagram illustrating another example in which the main configuration of the data display / sensor device (referred to as data display / sensor device B) of the present embodiment is implemented by hardware.

  This embodiment is different from the second embodiment only in that the transistor 405b is not provided and a transistor (imaging control means) 406w is provided instead of the transistor with an inverter 406b. The configuration is the same as in the second embodiment. Since the operation is the same as that of the second embodiment, the description is omitted here, and the difference in operation between the second embodiment and the present embodiment will be described.

  In the data display / sensor device A according to the second embodiment, each light of R, G, and B is sequentially emitted, and each light of R, G, and B is an image as detection information detected by the optical sensor 6. It is recorded in the storage 407r to the image storage 407b.

  On the other hand, in the third embodiment, each of R, G, and white (all of R to G are lit) emits light sequentially, and the R, G, and white lights are detected by the optical sensor 6. Information is recorded in the image storage 407r to image storage 407w.

  As described above, the contact position of the pen tip of the pen P that is in contact with the display surface of the input / output integrated display (liquid crystal panel 301 with a built-in sensor) useful for downsizing of the device can be detected regardless of the color of the pen tip. A possible data display / sensor device B can be provided.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the technical means disclosed in different embodiments can be appropriately combined. Such embodiments are also included in the technical scope of the present invention.

  Finally, each block of the data display / sensor device 100, particularly the main control unit 800, may be configured by hardware logic, or may be realized by software using a CPU as follows.

  That is, the data display / sensor device 100 includes a CPU (central processing unit) that executes instructions of a control program that realizes each function, a ROM (read only memory) that stores the program, and a RAM (random access) that expands the program. memory), a storage device (recording medium) such as a memory for storing the program and various data. An object of the present invention is a record in which a program code (execution format program, intermediate code program, source program) of a control program for the data display / sensor device 100, which is software for realizing the functions described above, is recorded so as to be readable by a computer. This can also be achieved by supplying a medium to the data display / sensor device 100 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and a compact disk-ROM / MO / MD / digital video disk / compact disk-R. A disk system including an optical disk, a card system such as an IC card (including a memory card) / optical card, or a semiconductor memory system such as a mask ROM / EPROM / EEPROM / flash ROM can be used.

  The data display / sensor device 100 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention can be applied to an input / output integrated information processing apparatus including a plurality of unit pixels and a plurality of imaging sensors. Specifically, the present invention can be widely applied to electronic devices that can perform pen input directly on the display screen.

It is a schematic diagram which shows the outline structure of the data display / sensor apparatus which concerns on one Embodiment of this invention, and the outline | summary of the function. (A) is a schematic diagram which shows an example of the cross section of the liquid crystal panel with a sensor with which the said data display / sensor apparatus is equipped, (b) is another example of the liquid crystal panel with a sensor with which the said data display / sensor apparatus is equipped. It is a schematic diagram which shows 1 pixel. (A) is a schematic diagram which shows a mode that the position which the user touched is detected by detecting a reflected image with the sensor built-in liquid crystal panel with which the said data display / sensor apparatus is equipped, (b) is the said data It is a schematic diagram which shows a mode that the position which the user touched is detected by detecting a shadow image with the sensor built-in liquid crystal panel with which a display / sensor apparatus is provided. It is a block diagram which shows the principal part structure of the said data display / sensor apparatus. It is a figure which shows typically an example of the frame structure of the command used with the said data display / sensor apparatus. It is a figure explaining an example of the value which can be specified to each field contained in the command shown in FIG. 5, and its outline. (A) is image data obtained as a result of scanning the entire liquid crystal panel with built-in sensor when the contact object is not placed on the liquid crystal panel with built-in sensor in the data display / sensor device, (b) Is image data obtained as a result of scanning when the user touches the sensor built-in liquid crystal panel with a finger in the data display / sensor device. It is a block diagram which shows the structure of the liquid crystal panel with a sensor with which the said data display / sensor apparatus is provided, and the structure of its peripheral circuit. It is a block diagram which shows the detail of a main control part and a memory | storage part among the principal part structures of the said data display / sensor apparatus. It is a timing chart of the contact position specific process of the data display / sensor device. It is a flowchart which shows operation | movement of the said data display / sensor apparatus. It is a block diagram which shows an example at the time of mounting the principal part structure of the said data display / sensor apparatus by hardware. (A) is a figure which shows a mode that R pixel is lighting when the principal part structure of the said data display / sensor apparatus is mounted by hardware, (b) is a mode that G pixel is lighting. (C) is a figure which shows a mode that B pixel is lighted. It is a block diagram which shows the other example at the time of mounting the principal part structure of the said data display / sensor apparatus by hardware. (A) And (b) is a conceptual diagram for demonstrating the relationship of the magnitude | size of an image sensor and a color filter in the case of providing a color filter between a light source and an image sensor in the conventional color scanner. (C) is a figure which shows a mode that red, blue, and green LED (light-emitting diode) is lighted one by one in the conventional color scanner, and each light is scanned.

Explanation of symbols

6 Optical sensor (imaging sensor)
12 Scan processing unit (imaging control means)
14A Contact position determination processing unit (contact position specifying means, contact determination means)
14B coordinate calculation unit (contact position specifying means, contact determining means)
18 Image display processing unit (color development control means, position specifying result display control means)
22 Scan attribute information storage unit 23 Image data storage unit 24 Display attribute information storage unit 31 Pixel circuit (unit pixel)
31r R pixel circuit (color pixel, R pixel)
31g G pixel circuit (color pixel, G pixel)
31b B pixel circuit (color pixel, B pixel)
53r, 53g, 53b Color filters (3 primary color filters)
100 Data display / sensor device (position identification device)
131 Scan execution unit (imaging control means)
132 Scan execution instruction section (imaging control means)
133 Scan data storage (imaging control means)
134 Scan data holding unit 135 Scan attribute information setting unit 141 Contact position specifying unit (contact position specifying means)
142 contact determination unit (contact determination means)
143 Interpolation processing unit (interpolation line specifying means)
144 Contact position specifying result holding unit 181 Primary color image creating unit (coloring control means)
182 Contact position specifying result display image creating unit (position specifying result display control means)
183 Normal image creation unit 184 Image display timing control means (period control means)
222 Contact position specification command storage unit 223 Position specification result display command storage unit 231 Contact position specification data storage unit 232 Contact position specification result storage unit 241 Primary color image creation data storage unit 242 Image display timing storage unit 300 Display / light sensor unit 301 Liquid crystal panel with built-in sensor (planar member)
301A Sensor built-in liquid crystal panel (planar member)
301B Liquid crystal panel with built-in sensor (planar member)
401 shift register (color development control means)
402r inverter (coloring control means)
402g XNOR circuit (color control means)
402b Inverter (color development control means)
403 AND circuit (imaging control means)
404 Shift clock generator (color control means)
405r, 405g, 405b Transistor (imaging control means)
406r, 406g, 406b Transistor with inverter (imaging control means)
406w transistor (imaging control means)
407r, 407g, 407b Image storage 407w Image storage 600 Circuit control unit 601 Display control unit 602 Sensor control unit 603 Backlight control unit 700 Data processing unit 701 Display data processing unit 703 Sensor data processing unit 800 Main control unit 901 Storage unit 903 Operation Part P Pen (contact object)
PP region L locus H hinge T _ON color development operation period T _OFF color development pause period T _R R pixel lighting period T _G G pixel lighting period T _B B pixel lighting period

Claims (11)

A planar member including a plurality of unit pixels and a plurality of image sensors provided at least one for each unit pixel;
Color development control means for sequentially developing each of the unit pixels in a first color, a second color, and a third color that satisfy the principle of color matching,
Each time the coloring control means develops each of the unit pixels in the first color, the second color, and the third color, an image of a contact object that contacts the planar member is used for each imaging sensor. Imaging control means for imaging
The contact position of the contact object on the planar member is determined based on each of the images of the contact object imaged each time the first color control unit, the second color, and the third color are developed by the imaging control unit. A position specifying device comprising a position specifying means for specifying.   A color development pause period is provided between the color development operation period from the start of the operation of developing the first color until the end of the operation of developing the third color and the next color development operation period. The position specifying device according to claim 1, further comprising a period control unit.   The position specifying device according to claim 2, wherein the period control unit adjusts the period between the coloring operation period and the coloring stop period. Contact determining means for determining whether or not the contact object is in contact with the planar member;
The said period control means shortens the length of the said color development rest period, when the said contact determination means determines with the said contact thing contacting the said planar member. Positioning device.   The said period control means lengthens the length of the said color development rest period, when the said contact determination means determines with the said contact thing not contacting the said planar member. Positioning device. When the second contact position is specified after the first contact position is specified by the position specifying means,
Interpolation line specifying means for specifying an interpolation line for interpolating between the first contact position and the second contact position specified by the position specifying means,
The position specifying result display control means for displaying the first contact position, the second contact position, and the interpolation line is provided. Positioning device. The first color is one of four colors consisting of red, green, blue and white,
The second color is one of the remaining three colors that are not the first color among the four colors.
The position specifying device according to any one of claims 1 to 6, wherein the third color is one of the remaining two colors that are not the second color among the three colors. .   The position specifying device according to any one of claims 1 to 7, wherein the number of the unit pixels and the number of the imaging sensors are one-to-one and the same number.   A position specifying program for causing a computer to operate as each means in the position specifying device according to claim 1.   A computer-readable recording medium on which the position specifying program according to claim 9 is recorded. A position specifying method using a planar member including a plurality of unit pixels and a plurality of imaging sensors provided for each unit pixel,
A coloring control step of sequentially developing each of the unit pixels in a first color, a second color, and a third color that satisfy the principle of color matching;
In the color development control step, each time the unit pixels are colored with the first color, the second color, and the third color, an image of a contact object that contacts the planar member is used for each imaging sensor. An imaging control step for imaging,
The contact position of the contact object on the planar member is determined based on each of the images of the contact object imaged each time the first, second, and third colors are developed in the imaging control step. And a position specifying step for specifying.

Images