JP2010096909A - Image display/image detection device and inspection method for display unit provided in image display/image detection device - Google Patents

Abstract

An image display / image detection apparatus having a function capable of easily and surely detecting a shift in the relative position of display panels provided in different housings.
A data display / sensor device 100 according to the present invention includes an image display control unit 16 for displaying a predetermined inspection image on a display unit 310 in a state where the display / light sensor unit 300 and the display unit 310 face each other. And a determination unit 11 that determines whether the image information of the detection image detected by the optical sensor in the display / light sensor unit 300 matches the image information of the inspection image.
[Selection] Figure 1

Description

  The present invention relates to an image display device that displays an image, and in particular, an image display that includes a plurality of display units each having a display panel that displays an image, and at least one of the display units includes a display panel with a built-in sensor. The present invention relates to an apparatus and an inspection method for inspecting misalignment between display units provided in the image display apparatus.

  In recent years, in portable information terminals such as mobile phones and portable game devices, devices having a plurality of display units for displaying images are increasing. In these devices, in addition to the function of displaying an image, at least one of the display units has a so-called touch panel function that can detect a position pointed (touched) by the user with a finger or the like. There are many.

  Thus, in a device having a plurality of display units, it is common that each display unit is provided in a different housing. And each housing | casing with which the display part is provided is mutually connected by the connection part, and when the apparatus is not used, the display parts face each other so that the display part is not damaged so that the display part is not damaged. It can be folded into a state to do.

  Further, even in a scanner device that can read information described in a document and digitize it as image data, there is a scanner device that can simultaneously read both sides of a document with a single operation in order to improve user convenience. Conventionally used. Such a scanner device is generally a large and stationary type such as a copying machine, but a portable type has also appeared (for example, see Patent Document 1).

The scanner device disclosed in Patent Document 1 has a structure in which two scanner units are folded at a hinge portion, and can simultaneously read both sides of a document sandwiched between the folded scanner units.
JP 2008-66961 A (publication date: March 21, 2008)

  However, since the connecting portions that connect the housings are easily distorted, when the device is folded so that the display portions face each other, the display provided in each housing is caused by the distortion of the connecting portions. There is a possibility that the relative positions of the parts shift. Further, the display unit may be distorted and attached to the casing due to a manufacturing error in manufacturing the device, specifically, a manufacturing error when the display unit is attached to the casing.

  In such a case, it is common for the inspector to visually check the manufactured device to check the relative position shift between the display units and the display unit mounting error at the time of manufacturing, but the efficiency is low. In addition, there is a problem that it is very difficult to reliably detect defective products.

  Further, as a detection method for automatically detecting a shift in the relative position between display units without depending on the visual inspection of an inspector, for example, a method of performing image processing on an image captured using an imaging device such as a camera is known. ing. However, such a detection method requires a separate inspection device such as an imaging device and an image processing device, which increases the cost in the inspection processing step and, in turn, increases the manufacturing cost of the device itself. have.

  The present invention has been made in view of the above problems, and its main purpose is to easily and reliably detect a shift in the relative position of display units provided in different housings. An object of the present invention is to provide an image display device (image display / image detection device) having a function capable of being performed.

In order to solve the above problems, an image display / image detection apparatus according to the present invention is provided.
A first planar member for displaying an image is disposed on one inner surface of two inner surfaces facing each other in a folded state, the housing having a foldable casing, and the other inner surface. And an image display / image detection device provided with a second planar member for detecting a nearby image,
Display control means for displaying a predetermined inspection image on the first planar member in a state in which the housing is folded;
Determination means for determining whether or not an image of the first planar member detected by the second planar member in a state in which the housing is folded matches the inspection image;
It is characterized by having.

  The image display / image detection apparatus according to the present invention is configured to perform predetermined inspection on the first planar member in a state where the casing is folded, that is, in a state where the first planar member and the second planar member are opposed to each other. The image is displayed, and the displayed inspection image is detected on the second planar member. Then, the determination unit determines whether or not the image of the first planar member detected by the second planar member matches the inspection image.

  That is, in the determination means in the image display / image detection apparatus according to the present invention, when the image of the first planar member detected by the second planar member matches the inspection image, the first planar shape It is determined that there is no deviation in the relative position of the second planar member with respect to the member, and if they do not match, it is determined that there is a deviation in the relative position of the second planar member with respect to the first planar member. To do.

  As described above, in the image display / image detection apparatus according to the present invention, the relative position of the second planar member relative to the first planar member can be shifted in the own apparatus without relying on the visual inspection by the inspector or another inspection device. Can be detected. Moreover, the shift | offset | difference of the relative position of the 2nd planar member with respect to a 1st planar member can be detected only by the operation | movement which folds a housing | casing so that a 1st planar member and a 2nd planar member may oppose. . That is, it is possible to detect a shift in the relative position of the second planar member with respect to the first planar member without forcing the examiner to perform a complicated operation.

  Thus, the image display / image detection apparatus according to the present invention has an effect that the displacement of the relative position of the second planar member with respect to the first planar member can be easily and reliably detected.

In the image display / image detection apparatus according to the present invention, it is further preferable that the determination unit does not match the image of the first planar member detected by the second planar member with the inspection image. Calculating means for calculating a deviation amount between the image of the first planar member detected by the second planar member and the inspection image when determined;
Correction means for correcting the position coordinates of the high brightness region in the image detected by the second planar member based on the shift amount;
Is preferably further provided.

  According to said structure, based on the calculated deviation | shift amount, the position coordinate of the high-intensity area | region in the image detected by the said 2nd planar member can be correct | amended. Thereby, there is an effect that it is possible to create corrected image data of a high luminance region in the image detected by the second planar display member.

  In the present specification and the like, the “high luminance region” is a minimum rectangle that includes all portions in which brightness of pixel values is higher than a predetermined threshold in an image detected by a detection unit that detects a nearby image. Means an area.

  In the image display / image detection apparatus according to the present invention, the image detected by the second planar member is further corrected according to the amount of deviation calculated by the calculating means, and then the first planar member. It is preferable to further include a corrected image display control means for displaying on the screen.

  According to said structure, there exists an effect which can display the correction | amendment image which considered deviation | shift amount on a 1st planar member.

  In the image display / image detection apparatus according to the present invention, the first surface detected by the second planar member in a state where the display control unit displays a black image on the first planar member. Preferably, the image forming apparatus further includes a specifying unit that specifies a position of a region having a luminance equal to or higher than a predetermined threshold in the first planar member based on the image of the member.

  In the image display / image detection apparatus according to the present invention, the first surface detected by the second planar member in a state where the display control unit displays a white image on the first planar member. Preferably, the image forming apparatus further includes a specifying unit that specifies a position of a region having a luminance equal to or lower than a predetermined threshold in the first planar member based on the image of the member.

  According to said structure, there exists an effect which can also detect the detection of a bright spot or the detection of a black spot other than the detection of the shift | offset | difference of the relative position of a 1st planar member and a 2nd planar member in an own apparatus. .

  In the present specification and the like, the “black image” means an image in which the pixel values of the respective pixels in the first planar member are all maximum values, and the “white image” means the first planar member. This means an image in which the pixel values of all the pixels in FIG.

  In the image display / image detection apparatus according to the present invention, when the region specified by the specifying unit is displayed on the first planar member, the region is displayed on the first planar member. It is preferable that marker display control means for displaying the marker on the first planar member is further provided.

  According to said structure, there exists an effect which an inspector can determine easily whether a bright spot or a black spot exists in a 1st planar member.

  Further, by displaying the area specified by the specifying means on the first planar member, a bright spot or a black spot is caused by a defect of a detection unit for detecting an image in the vicinity provided on the second planar member. There is an effect that it can be determined whether or not it is caused by missing dots in the first planar member.

  In the image display / image detection apparatus according to the present invention, it is further preferable that the first planar member displays an image and detects a nearby image.

  According to said structure, there exists an effect which can detect simultaneously the image of the object inserted between housing | casings in a 1st planar member and a 2nd planar member.

In order to solve the above problems, an inspection method for an image display / image detection apparatus according to the present invention provides:
A first planar member for displaying an image is disposed on one inner surface of two inner surfaces facing each other in a folded state, the housing having a foldable casing, and the other inner surface. And an image display / image detection apparatus inspection method in which a second planar member for detecting a nearby image is provided,
A display control step for displaying a predetermined inspection image on the first planar member in a state in which the housing is folded;
A determination step of determining whether an image of the first planar member detected by the second planar member in a state in which the housing is folded is coincident with the inspection image;
It is characterized by including.

  According to said structure, there exists an effect similar to the image display / image detection apparatus which concerns on this invention.

  Also, a program for operating the image display / image detection apparatus according to the present invention, which is characterized in that the computer is driven as each of the above-mentioned means and a computer-readable recording medium on which the program is recorded. It is included in the category of the present invention.

  As described above, the image display / image detection apparatus according to the present invention has the first planar member in a state where the casing is folded, that is, in a state where the first planar member and the second planar member are opposed to each other. Image display control means for displaying a predetermined image for inspection, and determination means for determining whether or not the image of the first planar member detected by the second planar member matches the inspection image. I have. Thus, the image display / image detection apparatus according to the present invention has an effect that the displacement of the relative position of the second planar member with respect to the first planar member can be easily and reliably detected.

  An embodiment of a data display / sensor device (image display / image detection device) according to the present invention will be described below with reference to FIGS.

(Data display / appearance of sensor device)
As shown in FIG. 2, the data display / sensor device 100 includes an upper housing 40, a lower housing 41, and a connecting portion 42. FIG. 2 is a diagram showing an external appearance of the data display / sensor device 100. As shown in FIG. 2, the upper casing 40 is provided with a display unit 310, and the lower casing 41 is provided with a display / light sensor unit 300. The connecting portion 42 serves as a hinge for connecting the upper housing 40 and the lower housing 41 to each other and folding the data display / sensor device 100 so that the display / light sensor portion 300 and the display portion 310 face each other. Also have.

  Although FIG. 2 shows the connecting portion 42 formed by connecting a part of the upper housing 40 and a part of the lower housing 41 to each other, it is of course not limited to this. For example, a hinge may be attached to the data display / sensor device 100 as the connecting portion 42.

  2 illustrates the foldable data display / sensor device 100, the form of the data display / sensor device 100 is not limited to this, and the data display / sensor device 100 is not used. Any form that can be arranged so that the display / light sensor unit 300 and the display unit 310 sometimes face each other is acceptable. For example, the form of the data display / sensor device 100 is a so-called slide type in which the upper housing 40 (or the lower housing 41) is slid so that the display / light sensor unit 300 and the display unit 310 face each other. It may be a form.

(Configuration of data display / sensor device)
The main configuration of the data display / sensor device 100 according to the present invention will be described below with reference to FIG. FIG. 1 is a block diagram showing a main configuration of the data display / sensor device 100. As shown in FIG. 1, the data display / sensor device 100 includes a display / light sensor unit 300, a display unit 310, a main control unit 800, an image display control unit 16, a corrected image display control unit 17, a marker display control unit 18, And a storage unit 901. Details of each member will be described below.

(Display / light sensor unit 300)
The display / light sensor unit 300 is a so-called liquid crystal display device with a built-in light sensor. In addition to the sensor built-in liquid crystal panel 301, the display / light sensor unit 300 includes a backlight, and a sensor built-in liquid crystal panel 301 and a peripheral circuit for driving the backlight. The sensor built-in liquid crystal panel 301 includes a plurality of pixel circuits and photosensor circuits arranged in a matrix. Since a more detailed configuration of the sensor built-in liquid crystal panel 301 will be described later, the description thereof is omitted here.

(Display unit 310)
The display unit 310 is a conventionally known display device that includes a liquid crystal panel 311 that displays data. The display panel included in the display unit 310 is not limited to a liquid crystal panel, and may be an organic EL (Electro Luminescence) panel or the like. Further, the liquid crystal panel 311 of the display unit 310 may be a sensor built-in liquid crystal panel similarly to the display / light sensor unit 300.

  The shape of the display / light sensor unit 300 and the shape of the display unit 310 are not limited to the same shape, and may be different shapes.

(Main control unit 800)
As shown in FIG. 1, the main control unit 800 includes a determination unit 11, a calculation unit 12, a correction unit 13, a detection unit 14, and a conversion unit 15. Each member in the main control unit 800 will be described below.

(Determination unit 11)
The determination unit 11 compares the image information of the detected image detected by the optical sensor included in the display / light sensor unit 300 with the image information of the inspection image, and determines whether the compared image information matches. judge.

  Note that the “detected image” in the present embodiment means a high luminance region in the image of the display unit 310 detected by the optical sensor provided in the display / light sensor unit 300.

(Calculation unit 12)
In the calculation unit 12, the image information of the detected image detected by the optical sensor provided in the display / light sensor unit 300 in the determination unit 11 matches the image information of the inspection image displayed on the display unit 310. When it is determined that the display / light sensor unit 300 is not to be moved, a shift amount indicating the degree of shift of the relative position of the display / light sensor unit 300 with respect to the display unit 310 is calculated.

(Correction unit 13)
The correction unit 13 corrects the position coordinates of each pixel in the detection image detected by the display / light sensor unit 300 based on the shift amount calculated by the calculation unit 12. The correction unit 13 also corrects the luminance information of each pixel based on the shift amount. That is, the correction unit 13 generates corrected image data of the detected image based on the shift amount.

  In the present specification and the like, “positional coordinates” means data indicating the positional coordinates of pixels.

(Detector 14)
The detection unit 14 detects a pixel having a luminance value equal to or higher than a predetermined threshold in a detection image detected by the optical sensor provided in the display / light sensor unit 300. Further, when there is a pixel having a luminance value equal to or higher than a predetermined threshold, the position coordinate of the pixel in the sensor built-in liquid crystal panel 301 is detected.

(Conversion unit 15)
The conversion unit 15 converts the position coordinates so that an image displayed based on the corrected image data generated by the correction unit 13 is displayed upside down or horizontally reversed. This is because the image detected by the display / light sensor unit 300 is a mirror image of the original image. Note that a conventionally known algorithm may be used as the algorithm at this time.

(Image display control unit 16, corrected image display control unit 17)
The image display control unit 16 displays the inspection image on the display unit 310 when the display / light sensor unit 300 and the display unit 310 are folded. The corrected image display control unit 17 displays the corrected image data generated by the correcting unit 13 on the display unit 310 based on the position coordinates converted by the converting unit 15. That is, the corrected image display control unit 17 displays the image detected by the display / light sensor unit 300 by vertically or horizontally inverting it.

(Marker display controller 18)
The marker display control unit 18 displays a marker indicating the pixel specified by the position coordinates (position coordinates in the display unit 310) converted by the conversion unit 15 on the display unit 310.

(Storage unit 901)
The storage unit 901 records the deviation amount calculated by the calculation unit 12. The storage unit 901 is not limited to the internal memory provided in the data display / sensor device 100. The storage unit 901 may be an external memory attached to the outside of the data display / sensor device 100.

(Data display / inspection method of display unit in sensor device 100)
Next, an outline of a display unit inspection method in the data display / sensor device 100 will be described below with reference to FIG. FIG. 3 is a flowchart showing an inspection method of the display unit 310 in the data display / sensor device 100.

  First, when the data display / sensor device 100 is folded, that is, when the display / light sensor unit 300 provided in the upper housing 40 and the display unit 310 provided in the lower housing 41 face each other, the image display control unit 16 Then, a predetermined inspection image is displayed on the liquid crystal panel 311 of the display unit 310 (step S1). When the inspection image is displayed on the liquid crystal panel 311, the optical sensor provided in the sensor built-in liquid crystal panel 301 of the display / light sensor unit 300 detects the inspection image displayed on the liquid crystal panel 311 (step S <b> 2). ).

  Subsequently, the determination unit 11 compares the image information of the detection image detected by the optical sensor provided in the sensor built-in liquid crystal panel 301 with the image information of the inspection image displayed on the liquid crystal panel 311, and compares them. It is determined whether or not the obtained image information matches (step S3). When the determination unit 11 determines that the image information of the detected image matches the image information of the inspection image (YES in step S4), the determination unit 11 displays the display / light sensor in the data display / sensor device 100. It is determined that there is no deviation in the relative position between the unit 300 and the display unit 310, and the inspection process is terminated.

  On the other hand, when the determination unit 11 determines that the image information of the detected image and the image information of the inspection image do not match, that is, the determination unit 11 displays the display / light sensor unit 300 in the data display / sensor device 100. When it is determined that there is a deviation in the relative position between the display unit 310 and the display unit 310 (NO in step S4), the calculation unit 12 calculates a deviation amount indicating the degree of the relative position deviation between the display / light sensor unit 300 and the display unit 310. Calculate (step S5).

  The calculation unit 12 calculates a deviation amount and determines whether the calculated deviation amount is equal to or less than a predetermined threshold Th1 (step S6). When the calculated shift amount exceeds the predetermined threshold Th1 (NO in step S6), the data display / sensor device 100 detects the shift of the image due to the shift of the relative position between the display / light sensor unit 300 and the display unit 310. It is determined that correction is impossible, and the inspection process is terminated. That is, it is determined that the data display / sensor device 100 is defective. When the calculated deviation amount is equal to or less than the predetermined threshold Th1 (YES in step S6), the calculation unit 12 records the calculated deviation amount in the storage unit 901 and ends the inspection process (step S7).

  As described above, in the data display / sensor device 100, when the deviation amount exceeds the threshold value Th1, it is determined that the device itself is a defective product. When it is determined that the device itself is defective, the data display / sensor device 100 preferably notifies the inspector to that effect using sound, light, or the like.

  On the other hand, when the deviation amount is equal to or less than the threshold Th1, even if a deviation has occurred, it is possible to cover the deviation amount by correcting the deviation amount as described later. Not judged as defective.

  In the above-described inspection process, the case where the data display / sensor apparatus 100 is folded and the inspection process is executed at the same time is described as an example, but the present invention is not limited to this. For example, the above-described inspection process is performed only when the inspector (or user) presses a switch provided in the data display / sensor device 100 to switch the data display / sensor device 100 to the inspection mode. May be. The switch for setting the data display / sensor device 100 to the inspection mode is not limited to a hardware switch, and may be a software switch.

  The inspection image used in the above-described inspection process is not particularly limited as long as it is recorded in advance in the data display / sensor device 100. Further, an inspector (or user) may be able to set an inspection image to be displayed.

(Advantages of data display / sensor device)
As described above, in the data display / sensor device 100, when the data display / sensor device 100 is folded and disposed at a position where the display / light sensor unit 300 and the display unit 310 face each other, the image display control unit 16 Displays a predetermined inspection image on the display unit 310, and the optical sensor in the display / light sensor unit 300 detects the displayed inspection image. Then, the determination unit 11 determines whether the image information of the detected image in the image of the display unit 310 detected by the display / light sensor unit 300 matches the image information of the inspection image.

  In this manner, the data display / sensor device 100 can detect a shift in the relative position of the display / light sensor unit 300 with respect to the display unit 310 in the device itself without relying on visual inspection by the inspector or another inspection device. it can.

  Further, in the data display / sensor device 100, the display / light sensor unit 300 is relative to the display unit 310 only by folding the data display / sensor device 100 so that the display / light sensor unit 300 and the display unit 310 face each other. A positional shift can be detected. That is, the data display / sensor device 100 can detect a shift in the relative position of the display / light sensor unit 300 with respect to the display unit 310 without forcing the examiner to perform complicated operations.

  As a result, the data display / sensor device 100 can easily and reliably detect a shift in the relative position of the display / light sensor unit 300 with respect to the display unit 310.

  Further, in the data display / sensor device 100, when the determination unit 11 determines that the image information of the detected image and the image information of the inspection image displayed on the display unit 310 do not match, the calculation unit 12 Calculates the shift amount of the position coordinates of the display / light sensor unit 300 with respect to the position coordinates of the display unit 310. Then, the correction unit 13 corrects the position coordinates of each pixel in the detection image detected by the display / light sensor unit 300 based on the shift amount calculated by the calculation unit 12.

  As a result, the data display / sensor device 100 can display the corrected image displayed based on the corrected image data obtained by correcting the detected image based on the calculated deviation amount to the user.

(Detail of calculation method of deviation)
Next, details of the amount of deviation calculated by the calculation unit 12 will be described below with reference to FIGS. FIGS. 4A to 4F are diagrams showing a detailed method of detecting a relative position shift between the display / light sensor unit 300 and the display unit 310. FIG. 4A shows an inspection image. b) shows a detected image, (c) is a diagram showing calculation of image information in the detected image, (d) is a diagram showing calculation of a center point, and (e) is a diagram of image information in the inspection image. It is a figure which shows calculation, (f) is a figure which shows the comparison with the image information of the image for a test | inspection, and the image information of a detection image.

  First, the image display control unit 16 displays an inspection image as shown in FIG. 4A on the liquid crystal panel 311 of the display unit 310. Subsequently, the optical sensor provided in the sensor built-in liquid crystal panel 301 of the display / optical sensor unit 300 detects the inspection image shown in FIG.

  Next, as illustrated in FIG. 4C, the determination unit 11 acquires the position coordinates of the center points a1, b1, and c1 of the detected image. Further, as shown in FIG. 4D, the position coordinate of the center point d1 is acquired based on the acquired position coordinates of b1 and c1. The center point d1 calculates the position coordinates of the intermediate point of a vector having the center point b1 as the start point and the center point c1 as the end point (or a vector having the center point c1 as the start point and the center point b1 as the end point). Can be obtained.

  At the same time, as shown in FIG. 4E, the determination unit 11 calculates the center points a2, b2, and c2 of the inspection mark in the inspection image and determines the position coordinates of the center point d2 that is the center of the inspection image. calculate. The center point d2 is the center of the liquid crystal panel 311 in the display unit 301. The “inspection mark” in this specification and the like means a region of an inspection image that is detected as a high luminance region by a detection unit that detects a nearby image.

  Thus, in this embodiment, the position coordinates of the center points a1, b1, c1, and d1 are the image information of the detected image, and the position coordinates of the center points a2, b2, c2, and d2 are the image information of the inspection image. . The position coordinates of the center points a2, b2, and c2 do not need to be calculated every time the shift amount is calculated. For example, what is calculated in advance and recorded may be read. In addition, as a calculation method of the position coordinates of the center points a2, b2, and c2, a conventionally known calculation method may be used.

  Finally, as shown in FIG. 4 (f), the determination unit 11 includes a center point a1 and a center point a2, a center point b1 and a center point b2, a center point c1 and a center point c2, and a center point d1 and a center point d2. By comparing the position coordinates, it is determined whether or not there is a deviation between the detected image and the inspection image.

  That is, when the position coordinates of the center points a1, b1, c1, and d1 and the position coordinates of the center points a2, b2, c2, and d2 completely match, the determination unit 11 shifts between the detection image and the inspection image. It is determined that it has not occurred. That is, it is determined that there is no deviation in the relative position of the display / light sensor unit 300 with respect to the display unit 310.

  In FIG. 4F, for the sake of convenience, the outer frame of the display / light sensor unit 300 is indicated by a broken line in order to facilitate understanding of the shift between the detection image and the inspection image. Further, the center points a2, b2 and c2 are not shown.

  In the example shown in FIG. 4F, the position coordinates of the center points a1, b1, c1, and d1 do not match the position coordinates of the center points a2, b2, c2, and d2. Therefore, the determination unit 11 determines that there is a difference between the detection image and the inspection image. That is, the determination unit 11 determines that a shift has occurred in the relative position of the display / light sensor unit 300 with respect to the display unit 310.

  The calculation unit 12 calculates a translational shift amount (dx, dy) of the relative position of the display / light sensor unit 300 with respect to the display unit 310 based on the shift between the position coordinates of the center point d1 and the position coordinates of the center point d2. . In addition, the display / light sensor for the display unit 310 is based on the angle deviation between the vector having the center point a1 as the start point, the vector having the center point c1 as the end point, and the vector having the center point c2 as the start point and the center point c2 as the end point. The rotational deviation amount θ of the relative position of the unit 300 is calculated. That is, the “deviation amount” in this specification and the like refers to the translation deviation amount (dx, dy) of the position coordinates between the center points d1 and d2, and the vector and the center point a2 having the center point c1 as the end point. , And a rotational deviation amount θ with respect to a vector having the center point c2 as an end point.

  The calculation unit 12 determines whether or not the data display / sensor device 100 is a defective product based on the calculated translational deviation amounts (dx, dy) and the rotational deviation amount θ. At this time, it is preferable that a threshold value is set for each of the translational deviation amounts (dx, dy) and the rotational deviation amount θ.

  The calculation unit 12 determines the data display / sensor device 100 as a defective product when at least one of the calculated translational deviation amount (dx, dy) and the rotational deviation amount θ exceeds a predetermined threshold. Alternatively, when both the calculated translational deviation amount (dx, dy) and the rotational deviation amount θ exceed a predetermined threshold value, it may be determined that the data display / sensor device 100 is defective.

  When the shapes of the display / light sensor unit 300 and the display unit 310 are different, the inspection mark in the inspection image displayed on the display unit 310 is displayed at a position that can be detected by the display / light sensor unit 300. It is preferable to do.

(Details of detection image correction method)
The correction unit 13 corrects the detection image detected by the display / light sensor unit 300 based on the shift amount calculated by the calculation unit 12 and generates corrected image data. More specifically, first, the correction unit 13 converts the position coordinates of each pixel in the detection image detected by the display / light sensor unit 300 using the following formula (1).

  Here, in Expression (1), coordinates (x1, y1) are position coordinates of an arbitrary pixel in the detection image detected by the display / light sensor unit 300, and coordinates (x2, y2) are coordinates (x1, y2). y1) is a position coordinate after conversion, (dx, dy) is a translational deviation amount, and θ is a rotational deviation amount.

  The equation (1) is an equation for detecting and correcting the image displayed on the display unit 310 in the upper housing 40 by the display / light sensor unit 300 in the lower housing 41. For example, the display unit 300 in the upper housing 40 is also provided with an optical sensor, and an image (or a graphic placed on the lower housing 41) displayed on the display / light sensor unit 300 in the lower housing 41 is displayed in the upper housing. In the case of detection and correction by the optical sensor in the body 40, the correction direction is reversed. Therefore, it is necessary to reverse the signs of θ and (dx, dy) in the above equation (1). In this case, the correction unit 13 converts the position coordinates of each pixel using the following equation (2).

  The correction unit 13 applies the color of the pixel at the coordinates (x1, y1) to the pixel at the coordinates (x2, y2) converted using the above formula (1) or (2). Accordingly, the correction unit 13 can create corrected image data obtained by correcting the detection image detected by the optical sensor of the display / light sensor unit 300. Further, by converting the position coordinates in the corrected image data so that the correction image is displayed upside down in the conversion unit 14, the correction in consideration of the relative position shift between the display / light sensor unit 300 and the display unit 310 is performed. Images can also be displayed.

  Note that when the shift amount is recorded in the storage unit 901, the processing in the determination unit 11 and the calculation unit 12 can be skipped. In this case, the correction unit 13 may generate corrected image data by correcting the detected image detected by the sensor built-in liquid crystal panel 301 based on the shift amount recorded in the storage unit 901.

(Details of correction of detected image)
Here, a specific example of correction of the detected image detected by the display / light sensor unit 300 will be described below with reference to FIGS. FIGS. 5A to 5E are diagrams showing specific examples of correction of the detected image detected by the display / light sensor unit 300. FIG. 5A is a display image displayed on the display unit 310. b) is a detection image detected by the display / light sensor unit 300, (c) is a diagram showing correction of the detection image, (d) is a detection image after correction, and (e) is after correction. It is a figure which shows the state which reversed the detected image of the top and bottom.

  First, the image display control unit 16 displays an “R” image as shown in FIG. 5A on the liquid crystal panel 311 of the display unit 310. Subsequently, the sensor built-in liquid crystal panel 301 of the display / light sensor unit 300 detects the image shown in FIG. At this time, as shown in FIG. 5B, an image having a mirror image relationship with the display image shown in FIG. 5A is detected on the sensor built-in liquid crystal panel 301 of the display / light sensor unit 300. Further, in the detection image detected by the display / light sensor unit 300, as shown in FIG. 5B, a shift due to a shift in the relative position between the display / light sensor unit 300 and the display unit 310 occurs. ing.

  As shown in FIG. 5C, the correction unit 13 determines the position coordinates of the pixels in the detected image based on the translational deviation amounts (dx, dy) and the rotational deviation amount θ recorded in the storage unit 901. Convert. At this time, the correction unit 13 converts the position coordinates of the pixel using the above equation (1). FIG. 5D shows an image obtained by converting the pixel position coordinates using the above equation (1).

  However, as shown in FIG. 5D, only by correcting based on the shift amount, the display image displayed on the liquid crystal panel 311 of the display unit 310 is an image having a mirror image relationship, and the letter “R” is a mirror image. It is in a character state. Therefore, the conversion unit 15 converts the position coordinates so that the corrected image is displayed as shown in FIG. That is, the conversion unit 15 flips the corrected image upside down. As a result, an image similar to the image displayed on the liquid crystal panel 311 of the display unit 310 can be displayed on the sensor built-in liquid crystal panel 301 of the display / light sensor unit 300. Of course, the image shown in FIG. 5E may be displayed on the liquid crystal panel 311 of the display unit 310.

(Application to bright spot detection)
When calculating the shift amount indicating the shift of the relative position between the display / light sensor unit 300 and the display unit 310, the display / light sensor unit 300 in the data display / sensor device 100 and the bright spot in the display unit 310 are calculated. Can be detected. That is, the data display / sensor device 100 determines whether or not there is a defective portion in the optical sensor of the display / light sensor unit 300 or whether there is a missing dot in the liquid crystal panel of the display unit 310. be able to. The detection of bright spots in the display / light sensor unit 300 and the display unit 310 will be described below with reference to FIG. FIG. 6 is a flowchart showing a bright spot detection method in the display / light sensor unit 300 and the display unit 310.

  First, when the data display / sensor device 100 is folded and the display / light sensor unit 300 provided in the upper housing 40 and the display unit 310 provided in the lower housing 41 face each other, the image display control unit 16 displays a predetermined inspection image (black image) on the liquid crystal panel 311 of the display unit 310 (step S10). When the inspection image is displayed on the liquid crystal panel 311, the optical sensor provided in the sensor built-in liquid crystal panel 301 of the display light sensor unit 300 detects the inspection image displayed on the liquid crystal panel 311 (step S11). .

  Subsequently, the detection unit 14 determines whether or not there is a pixel having a luminance equal to or higher than a predetermined threshold Th2 in the detected detection image (step S12). In the detected image that has been detected, when there is no pixel whose luminance is equal to or greater than the threshold Th2 (NO in step S12), the detection unit 14 has a defect in the optical sensor in the display / light sensor unit 300 and missing dots in the display unit 310. It judges that it does not carry out, and complete | finishes an inspection process.

  On the other hand, in the detected image, if there is a pixel whose luminance is greater than or equal to the threshold Th2 (YES in step S12), the detection unit 14 has a defect in the optical sensor of the display / light sensor unit 300 or displays It is determined that there is a missing dot in the liquid crystal panel 311 of the unit 310, and the position coordinates of a pixel having a luminance value equal to or greater than the threshold Th2 is detected (step S13).

  Next, the correction unit 13 converts the position coordinates of a pixel having a luminance value equal to or higher than the threshold Th2 based on the shift amount of the data display / sensor device 100 recorded in the storage unit 901 (step S14). The conversion unit 15 converts the position coordinates of the pixels detected by the detection unit 14 into position coordinates on the display unit 310 (step S15).

  Finally, the marker display control unit 18 displays a marker indicating the pixel indicated by the position coordinates converted by the conversion unit 15 on the liquid crystal panel 311 (step S16).

  Note that whether or not a bright spot has occurred in the pixel indicated by the marker is determined by visual inspection by an inspector. That is, when a bright spot is generated in the pixel on which the marker is displayed on the liquid crystal panel 311 of the display unit 310, the inspector detects that the bright spot detected in the detected image is a missing dot in the liquid crystal panel 311 of the display unit 310. Judge that it is caused. On the other hand, when the pixel on which the marker is displayed on the liquid crystal panel 311 of the display unit 310 does not have a bright spot, the inspector determines that the bright spot detected in the detected image is due to a defective optical sensor in the display / photosensor unit 300. It is determined that

  If a plurality of pixels are detected in step S13, the above processing is executed for all the detected pixels. That is, the inspector detects in step S13 whether the bright spot is caused by a defect in the optical sensor of the sensor built-in liquid crystal panel 301 or caused by missing dots in the liquid crystal panel 311. All the pixels are determined.

  Note that the image display control unit 16 is configured to display the white image in a state where the data display / sensor device 100 is folded and the display / light sensor unit 300 included in the upper housing 40 and the display unit 310 included in the lower housing 41 face each other. Can be detected as an inspection image to detect not a bright spot but a black spot. However, in this case, the detection unit 14 detects a pixel having a luminance equal to or lower than a predetermined threshold as a black point.

(Details of bright spot detection)
Next, details of the bright spot detection method will be described below with reference to FIGS. FIGS. 7A to 7F are diagrams showing specific examples of a method for detecting a bright spot in the sensor built-in liquid crystal panel 301 of the display / light sensor unit 300 and the liquid crystal panel 311 of the display unit 310. FIG. It is a figure which shows the image for an inspection, (b) is a figure which shows a detection image, (c) is a figure which shows correction | amendment of the position in a detection image, (d) is a figure which shows correction | amendment of the angle in a detection image. (E) is a diagram showing rotation of the orientation of the corrected image, and (f) is a diagram showing marker display on the liquid crystal panel 311.

  First, the image display control unit 16 displays an inspection image as shown in FIG. 7A on the liquid crystal panel 311 of the display unit 310. Subsequently, the sensor built-in liquid crystal panel 301 of the display / light sensor unit 300 detects an inspection image as shown in FIG. Here, the bright spots detected by the detector 14 are shown as white dots in FIG.

  As illustrated in FIG. 7C, the correction unit 13 converts the position coordinates of the pixels that are the bright spots in the detected image based on the translational deviation amounts (dx, dy) recorded in the storage unit 901. At the same time, as shown in FIG. 7D, based on the rotational shift amount θ recorded in the storage unit 901, the position coordinates of the pixels detected as bright spots in the detected image are converted. At this time, the correction unit 13 converts the position coordinates of the pixel using the above equation (1). In FIGS. 7C and 7D, the position coordinates are converted based on the rotational shift amount θ after the position coordinates are converted based on the translational shift amounts (dx, dy). However, the present invention is not limited to this. Instead, the position coordinates of the pixels detected as the bright spots in the detected image may be converted in the reverse order.

  Next, as illustrated in FIG. 7E, the position coordinates of the corrected image data generated in the correction unit 13 are converted into position coordinates in the display unit 310, and an image based on the converted position coordinates is displayed on the display unit 310. Displayed on the liquid crystal panel 311. In other words, the direction of the corrected detection image is inverted and displayed on the liquid crystal panel 311 of the display unit 310. At the same time, the marker display control unit 18 displays, on the liquid crystal panel 311, markers that indicate position coordinates corresponding to the position coordinates in the sensor built-in liquid crystal panel 301, as shown in FIG. That is, it indicates a pixel where a bright spot will be displayed. At this time, as shown in FIG. 7F, a message asking the inspector whether or not a bright spot is visible at the position indicated by the marker may be displayed at the same time.

  Note that, similarly to the detection of the displacement of the relative position of the display / light sensor unit 300 with respect to the display unit 310, the detection of the bright spot is also performed by the user pressing a switch provided in the data display / sensor device 100. Only when the sensor device 100 is switched to the inspection mode, the bright spot detection process can be performed. The switch for setting the data display / sensor device 100 to the bright spot detection mode is not limited to a hardware switch, and may be a software switch.

  Further, in this section, the case where the bright spot detection is performed after the inspection of the relative position deviation between the display / light sensor unit 300 and the display unit 310 is illustrated, but the present invention is not limited thereto. Instead, the bright spot detection process may be performed by an inspection process of a relative position shift between the display / light sensor unit 300 and the display unit 310 and a series of processes.

  The image shown in FIG. 7E on the liquid crystal panel 311 of the display unit 310 is changed from the inspection image when the inspector opens the data display / sensor device 100 in order to confirm the pixel indicated by the marker. The image may be displayed by changing the image for inspection by pressing an image switching switch.

  It should be noted that black dots can also be detected by using a test image as shown in FIG. 7A as a white image. In this case, the white point in FIG. 7B becomes a black point, and the background color changes from black to white.

  As described above, by performing the above-described operation, a failure of the optical sensor in the display / photosensor unit 300 or missing dots in the display unit 310 can be detected by the own device without using another inspection device. Can do.

  Further, by displaying a marker indicating the position coordinate on the liquid crystal panel 311 of the display unit 310 corresponding to the position coordinate of the bright spot detected in the detected image, the bright spot is displayed / the sensor built-in liquid crystal panel 301 of the optical sensor unit 300. It is possible to determine whether it is caused by a defect of the photosensor or by a missing dot of the liquid crystal panel 311 of the display unit 310.

  Thus, the inspector can easily and surely determine the defect of the optical sensor in the display / light sensor unit 300 or the missing dot of the liquid crystal panel 311 in the display unit 310. Further, by displaying the marker on the display unit 310, the inspector can more easily determine whether or not there is a bright spot on the display unit 310.

(Modification of Data Display / Sensor Device 100)
When the data display / sensor device 100 is applied to a scanner capable of simultaneously scanning both sides of a document sandwiched between folded housings, the upper housing 40 is scanned using the calculated displacement amount. It is possible to perform alignment between the captured image and the image scanned in the lower housing 41.

  When the data display / sensor device 100 is a scanner capable of simultaneously scanning both sides of the scan target, the display unit 310 in the upper housing 40 may have the same configuration as the display / light sensor unit 300. preferable. An example of the configuration of such a data display / sensor device is shown in FIG. FIG. 8 is a block diagram showing a main configuration of a data display / sensor device 100b which is a modification of the data display / sensor device 100. As shown in FIG. In FIG. 8, the determination unit 11, the calculation unit 12, the detection unit 14, and the marker display control unit 18 illustrated in FIG. 1 are not illustrated for convenience.

  Further, in this section, it is assumed that the first display / light sensor unit 300A is provided in the lower housing 41, and the second display / light sensor unit 300B is provided in the upper housing 40. The first display / light sensor unit 300A includes a sensor built-in liquid crystal panel 301A, and the display / sensor unit 300B includes a sensor built-in liquid crystal panel 301B.

  In this specification and the like, when the first display / light sensor unit 300A and the second display / light sensor unit 300B are not distinguished from each other, they are simply referred to as the display / light sensor unit 300, and the sensor built-in liquid crystal panel 301A and the sensor When the built-in liquid crystal panel 301B is not distinguished, it is simply referred to as a sensor built-in liquid crystal panel 301.

  FIG. 9 shows details of a method of aligning the detection image detected in the second display / light sensor unit 300B in the upper housing 40 and the detection image detected in the first display / light sensor unit 300A in the lower housing 41. The following will be described with reference to FIG. FIG. 9 is a flowchart showing a correction method for correcting the detected image detected by the second display / light sensor unit 300B.

  Here, “W” is the number of pixels in the horizontal direction of the image detected in the second display / light sensor unit 300B, and “H” is the number of pixels in the vertical direction of the image detected in the second display / light sensor unit 300B. It is. Hereinafter, an alignment method for aligning the detection image detected by the second display / light sensor unit 300B and the detection image detected by the first display / light sensor unit 300A will be described.

  First, the correction unit 13 initializes the value of y1 (step S20), and then initializes the value of x1 (step S21). That is, x1 = 0 and y1 = 0 are set. Next, the correction unit 13 calculates the coordinates (x2, y2) from the coordinates (x1, y1) in the image detected by the second display / light sensor unit 300B using the above formula (2) (step S22). . Subsequently, the correcting unit 13 determines whether the calculated x2 and y2 satisfy 0 ≦ x2 ≦ W and 0 ≦ y2 ≦ H (step S23).

  When the calculated x2 and y2 satisfy 0 ≦ x2 ≦ W and 0 ≦ y2 ≦ H (YES in step S23), the correction unit 13 determines the luminance information of the pixel at the coordinates (x2, y2) as coordinates It is set to be the same as the luminance information of the pixel (x1, y1) (step S24). On the other hand, when the calculated x2 and y2 do not satisfy 0 ≦ x2 ≦ W and 0 ≦ y2 ≦ H (NO in step S23), the correction unit 13 obtains the luminance information of the pixel at the coordinates (x2, y2). The same luminance information as the background portion is set (step S25).

  Next, the correction unit 13 adds “1” to the value of x1 (step S26). Then, it is determined whether or not the value of x1 obtained by adding “1” matches the value of the number of pixels W in the horizontal direction (step S27). When the value of x1 does not coincide with the value of W (NO in step S27), the correction unit 13 executes the processing from step S22 again.

  On the other hand, when the value of x1 matches the value of W (YES in step S27), the correction unit 13 adds “1” to the value of y1 (step S28). Then, it is determined whether or not the value of y1 obtained by adding “1” matches the value of the number of pixels H in the vertical direction (step S29).

  When the value of y1 does not match the value of H (NO in step S29), the correction unit 13 executes the processing from step S21 again. By repeating this, the correction unit 13 generates corrected image data of the image detected by the second display / light sensor unit 300B. On the other hand, if the value of y1 matches the value of H (YES in step S29), the conversion unit 15 displays the position of each pixel so that the corrected image indicated by the generated corrected image data is displayed upside down. The coordinates are converted (step S30).

  The image corrected by the above-described processing may be displayed on the second display / light sensor unit 300B in the upper housing 40, or may be displayed on the first display / light sensor unit 300A in the lower housing 41. It may be displayed.

(Details of alignment processing)
Next, a specific example of a correction method for correcting the detection image detected in the second display / light sensor unit 300B in the upper housing 40 will be described below with reference to FIGS. 10 (a) to 10 (g). FIGS. 10A to 10G are diagrams illustrating a specific example of a correction method for correcting a detection image detected by the second display / light sensor unit 300B. FIG. 10A illustrates a document printed on both sides. It is the figure mounted on the lower housing | casing 41, (b) is the detection image detected in the 2nd display / light sensor part 300B, (c) is the detection image detected in the 1st display / light sensor part 300A. (D) is a figure which shows the state which correct | amends the detection image detected in the 2nd display / light sensor part 300B, (e) corrected the detection image detected in the 2nd display / light sensor part 300B. (F) is an image obtained by vertically inverting the corrected image, and (g) is an image obtained by horizontally inverting the detected image detected by the first display / light sensor unit 300A.

  In this section, as shown in FIGS. 10A to 10G, “R” is printed on the side detected by the second display / light sensor unit 300B, and the first display / light sensor unit An example will be described in which a document on which “Q” is printed on the side detected in 300A is read.

  First, as shown in FIG. 10A, a document on which characters are printed on both sides is placed on the lower housing 41. Subsequently, the second display / light sensor unit 300B detects the letter “R” shown in FIG. At this time, the data display / sensor device 100 also acquires values of the number of pixels “W” in the horizontal direction and the number of pixels “H” in the vertical direction of the detected image. As shown in FIG. 10B, the image of the letter “R” detected by the second display / light sensor unit 300B has a mirror image relationship with the image of the letter “R” shown in FIG. A shift due to a shift in relative position between the first display / light sensor unit 300A and the second display / light sensor unit 300B occurs.

  Simultaneously with the detection processing of the image of the letter “R” in the second display / light sensor unit 300B, the first display / light sensor unit 300A also detects the image of the character “Q”. The letter “Q” detected by the first display / light sensor unit 300A is in a mirror character state as shown in FIG.

  Next, as shown in FIG. 10D, the correction unit 13 corrects the deviation based on the deviation amount calculated in advance, and the first display / light sensor unit 300A and the second display / light sensor unit 300B. The image shift due to the relative position shift with respect to is corrected. At this time, the correction unit 13 corrects the image shift shown in FIG. 10B by using the above equation (2). FIG. 10E shows an image of the letter “R” obtained by converting the pixel position coordinates using the above equation (2).

  In this state, since the character “R” detected in the second display / light sensor unit 300B is a mirror character, the conversion unit 15 is displayed as shown in FIG. The corrected image position coordinates shown in 10 (e) are converted. Similarly, the conversion unit 15 converts the position coordinates of the image shown in FIG. 10C so as to be displayed as shown in FIG.

  As described above, when the data display / sensor device 100 is applied to a scanner capable of scanning both sides simultaneously, by applying the shift amount calculated based on the shift amount calculation process described above, the lower housing It is possible to correct a positional deviation caused by a relative positional deviation of the second display / light sensor unit 300B in the upper housing 40 with respect to the first display / light sensor unit 300A in 41. Accordingly, even when the relative position of the second display / light sensor unit 300B is shifted from the first display / light sensor unit 300A, the image detected by the first display / light sensor unit 300A and the first The image can be stored in a state where the front and back of the image detected by the 2 display / light sensor unit 300B are not shifted.

  In this section, the case of correcting the relative position shift of the second display / light sensor unit 300B with respect to the first display / light sensor unit 300A is taken as an example, but the reverse is also true.

  The scan may be executed at the same time as the data display / sensor device 100 is folded, or may be executed when the user presses a switch provided in the data display / sensor device 100. Note that the switch for executing the scan is not limited to the hardware switch, and may be a software switch.

(Other examples of inspection)
As described above, according to the data display / sensor device 100 and the data display / sensor device 100b, the shift of the display / light sensor unit 300 with respect to the display unit 310 can be detected, and the shift can be corrected. Bright spot defects and black spot defects of the part 310 and the display / light sensor part 300 can also be detected.

  However, the defects that can be detected by the data display / sensor device 100 and the data display / sensor device 100b are not limited to the above example. That is, in the data display / sensor device 100 and the data display / sensor device 100b, any defect is detected as long as it is a defect that can be detected based on a detection image obtained by scanning the display unit 310 with the display / light sensor unit 300. It is possible.

  For example, it is possible to determine whether or not a line defect has occurred in the display unit 310 by displaying an inspection image for detecting a line defect on the display unit 310. Also, for example, by displaying a moving image as an inspection image on the display unit 310, it is possible to determine the display performance of the moving image. Further, for example, it is possible to determine whether the brightness and color tone of the detected image obtained by scanning the inspection image is normal.

  In the above example, the example in which the inspector opens and closes the housing of the data display / sensor device 100 has been described. However, the inspection apparatus executes a part or all of the processing performed by the inspector. Is also possible. For example, the processing performed by the inspector can be reduced by folding the housing of the data display / sensor device 100 and causing the inspection device to perform an operation of opening the housing after scanning of the inspection image is completed. Inspection time and labor costs can also be reduced.

  Furthermore, the data display / sensor device 100 and the inspection device are configured to be communicable, and the detection unit 14 transmits the presence / absence of detection of the defect to the inspection device, so that the inspection unit can detect the defect. Only the detected data display / sensor device 100 can be used for an inspector's visual inspection. As a result, the number of data display / sensor devices 100 on which the inspector conducts visual inspection can be minimized, so that the inspection time and labor costs can be reduced.

(Outline of LCD panel with built-in sensor)
Here, an outline of the sensor built-in liquid crystal panel 301 included in the data display / sensor device 100 according to the present invention will be described.

  The sensor built-in liquid crystal panel 301 provided in the data display / sensor device 100 is a liquid crystal panel capable of detecting an image of an object in addition to displaying data. Here, the image detection of the object is, for example, detection of a position pointed (touched) by the user with a finger or a pen, or reading (scanning) of an image of a printed material or the like. The device used for display is not limited to a liquid crystal panel, and may be an organic EL (Electro Luminescence) panel or the like.

  The structure of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 11 is a diagram schematically showing a cross section of the sensor built-in liquid crystal panel 301. The sensor built-in liquid crystal panel 301 described here is an example, and any structure can be used as long as the display surface and the reading surface are shared.

  As shown in the figure, the sensor built-in liquid crystal panel 301 includes an active matrix substrate 51A disposed on the back surface side and a counter substrate 51B disposed on the front surface side, and has a structure in which a liquid crystal layer 52 is sandwiched between these substrates. is doing. The active matrix substrate 51A is provided with a pixel electrode 56, a data signal line 57, an optical sensor circuit 32 (not shown), an alignment film 58, a polarizing plate 59, and the like. The counter substrate 51B is provided with color filters 53r (red), 53g (green), 53b (blue), a light shielding film 54, a counter electrode 55, an alignment film 58, a polarizing plate 59, and the like. In addition, a backlight 307 is provided on the back surface of the sensor built-in liquid crystal panel 301.

  The photodiode 6 included in the photosensor circuit 32 is provided in the vicinity of the pixel electrode 56 provided with the blue color filter 53b, but is not limited to this configuration. It may be provided in the vicinity of the pixel electrode 56 provided with the red color filter 53r, or may be provided in the vicinity of the pixel electrode 56 provided with the green color filter 53g.

  Next, with reference to FIGS. 12A and 12B, two types of methods for detecting a position where the user touches the sensor built-in liquid crystal panel 301 with a finger or a pen will be described.

  FIG. 12A is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a reflected image. When the light 63 is emitted from the backlight 307, the optical sensor circuit 32 including the photodiode 6 detects the light 63 reflected by the object 64 such as a finger. Thereby, the reflected image of the target object 64 can be detected. Thus, the sensor built-in liquid crystal panel 301 can detect the touched position by detecting the reflected image.

  FIG. 12B is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a shadow image. As shown in FIG. 12B, the optical sensor circuit 32 including the photodiode 6 detects the external light 61 transmitted through the counter substrate 51B and the like. However, when there is an object 62 such as a pen, the incident of the external light 61 is hindered, so that the amount of light detected by the optical sensor circuit 32 is reduced. Thereby, a shadow image of the object 62 can be detected. Thus, the sensor built-in liquid crystal panel 301 can also detect a touched position by detecting a shadow image.

  As described above, the photodiode 6 may detect reflected light (shadow image) of the light emitted from the backlight 307 or may detect a shadow image caused by external light. Further, the two types of detection methods may be used in combination to detect both a shadow image and a reflected image at the same time.

(Data display / sensor configuration)
Next, the configuration of the main part of the data display / sensor device 100 will be described with reference to FIG. FIG. 13 is a block diagram showing a main configuration of the data display / sensor device 100. As illustrated, the data display / sensor device 100 includes one or more display / light sensor units 300, a circuit control unit 600, a data processing unit 700, a main control unit 800, a storage unit 901, a primary storage unit 902, and an operation unit. 903, an external communication unit 907, an audio output unit 908, and an audio input unit 909. Here, the data display / sensor device 100 will be described as including two display / light sensor units 300 (first display / light sensor unit 300A and second display / light sensor unit 300B). When the first display / light sensor unit 300A and the second display / light sensor unit 300B are not distinguished, they are referred to as the display / light sensor unit 300.

  In FIG. 1, each block included in the main control unit 800 has been described in detail. However, in this section, configurations other than the blocks included in the main control unit 800 will be described in detail.

  The display / light sensor unit 300 is a so-called liquid crystal display device with a built-in light sensor. The display / light sensor unit 300 includes a sensor built-in liquid crystal panel 301, a backlight 307, and a peripheral circuit 309 for driving them.

  The sensor built-in liquid crystal panel 301 includes a plurality of pixel circuits 31 and photosensor circuits 32 arranged in a matrix. The detailed configuration of the sensor built-in liquid crystal panel 301 will be described later.

  The peripheral circuit 309 includes a liquid crystal panel drive circuit 304, an optical sensor drive circuit 305, a signal conversion circuit 306, and a backlight drive circuit 308.

  The liquid crystal panel driving circuit 304 outputs a control signal (G) and a data signal (S) in accordance with the timing control signal (TC1) and the data signal (D) from the display control unit 601 of the circuit control unit 600, and the pixel circuit 31. It is a circuit which drives. Details of the driving method of the pixel circuit 31 will be described later.

  The optical sensor driving circuit 305 is a circuit that drives the optical sensor circuit 32 by applying a voltage to the signal line (R) in accordance with a timing control signal (TC2) from the sensor control unit 602 of the circuit control unit 600. Details of the driving method of the optical sensor circuit 32 will be described later.

  The signal conversion circuit 306 is a circuit that converts the sensor output signal (SS) output from the optical sensor circuit 32 into a digital signal (DS) and transmits the converted signal to the sensor control unit 602.

  The backlight 307 includes a plurality of white LEDs (Light Emitting Diodes) and is disposed on the back surface of the sensor built-in liquid crystal panel 301. When a power supply voltage is applied from the backlight drive circuit 308, the backlight 307 is turned on and irradiates the sensor built-in liquid crystal panel 301 with light. Note that the backlight 307 is not limited to white LEDs, and may include LEDs of other colors. The backlight 307 may include, for example, a cold cathode fluorescent lamp (CCFL) instead of the LED.

  The backlight driving circuit 308 applies a power supply voltage to the backlight 307 when the control signal (BK) from the backlight control unit 603 of the circuit control unit 600 is at a high level, and conversely, the backlight control unit 603. When the control signal from is at a low level, no power supply voltage is applied to the backlight 307.

  Next, the circuit control unit 600 will be described. The circuit control unit 600 has a function as a device driver that controls the peripheral circuit 309 of the display / light sensor unit 300. The circuit control unit 600 includes a display control unit 601, a sensor control unit 602, a backlight control unit 603, and a display data storage unit 604.

  Next, the circuit control unit 600 will be described. The circuit control unit 600 has a function as a device driver that controls the peripheral circuit 309 of the display / light sensor unit 300. The circuit control unit 600 includes a display control unit 601, a sensor control unit 602, a backlight control unit 603, and a display data storage unit 604.

  The display control unit 601 receives display data from the display data processing unit 701 of the data processing unit 700, and performs timing control on the liquid crystal panel driving circuit 304 of the display / light sensor unit 300 in accordance with an instruction from the display data processing unit 701. A signal (TC1) and a data signal (D) are transmitted, and the received display data is displayed on the sensor built-in liquid crystal panel 301.

  The display control unit 601 temporarily stores the display data received from the display data processing unit 701 in the display data storage unit 604. Then, a data signal (D) is generated based on the primary stored display data. The display data storage unit 604 is, for example, a video random access memory (VRAM).

  The sensor control unit 602 transmits a timing control signal (TC2) to the optical sensor driving circuit 305 of the display / optical sensor unit 300 in accordance with an instruction from the sensor data processing unit 703 of the data processing unit 700, and the sensor built-in liquid crystal panel 301. Run the scan with.

  In addition, the sensor control unit 602 receives a digital signal (DS) from the signal conversion circuit 306. Then, image data is generated based on the digital signal (DS) corresponding to the sensor output signal (SS) output from all the optical sensor circuits 32 included in the sensor built-in liquid crystal panel 301. That is, the image data read in the entire reading area of the sensor built-in liquid crystal panel 301 is generated. Then, the generated image data is transmitted to the sensor data processing unit 703.

  The backlight control unit 603 transmits a control signal (BK) to the backlight drive circuit 308 of the display / light sensor unit 300 in accordance with instructions from the display data processing unit 701 and the sensor data processing unit 703 to drive the backlight 307. Let

  When the data display / sensor device 100 includes a plurality of display / light sensor units 300, the display control unit 601 determines which display / light sensor unit 300 displays the display data from the data processing unit 700. When an instruction is received, the liquid crystal panel drive circuit 304 of the display / light sensor unit 300 is controlled according to the instruction. When the sensor control unit 602 receives an instruction from the data processing unit 700 as to which display / light sensor unit 300 is to scan the object, The sensor drive circuit 305 is controlled and a digital signal (DS) is received from the signal conversion circuit 306 of the display / light sensor unit 300 according to the instruction.

  Next, the data processing unit 700 will be described. The data processing unit 700 has a function as middleware that gives an instruction to the circuit control unit 600 based on a “command” received from the main control unit 800. Details of the command will be described later.

  The data processing unit 700 includes a display data processing unit 701 and a sensor data processing unit 703. When the data processing unit 700 receives a command from the main control unit 800, at least one of the display data processing unit 701 and the sensor data processing unit 703 depends on the value of each field (described later) included in the received command. One works.

  The display data processing unit 701 receives display data from the main control unit 800, and gives instructions to the display control unit 601 and the backlight control unit 603 according to the command received by the data processing unit 700, and displays the received display data. The image is displayed on the sensor built-in liquid crystal panel 301. The operation of the display data processing unit 701 according to the command will be described later.

  Note that the image display control unit 16, the corrected image display control unit 17, and the marker display control unit 18 in FIG. 1 are all functional blocks included in the display control unit 601. That is, in the operation of the data display / sensor device 100 described above, the description of the operation of the data processing unit 700 located between the main control unit 800 and the display / light sensor unit 300 is omitted.

  That is, to be precise, when displaying data and scanning an object, each unit of the main control unit 800 transmits a command to the data processing unit 700, and the data processing unit 700 performs circuit control based on the total. The circuit control unit 600 transmits a signal to the display / light sensor unit 300. In addition, the main control unit 800 receives the entire image data, partial image data (detected image), and coordinate data (position coordinates in the detected image) from the data processing unit 700 as a response to the command transmitted to the data processing unit 700. get.

  The sensor data processing unit 703 gives an instruction to the sensor control unit 602 and the backlight control unit 603 according to the command received by the data processing unit 700.

  The sensor data processing unit 703 receives image data from the sensor control unit 602 and stores the image data in the image data buffer 704 as it is. Then, in accordance with the command received by the data processing unit 700, the sensor data processing unit 703 performs “whole image data”, “partial image data (partial image coordinate data) based on the image data stored in the image data buffer 704. At least one of “including coordinate data” and “coordinate data” is transmitted to the main control unit 800. The whole image data, partial image data, and coordinate data will be described later. The operation of the sensor data processing unit 703 according to the command will be described later.

  Next, the main control unit 800 executes an application program. The main control unit 800 reads the program stored in the storage unit 901 into a primary storage unit 902 configured by, for example, a RAM (Random Access Memory) and executes the program.

  An application program executed by the main control unit 800 causes the data processing unit 700 to display display data on the sensor built-in liquid crystal panel 301 or to scan an object on the sensor built-in liquid crystal panel 301. Send commands and display data. When “data type” is designated as a command, at least one of whole image data, partial image data, and coordinate data is received from the data processing unit 700 as a response to the command.

  The circuit control unit 600, the data processing unit 700, and the main control unit 800 can be configured by a CPU (Central Processing Unit), a memory, and the like, respectively. The data processing unit 700 may be configured by a circuit such as an ASIC (application specific integrate circuit).

  Next, the storage unit 901 stores programs and data executed by the main control unit 800 as shown in the figure. The program executed by the main control unit 800 may be separated into an application-specific program and a general-purpose program that can be shared by each application.

  Next, the operation unit 903 receives an input operation of the user of the data display / sensor device 100. The operation unit 903 includes input devices such as a switch, a remote controller, a mouse, and a keyboard, for example. Then, the operation unit 903 generates a control signal corresponding to the user's input operation of the data display / sensor device 100, and transmits the generated control signal to the main control unit 800.

  As an example of the switch, a hinge switch 904 that is provided at the hinge portion of the housing and detects the open / closed state of the housing, a power switch 905 that switches power on and off, and a predetermined function are assigned in advance. A hardware switch such as a user switch 906 is assumed.

  In addition, the data display / sensor device 100 includes an external communication unit 907 for communicating with an external device by wireless / wired communication, an audio output unit 908 such as a speaker for outputting audio, and a microphone for inputting an audio signal. A voice input unit 909 such as the above may be provided as appropriate.

(Command details)
Next, details of commands transmitted from the main control unit 800 to the data processing unit 700 will be described with reference to FIGS. 14 and 15. FIG. 14 is a diagram schematically illustrating an example of a command frame structure. FIG. 15 is a diagram for explaining an example of values that can be specified for each field included in the command and an outline thereof.

  As shown in FIG. 14, the commands are “header”, “data acquisition timing”, “data type”, “scan method”, “scan image gradation”, “scan resolution”, “scan panel”, “display panel”. "And" Reserve "fields. In each field, for example, values shown in FIG. 15 can be designated.

  The “header” field is a field indicating the start of a frame. As long as it is possible to identify the “header” field, the value of the “header” field may be any value.

  Next, the “data acquisition timing” field is a field for designating a timing at which data should be transmitted to the main control unit 800. In the “data acquisition timing” field, for example, values “00” (sense), “01” (event), and “10” (all) can be specified.

  Here, “sense” designates that the latest data is transmitted immediately. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “sense”, the latest data specified in the “data type” field is immediately updated to the main control unit 800. Send to.

  The “event” designates transmission at a timing when a change occurs in the image data received from the sensor control unit 602. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “event”, the image that receives the data specified in the “data type” field from the sensor control unit 602. The data is transmitted to the main control unit 800 at a timing when a change larger than a predetermined threshold occurs.

  “All” designates data transmission at a predetermined cycle. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “all”, the data designated in the “data type” field is transferred to the main control unit 800 at a predetermined cycle. Send to. The predetermined period coincides with the period in which the optical sensor circuit 32 performs scanning.

  Next, the “data type” field is a field for designating the type of data acquired from the sensor data processing unit 703. In the “data type” field, for example, values of “001” (coordinates), “010” (partial image), and “100” (entire image) can be specified. Furthermore, by adding these values, “coordinates” and “partial image” / “whole image” can be specified simultaneously. For example, when “coordinate” and “partial image” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose value of the “data type” field is “whole image”, the sensor data processing unit 703 transmits the image data itself stored in the image data buffer 704 to the main control unit 800. The image data itself stored in the image data buffer 704 is referred to as “whole image data”.

  In addition, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 selects a portion where a change larger than a predetermined threshold has occurred from the image data received from the sensor control unit 602. A region to be included is extracted, and image data of the extracted region is transmitted to the main control unit 800. Here, the image data is referred to as “partial image data”. When a plurality of partial image data are extracted, the sensor data processing unit 703 transmits each extracted partial image data to the main control unit 800.

  Further, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 detects representative coordinates in the partial image data and indicates the position of the representative coordinates in the partial image data. The coordinate data is transmitted to the main control unit 800. The representative coordinates include, for example, the coordinates of the center of the partial image data, the coordinates of the center of gravity of the partial image data, and the like.

  Next, when receiving a command whose value of the “data type” field is “coordinate”, the sensor data processing unit 703 transmits coordinate data indicating the position of the representative coordinate in the entire image data to the main control unit 800. When a plurality of partial image data are extracted, the sensor data processing unit 703 detects representative coordinates in the entire image data of the extracted partial image data, and the coordinate data indicating the representative coordinates is detected. Each is transmitted to the main control unit 800 (multi-point detection).

  Specific examples of the whole image data, the partial image data, and the coordinate data will be described later with reference to schematic diagrams.

  Next, the “scan method” field is a field for designating whether or not the backlight 307 is turned on at the time of executing the scan. In the “scan method” field, for example, values of “00” (reflection), “01” (transmission), and “10” (reflection / transmission) can be designated.

  “Reflection” designates that scanning is performed with the backlight 307 turned on. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “reflection”, the sensor data processing unit 703 performs sensor control so that the optical sensor driving circuit 305 and the backlight driving circuit 308 operate in synchronization. An instruction is given to the unit 602 and the backlight control unit 603.

  “Transmission” specifies that scanning is performed with the backlight 307 turned off. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “transparent”, the sensor control unit 602 operates the optical sensor driving circuit 305 and does not operate the backlight driving circuit 308. Instructions to the backlight control unit 603. Note that “reflection / transmission” specifies that scanning is performed using both “reflection” and “transmission”.

  Next, the “scanned image gradation” field is a field for designating gradations of the partial image data and the entire image data. In the “scanned image gradation” field, for example, values of “00” (binary) and “01” (multivalue) can be designated.

  When the sensor data processing unit 703 receives a command whose “scan image gradation” field value is “binary”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as monochrome data. .

  When the sensor data processing unit 703 receives a command whose “scanned image gradation” field value is “multivalued”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as multitone data. To do.

  Next, the “scan resolution” field is a field for designating the resolution of the partial image data and the entire image data. In the “scan resolution” field, for example, values of “0” (high) and “1” (low) can be specified.

  Here, “high” designates a high resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “high”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 with high resolution. For example, it is desirable to designate “high” for image data (image data such as a fingerprint) to be subjected to image processing such as image recognition.

  “Low” designates a low resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “low”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 at a low resolution. For example, it is desirable to designate “low” for image data (such as touched finger or hand image data) that only needs to be recognized.

  Next, the “scan panel” field is a field for designating which display / light sensor unit 300 is to scan the object. In the “scan panel” field, for example, values “001” (first display / light sensor unit 300A) and “010” (second display / light sensor unit 300B) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when “first display / light sensor unit 300A” and “second display / light sensor unit 300B” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose “scan panel” field value is “first display / photosensor unit 300A”, the sensor data processing unit 703 and the photosensor drive circuit 305 of the first display / photosensor unit 300A and An instruction is given to the sensor control unit 602 and the backlight control unit 603 so as to control the backlight drive circuit 308.

  Next, the “display panel” field is a field for designating which display / light sensor unit 300 displays the display data. In the “display panel” field, for example, values of “001” (first display / light sensor unit 300A) and “010” (second display / light sensor unit 300B) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when “first display / light sensor unit 300A” and “second display / light sensor unit 300B” are specified at the same time, “011” can be specified.

  Here, for example, when the display data processing unit 701 receives a command whose value of the “display panel” field is “first display / light sensor unit 300A”, the display data processing unit 701 displays the display data on the first display / light sensor unit 300A. Therefore, an instruction is given to the display control unit 601 and the backlight control unit 603 to control the liquid crystal panel driving circuit 304 and the backlight driving circuit 308 of the first display / light sensor unit 300A.

  Next, the “reserved” field is a field that is appropriately specified when it is necessary to further specify information other than information that can be specified in the above-described fields.

  Note that an application executed by the main control unit 800 does not need to use all the above-described fields when transmitting a command, and an invalid value (such as a NULL value) may be set for a field that is not used. .

  When the user wants to acquire coordinate data of a position touched with a finger or pen, a command specifying “coordinate” in the “data type” field is transmitted to the data processing unit 700. Therefore, it is desirable to specify “all” in the “data acquisition timing” field of the command to acquire coordinate data. Further, since it is only necessary to acquire coordinate data of the touched position, the scanning accuracy may not be high. Therefore, “low” may be specified as the value of the “scan resolution” field of the above command.

  Further, when “coordinate” is specified in the “data type” field of the command, for example, when the user touches the sensor built-in liquid crystal panel 301 with a plurality of fingers or pens at the same time, the coordinate data of the touched position is used. Can be acquired (multi-point detection).

  When acquiring image data of an object such as a document, a command specifying “whole image” in the “data type” field is transmitted to the data processing unit 700. Since it is common to perform a scan in a stationary state, it is not necessary to periodically perform the scan. Therefore, in this case, it is desirable to designate “sense” or “event” in the “data acquisition timing” field. When scanning an object such as a document, it is desirable that the scanning accuracy is high so that the user can easily read the characters. Therefore, it is desirable to designate “high” in the “scan resolution” field.

(Whole image data / Partial image data / Coordinate data)
Next, the whole image data, the partial image data, and the coordinate data will be described with reference to FIG. The image data shown in FIG. 16A is image data obtained as a result of scanning the entire sensor built-in liquid crystal panel 301 when the object is not placed on the sensor built-in liquid crystal panel 301. The image data shown in FIG. 16B is image data obtained as a result of scanning the entire sensor-equipped liquid crystal panel 301 when the user touches the sensor-equipped liquid crystal panel 301 with a finger.

  When the user touches the sensor built-in liquid crystal panel 301 with a finger, the amount of light received by the photosensor circuit 32 in the vicinity of the touch changes, so that the voltage output from the photosensor circuit 32 changes, and as a result, In the image data generated by the sensor control unit 602, the brightness of the pixel value of the portion touched by the user changes.

  In the image data shown in FIG. 16B, the brightness of the pixel value of the portion corresponding to the user's finger is higher than that in the image data shown in FIG. In the image data shown in FIG. 16B, the smallest rectangular area (area PP) that includes all pixel values whose lightness changes more than a predetermined threshold is “partial image data”.

  The image data indicated by the area AP is “whole image data”.

  Also, the coordinate data in the whole image data (area AP) of the representative coordinates Z of the partial image data (area PP) is (Xa, Ya), and the coordinate data in the partial image data (area PP) is (Xp, Yp). It is.

(Operation example of main control unit 800)
Next, an operation example of the main control unit 800 will be described with reference to commands transmitted from the main control unit 800 to the data processing unit 700. In this section, the data display / sensor device 100 includes two display / light sensor units, displays an inspection image on the upper screen (second display / light sensor unit 300B), and displays the lower screen (first screen). A case where an inspection image is detected in the display / light sensor unit 300A) will be described as an example.

  First, the determination unit 11 transmits display data set in advance as an inspection image to the data processing unit 700. At this time, the determination unit 11 transmits a command specifying the upper screen (“010”) as the value of the “display panel” field to the data processing unit 700.

  Next, in order to perform detection processing for detecting the inspection image displayed on the upper screen on the lower screen, the determination unit 11 designates “event” or “all” as the value of the “data acquisition timing” field. Then, the command specifying “coordinate” (“001”) or both “coordinate” and “partial image” (“011”) as the value of the “data type” field is transmitted to the data processing unit 700. To simply detect a shift in the relative position of the first display / light sensor unit 300A with respect to the second display / light sensor unit 300B, a command designating “coordinates” may be transmitted, and the detected detection image is shifted. When correcting based on the amount, a command designating “coordinates” and “partial image” may be transmitted.

  By setting the command as described above, the optical sensor circuit 32 of the sensor built-in liquid crystal panel 301 in the lower screen (first display / optical sensor unit 300A) can detect the image of the inspection mark displayed on the inspection image and Since at least one of the reflected images is detected, as a result, the image data received by the data processing unit 700 from the sensor control unit 602 changes.

  Therefore, the data processing unit 700 determines, as a response to the above-described command, the coordinate data of the part where the change has occurred (position coordinates of the detected image) or the partial image data (detected image) including the coordinate data in the partial image. Send to.

  When the determination unit 11 receives coordinate data (center points a1, b1, c1, and d1) or partial image data including coordinate data from the data processing unit 700, the coordinate data of the inspection mark in the used inspection image is used. Partial image data including is acquired.

  In addition, when correcting the detected image based on the amount of deviation calculated in advance, the correction unit 13 uses a command specifying “partial image” (“010”) as the value of the “data type” field, What is necessary is just to transmit to the part 700. Then, the correction unit 13 corrects the partial image data (detected image) received from the data processing unit 700 based on the shift amount, and generates corrected image data.

  When displaying the corrected image based on the corrected image data generated by the correcting unit 13, the conversion unit 15 that has converted the position coordinates of the corrected image data displays the upper screen (“010”) as the value of the “display panel” field. ) And the generated corrected image data are transmitted to the data processing unit 700. Accordingly, the corrected image can be displayed on the second display / light sensor unit 300B.

(Configuration of sensor built-in liquid crystal panel)
Next, the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309 of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 17 is a block diagram showing the main part of the display / light sensor unit 300, particularly the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309.

  The sensor built-in liquid crystal panel 301 includes a pixel circuit 31 for setting light transmittance (brightness) and an optical sensor circuit 32 that outputs a voltage corresponding to the intensity of light received by the sensor. The pixel circuit 31 is used as a general term for the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b corresponding to the red, green, and blue color filters, respectively.

  The pixel circuits 31 are arranged on the sensor built-in liquid crystal panel 301 in the column direction (vertical direction) and 3n in the row direction (horizontal direction). A set of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b is continuously arranged in the row direction (lateral direction). This set forms one pixel.

  In order to set the light transmittance of the pixel circuit 31, first, the high level voltage (TFT 33 is turned on) to the scanning signal line Gi connected to the gate terminal of the TFT (Thin Film Transistor) 33 included in the pixel circuit 31. Voltage). Thereafter, a predetermined voltage is applied to the data signal line SRj connected to the source terminal of the TFT 33 of the R pixel circuit 31r. Similarly, the light transmittance is also set for the G pixel circuit 31g and the B pixel circuit 31b. Then, by setting these light transmittances, an image is displayed on the sensor built-in liquid crystal panel 301.

  Next, the photosensor circuit 32 is arranged for each pixel. One pixel may be arranged in the vicinity of each of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b.

  In order for the optical sensor circuit 32 to output a voltage corresponding to the light intensity, first, the sensor readout line RWi connected to one electrode of the capacitor 35 and the anode terminal of the photodiode 36 are connected. A predetermined voltage is applied to the sensor reset line RSi. In this state, when light is incident on the photodiode 36, a current corresponding to the amount of incident light flows through the photodiode 36. Then, according to the current, the voltage at the connection point (hereinafter referred to as connection node V) between the other electrode of the capacitor 35 and the cathode terminal of the photodiode 36 decreases. When the power supply voltage VDD is applied to the voltage application line SDj connected to the drain terminal of the sensor preamplifier 37, the voltage at the connection node V is amplified and output from the source terminal of the sensor preamplifier 37 to the sensing data output line SPj. Based on the output voltage, the amount of light received by the optical sensor circuit 32 can be calculated.

  Next, the liquid crystal panel drive circuit 304, the optical sensor drive circuit 305, and the sensor output amplifier 44, which are peripheral circuits of the sensor built-in liquid crystal panel 301, will be described.

  The liquid crystal panel drive circuit 304 is a circuit for driving the pixel circuit 31, and includes a scanning signal line drive circuit 3041 and a data signal line drive circuit 3042.

  The scanning signal line driving circuit 3041 sequentially selects one scanning signal line from the scanning signal lines G1 to Gm for each line time based on the timing control signal TC1 received from the display control unit 601, and A high level voltage is applied to the selected scanning signal line, and a low level voltage is applied to the other scanning signal lines.

  Based on the display data D (DR, DG, and DB) received from the display controller 601, the data signal line driver circuit 3042 generates a predetermined voltage corresponding to the display data for one row for each line time. The data signal lines SR1 to SRn, SG1 to SGn, and SB1 to SBn are applied (line sequential method). Note that the data signal line driver circuit 3042 may be driven by a dot sequential method.

  The optical sensor driving circuit 305 is a circuit for driving the optical sensor circuit 32. Based on the timing control signal TC2 received from the sensor control unit 602, the optical sensor driving circuit 305 selects a predetermined sensor readout signal line from the sensor readout signal lines RW1 to RWm for each line time. A read voltage is applied, and a voltage other than a predetermined read voltage is applied to the other sensor read signal lines. Similarly, based on the timing control signal TC2, a predetermined reset voltage is applied to the sensor reset signal line selected from the sensor reset signal lines RS1 to RSm for each line time, and the others. A voltage other than a predetermined reset voltage is applied to the sensor reset signal line.

  The sensing data output signal lines SP1 to SPn are grouped into p groups (p is an integer of 1 to n), and the sensing data output signal lines belonging to each group are connected via a switch 47 that is sequentially turned on in time division. And connected to the sensor output amplifier 44. The sensor output amplifier 44 amplifies the voltage from the group of sensing data output signal lines connected by the switch 47 and outputs the amplified voltage to the signal conversion circuit 306 as sensor output signals SS (SS1 to SSp).

(Program and recording medium)
Finally, the circuit control unit 600, the data processing unit 700, and the main control unit 800 included in the data display / sensor device 100 may be configured by hardware logic. Alternatively, it may be realized by software using a CPU (Central Processing Unit) as follows.

  That is, the circuit control unit 600, the data processing unit 700, and the main control unit 800 are a CPU such as an MPU that executes instructions of a program that realizes each function, a ROM (Read Only Memory) that stores the program, and executes the program. A RAM (Random Access Memory) that expands into a possible format, and a storage device (recording medium) such as a memory that stores the program and various data are provided.

  The object of the present invention is not limited to the case where the program memory of the circuit control unit 600, the data processing unit 700, and the main control unit 800 is fixedly supported, but the program code of the above program (execution format program, intermediate code) Also, a recording medium in which a program or a source program is recorded is supplied to the data display / sensor device 100, and the data display / sensor device 100 reads and executes the program code recorded in the recording medium. Achievable.

  The recording medium is not limited to a specific structure or type. That is, the recording medium includes, for example, a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and an optical disk such as a CD-ROM / MO / MD / DVD / CD-R. System, a card system such as an IC card (including a memory card) / optical card, or a semiconductor memory system such as a mask ROM / EPROM / EEPROM / flash ROM.

  The object of the present invention can be achieved even if the circuit control unit 600, the data processing unit 700, and the main control unit 800 (or the data display / sensor device 100) are configured to be connectable to a communication network. In this case, the program code is supplied to the circuit control unit 600, the data processing unit 700, and the main control unit 800 via the communication network. The communication network only needs to be able to supply program codes to the circuit control unit 600, the data processing unit 700, and the main control unit 800, and is not limited to a specific type or form. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication network, etc. may be used.

  The transmission medium constituting the communication network may be any medium that can transmit the program code, and is not limited to a specific configuration or type. For example, even with wired lines such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL (Asymmetric Digital Subscriber Line) line, infrared rays such as IrDA and remote control, Bluetooth (registered trademark), 802.11 wireless, HDR, mobile phone It can also be used by radio such as a telephone network, a satellite line, and a terrestrial digital network. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention has been specifically described above based on the embodiments. However, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims, and different implementations are possible. Embodiments obtained by appropriately combining the technical means disclosed in each form are also included in the technical scope of the present invention.

  The image display / image detection apparatus according to the present invention can be suitably used for a portable information terminal such as a mobile phone or a portable game machine. In addition, if any of the display units provided in the image display / image detection apparatus can acquire image information, the scanner may be capable of simultaneously scanning both sides of a document sandwiched between folded housings. Can be used.

It is a block diagram which shows the principal part structure of the data display / sensor apparatus which concerns on this invention. It is a figure which shows typically the external appearance of the data display / sensor apparatus which concerns on this invention. It is a flowchart which shows the test | inspection process which detects the shift | offset | difference of the relative position of the display parts in the data display / sensor apparatus which concerns on this invention. It is the figure which showed the detailed detection method of the shift | offset | difference of the relative position of a display / light sensor part and a display part, (a) shows the image for an inspection, (b) shows a detection image, (c) is detection It is a figure which shows calculation of the image information in an image, (d) is a figure which shows calculation of a center point, (e) is a figure which shows calculation of the image information in the image for an inspection, (f) is a figure for inspection It is a figure which shows the comparison with the image information of an image, and the image information of a detection image. It is a figure which shows the specific example of correction | amendment of the detection image detected in the display / light sensor part, (a) is a display image displayed on the display part, (b) is the detection detected in the display / light sensor part. (C) is a diagram illustrating correction of a detected image, (d) is a corrected detected image, and (e) is a diagram illustrating a state in which the corrected detected image is vertically inverted. . 4 is a flowchart illustrating a detection method for detecting a bright spot in a display unit provided in a data display / sensor device according to the present invention. It is a figure which shows the specific example of the detection method of the bright spot in the liquid crystal panel with a built-in sensor of a display / light sensor part, and the liquid crystal panel of a display part, (a) is a figure which shows the image for an inspection, (b) is a detection image. (C) is a figure which shows correction | amendment of the position in a detection image, (d) is a figure which shows correction | amendment of the angle in a detection image, (e) shows rotation of the direction of a correction image. (F) is a figure which shows the display of the marker in a liquid crystal panel. It is a block diagram which shows the principal part structure of the modification of the data display / sensor apparatus which concerns on this invention. It is a flowchart which shows the correction method of the image scanned in the upper housing | casing. It is a figure which shows the specific example of the correction method which correct | amends the detection image detected in the 2nd display / light sensor part, (a) is the figure which mounted the original printed on both surfaces on the lower housing | casing, (B) is a detection image detected in the second display / light sensor unit, (c) is a detection image detected in the first display / light sensor unit, and (d) is in the second display / light sensor unit. It is a figure which shows the state which correct | amends the detected detection image, (e) is a correction image which correct | amended the detection image detected in the 2nd display / light sensor part, (f) is the image which reversed the correction image up and down (G) is an image obtained by horizontally inverting the detection image detected in the first display / light sensor unit. It is a figure which shows typically the cross section of a liquid crystal panel with a built-in sensor. It is a figure which shows typically the detection method which detects the position which touched on the liquid crystal panel with a built-in sensor, (a) is a schematic diagram which shows a mode that the position which the user touched is detected by detecting a reflected image. (B) is a schematic diagram which shows a mode that the position which the user touched is detected by detecting a shadow image. It is a block diagram which shows the structure of the data display / sensor apparatus based on this invention. It is a figure which shows an example of the frame structure of a command typically. It is a figure explaining an example of the value which can be specified to each field contained in a command, and its outline. It is a figure which shows the image data obtained as a result of scanning the liquid crystal panel with a built-in sensor, (a) is a result of scanning the whole liquid crystal panel with a built-in sensor when the object is not placed on the liquid crystal panel with a built-in sensor. (B) is image data obtained as a result of scanning the entire sensor-equipped liquid crystal panel when the user touches the sensor-equipped liquid crystal panel with a finger. It is a block diagram which shows the structure of a liquid crystal panel with a built-in sensor, and the structure of a peripheral circuit.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Determination part 12 Calculation part 13 Correction | amendment part 14 Detection part (identification means)
DESCRIPTION OF SYMBOLS 15 Conversion part 16 Image display control part 17 Correction | amendment image display control part 18 Marker display control part 40 Upper housing | casing 41 Lower housing | casing 42 Connection part 100 Data display / sensor apparatus (image display / image detection apparatus)
100b Data display / sensor device (image display / image detection device)
300 Display / light sensor (first planar member)
300A First display / light sensor unit 300B Second display / light sensor unit 301 Liquid crystal panel with built-in sensor 310 Display unit (second planar member)
311 Liquid crystal panel 600 Circuit control unit 700 Data processing unit 800 Main control unit 901 Storage unit

Claims (10)

A first planar member for displaying an image is disposed on one inner surface of two inner surfaces facing each other in a folded state, the housing having a foldable casing, and the other inner surface. And an image display / image detection device provided with a second planar member for detecting a nearby image,
Display control means for displaying a predetermined inspection image on the first planar member in a state in which the housing is folded;
Determination means for determining whether or not an image of the first planar member detected by the second planar member in a state in which the housing is folded matches the inspection image;
An image display / image detection apparatus comprising: When the determination means determines that the image of the first planar member detected by the second planar member does not match the inspection image, the detection is detected by the second planar member. Calculating means for calculating a deviation amount between the image of the first planar member and the inspection image;
Correction means for correcting the position coordinates of the high brightness region in the image detected by the second planar member based on the shift amount;
The image display / image detection apparatus according to claim 1, further comprising:   A correction image display control unit configured to display the image on the first planar member after correcting the image detected by the second planar member according to the shift amount calculated by the calculating unit; The image display / image detection apparatus according to claim 2.   The first planar member is based on the image of the first planar member detected by the second planar member in a state where the display control means displays a black image on the first planar member. 4. The image display / image detection apparatus according to claim 1, further comprising a specifying unit that specifies a position of an area having a luminance equal to or higher than a predetermined threshold value.   The first planar member based on the image of the first planar member detected by the second planar member in a state where the display control means displays a white image on the first planar member. 4. The image display / image detection apparatus according to claim 1, further comprising a specifying unit that specifies a position of an area having a luminance equal to or lower than a predetermined threshold.   When displaying the area specified by the specifying means on the first planar member, a marker display for displaying a marker indicating the position where the area is displayed on the first planar member on the first planar member 6. The image display / image detection apparatus according to claim 4, further comprising a control unit.   The image display / image detection apparatus according to claim 1, wherein the first planar member displays an image and detects a nearby image. A first planar member for displaying an image is disposed on one inner surface of two inner surfaces facing each other in a folded state, the housing having a foldable casing, and the other inner surface. And an image display / image detection apparatus inspection method in which a second planar member for detecting a nearby image is provided,
A display control step for displaying a predetermined inspection image on the first planar member in a state in which the housing is folded;
A determination step of determining whether an image of the first planar member detected by the second planar member in a state in which the housing is folded is coincident with the inspection image;
An inspection method characterized by comprising.   A program for operating a computer included in the image display / image detection apparatus according to claim 1, wherein the program causes the computer to function as each unit.   A computer-readable recording medium in which the program according to claim 9 is recorded.

Images