WO2019100407A1 - Positioning of terminal screen based on transformation relation of coordinates of marking graphic points in pattern - Google Patents

Abstract

A method for positioning a terminal screen based on a transformation relation of coordinates of marking graphic points in a pattern, for solving the problem of low positioning accuracy of the terminal screen. The method comprises: obtaining a first image; determining a transformation relation between a first coordinate and a second coordinate; and determining an area of a screen in the first image according to the transformation relation. The first image is a photographed image of a terminal which is provided with a screen for displaying a pattern, and the pattern comprises regularly distributed marking graphics. The first coordinate is used for representing the position of the marking graphic on the screen, and the second coordinate is used for representing the position of the marking graphic in the first image.

Description

Positioning the terminal screen based on the conversion relationship of the coordinates of the marked graphic points in the pattern Technical field

The present application relates to the field of image processing technologies, and in particular, to a method, an apparatus, and a terminal for positioning a terminal screen.

Background technique

With the popularity of touch-screen terminals such as mobile phones and tablets, users have higher requirements for the sensitivity of the touch screen of the terminal. At present, in order to provide users with a better touch screen experience, the user can simulate the user's click and slide operation on the touch screen terminal through a device such as a robot arm, thereby completing the touch screen accuracy and sensitivity test through the test device, and according to the test result. The touch screen terminal is improved.

In the actual test process, the industrial camera can process the image of the captured terminal to achieve the positioning of the terminal screen boundary, and then find the intersection according to the identified four boundaries to determine the corner point of the terminal screen, and finally from the collected Position the terminal screen in the image and pass the positioning result to the robot arm. That is, a click, a slide, or the like is performed on a certain point of the captured image, and the robot arm performs a click and slide measurement on the position of the terminal screen corresponding to the point. In the process of image processing, the terminal screen area and the non-terminal screen area in the image are mainly divided according to the threshold, and the threshold size is different, which often affects the positioning result of the boundary. Therefore, the above implementation process cannot be accurately located. The boundary of the terminal screen, so that the mechanical arm cannot accurately simulate the actual click, slide, and the like of the user. Taking the click operation as an example, the position where the robot arm performs the click operation on the terminal screen does not correspond to the position that the user actually tries to click. This affects the entire testing process and ultimately makes the test results less accurate.

Summary of the invention

The embodiment of the invention provides a method, a device and a terminal for positioning a terminal screen, which can solve the problem that the terminal screen positioning accuracy is low.

In a first aspect, an embodiment of the present invention provides a method for locating a terminal screen. The method is applied to a terminal such as a test device. The method includes: acquiring a first image, then determining a conversion relationship between the first coordinate and the second coordinate, and determining an area of the screen in the first image according to the conversion relationship. The first image is an image of the photographed terminal, the photographed terminal is provided with a screen, and the screen displays a pattern, the pattern includes a regularly distributed logo graphic; the first coordinate is used to indicate the position of the logo graphic on the screen, The two coordinates are used to indicate the position of the logo graphic in the first image. Compared with the threshold image in the prior art, the terminal screen area and the non-terminal screen area are divided according to the threshold, and the embodiment of the present invention can effectively determine the coordinate conversion relationship between the screen and the image of the terminal, and then determine the terminal according to the conversion relationship. The area of the screen.

In a possible design, determining an area of the screen in the first image according to the conversion relationship may be implemented as: determining a fourth coordinate according to the conversion relationship and the third coordinate, and determining the fourth coordinate in the first image The area where the corresponding corner points are formed by the end points is the area of the screen in the first image. The third coordinate is used to indicate the position of each corner point on the screen in the screen, and the fourth coordinate is used to indicate the position of each corner point in the first image. The embodiment of the invention can accurately locate the corner position of the terminal screen in the image by means of coordinate conversion, and then accurately determine the edge of the screen according to the position of the four corner points, thereby accurately positioning the terminal screen in the image.

In one possible design, the pattern includes at least three logo patterns. Considering that the industrial camera is shooting the image of the mobile phone, it is very likely that the lens of the industrial camera is not completely parallel with the screen of the mobile phone, that is, the image is taken at a certain angle, so that the image obtained has some distortion, that is, the original rectangular mobile phone screen is very It may be a regular or irregular quadrangle. In general, two points can determine a straight line, and three points can determine a plane. In the embodiment of the present invention, if the image is distorted and an attempt is made to determine the area where the screen of the mobile phone is located in the image, it can be regarded as determining a plane. The process, so at least the position of 3 points needs to be determined before the plane can be derived. The position of each point may correspond to a logo pattern. Therefore, in the embodiment of the present invention, the number of the logo patterns in the pattern may be set to be greater than or equal to three.

In one possible design, the regular distribution includes the center point equidistance of the logo pattern distributed in a matrix in rows and columns. Taking the logo pattern as a circle, that is, the distance between every two adjacent centers in the lateral direction is the same, and the distance between every two adjacent centers in the longitudinal direction is the same. In addition, in the embodiment of the present invention, the flag pattern may be distributed according to other rules, and is not limited to the rule in which the spacing between the center points of two adjacent flag patterns in the same direction is equal, or may be in the same direction. The step size of the variability, setting the position of each of the logo patterns, for example, in the same direction, the distance between the center point of the first logo pattern and the second logo pattern is X, and the second logo pattern and the The spacing between the three logo patterns is 2X, and so on, for each additional logo pattern, the increased spacing between the logo pattern and the adjacent previous logo pattern is increased by X; or, in the same direction, The distance between the center point of one logo graphic and the second logo graphic is X, and the distance between the second logo graphic and the third logo graphic is X+2, and so on, each additional logo graphic The spacing between the added logo pattern and the adjacent previous logo pattern is increased by 2, etc., and is not limited herein. Of course, the variability step size may be a function in which an independent variable changes continuously, and may be other implementation forms, which is not limited herein.

In one possible design, the logo pattern includes a regular logo pattern and a special logo pattern. The regular logo graphics are identical, the special logo graphics are different from the conventional logo graphics, and the special logo graphics are used to distinguish the orientation of the terminal when shooting. For example, the area of a special logo graphic can be significantly larger or smaller than a conventional logo graphic in a pattern.

In one possible design, the logo pattern is black, and the area of the pattern other than the logo pattern is white. Considering that the image obtained after the industrial camera is taken is an image with gray scale, if there are only two colors of black and white in the pattern, the imaging effect will be better. It can be seen that the color of the pattern is not limited, and the arrangement pattern of the other colors can be clearly distinguished. For example, the logo pattern is dark, and the rest is light color.

In a possible design, after the logo pattern is deformed due to a change in the shooting angle, the center point position of the logo pattern does not change. It should be noted that each of the logo patterns in the pattern may also be a circle, an ellipse or the like, and the shape of the center point is not affected even if the deformation occurs, and the shape of the logo pattern is not limited herein.

In one possible design, the resolution of the pattern is the same as the resolution of the screen. Of course, the resolution of the pattern can also be proportional to the resolution of the screen, that is, the screen of the mobile phone can generate a pattern with the same resolution or proportional to the resolution of the screen of the mobile phone according to the resolution of the screen of the mobile phone. For example, if the resolution of the mobile phone screen is 1280*720, then the resolution of the pattern can be 1280*720 or A*B, where A is the product of 1280 and the first ratio, and B is 720. In the second aspect, the first ratio and the second ratio may be the same or different. In the embodiment of the present invention, the first ratio and the second ratio are positive, and the logo pattern in the pattern is distributed according to a certain rule. The pattern is displayed on the phone display.

In a second aspect, an embodiment of the present invention provides a method for locating a terminal screen. The method includes: the terminal displays the pattern through the screen, the industrial camera acquires the first image and determines the second coordinate, the testing device acquires the first coordinate and the second coordinate, and determines a conversion relationship between the first coordinate and the second coordinate, and according to Convert the relationship to determine the area of the screen in the first image. Wherein, the pattern comprises a regularly distributed logo graphic, the first coordinate is used to indicate the position of the logo graphic in the screen; the second coordinate is used to indicate the position of the logo graphic in the first image, and the first image is an image of the photographed terminal.

In a possible design, after the testing device determines the conversion relationship between the first coordinate and the second coordinate, the method further includes: the testing device acquires the fifth coordinate, and converts the fifth coordinate according to the conversion relationship, to obtain The sixth coordinate is then sent to the robotic arm; the robotic arm performs an operation at the position of the sixth coordinate in the first image in response to the message sent by the test device. The fifth coordinate is used to indicate the position to be tested on the screen, and the sixth coordinate is used to indicate the position of the position to be tested in the first image.

In a third aspect, an embodiment of the present invention provides a terminal cleaning apparatus. The device can implement the functions implemented by the terminal in the foregoing method embodiments, and the functions can be implemented by using hardware or by executing corresponding software through hardware. The hardware or software includes one or more modules corresponding to the above functions.

In a fourth aspect, an embodiment of the present invention provides a terminal. The structure of the terminal includes a display screen, a memory, one or more processors, a plurality of applications, and one or more programs; wherein the one or more programs are stored in the memory; the one Or the plurality of processors, when executing the one or more programs, causing the terminal to implement the method of any of the first aspect and its various possible designs.

In a fifth aspect, an embodiment of the present invention provides a readable storage medium, including instructions. When the instruction is run on the terminal, the terminal is caused to perform the method of any of the above first aspects and its various possible designs.

In a sixth aspect, an embodiment of the present invention provides a computer program product, the computer program product comprising software code for performing the method of any of the above first aspects and various possible designs thereof.

DRAWINGS

FIG. 1 is a schematic structural diagram of a first terminal according to an embodiment of the present disclosure;

2 is a schematic diagram of a process of photographing a mobile phone by an industrial camera according to an embodiment of the present invention;

3 is an image of a mobile phone photographed by a first industrial camera according to an embodiment of the present invention;

4 is a schematic diagram of a pattern according to an embodiment of the present invention;

FIG. 5 is an image of a mobile phone photographed by a second industrial camera according to an embodiment of the present invention; FIG.

FIG. 6 is a schematic diagram of a dot location for testing a screen sensitivity of a mobile phone according to an embodiment of the present invention; FIG.

FIG. 7 is a schematic diagram of a straight line fitting to determine a screen area of a mobile phone according to an embodiment of the present invention; FIG.

FIG. 8 is a first black and white image after processing according to an embodiment of the present invention; FIG.

FIG. 9 is a schematic diagram of a main interface of a mobile phone according to an embodiment of the present invention;

FIG. 10 is a schematic diagram of establishing a two-dimensional coordinate system on a screen of a mobile phone according to an embodiment of the present invention; FIG.

FIG. 11 is a second processed black and white image according to an embodiment of the present invention;

FIG. 12 is a schematic structural diagram of a device for positioning a terminal screen according to an embodiment of the present invention;

FIG. 13 is a schematic structural diagram of a second terminal according to an embodiment of the present invention.

Description of the reference signs:

201- lens of industrial camera;

202-phone;

203 - the placement area of the test equipment;

204-Special logo pattern in the pattern;

205-interference pattern;

206-phone screen;

207 - The edge of the bright area of the phone.

Detailed ways

The embodiments of the present invention may be used in a terminal, which may include a notebook computer, a smart phone, an industrial camera, a test device, a virtual reality (VR) device, an augmented reality (AR), an in-vehicle device, Devices such as smart wearable devices. The terminal is provided with at least a display screen, an input device and a processor. Taking the terminal 100 as an example, as shown in FIG. 1 , the terminal 100 includes a processor 101, a memory 102, a camera 103, an RF circuit 104, an audio circuit 105, and a speaker. 106, a microphone 107, an input device 108, other input devices 109, a display screen 110, a touch panel 111, a display panel 112, an output device 113, and a power source 114. The display screen 110 is composed of at least a touch panel 111 as an input device and a display panel 112 as an output device. It should be noted that the terminal structure shown in FIG. 1 does not constitute a limitation on the terminal, and may include more or less components than those illustrated, or combine some components, or split some components, or different. The component arrangement is not limited herein.

The components of the terminal 100 will be specifically described below with reference to FIG. 1 :

The radio frequency (RF) circuit 104 can be used for receiving and transmitting information during the transmission or reception of information or during a call. For example, if the terminal 100 is a mobile phone, the terminal 100 can receive the downlink information sent by the base station through the RF circuit 104. Thereafter, it is transmitted to the processor 101 for processing; in addition, data related to the uplink is transmitted to the base station. Generally, RF circuits include, but are not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. In addition, RF circuitry 104 can also communicate with the network and other devices via wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (Code Division). Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), E-mail, Short Messaging Service (SMS), and the like.

The memory 102 can be used to store software programs and modules, and the processor 101 executes various functional applications and data processing of the terminal 100 by running software programs and modules stored in the memory 102. The memory 102 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function (for example, a sound playing function, an image playing function, etc.); and the storage data area may be Data (such as audio data, video data, etc.) created in accordance with the use of the terminal 100 is stored. Moreover, memory 102 can include high speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

Other input devices 109 can be used to receive input numeric or character information, as well as to generate key signal inputs related to user settings and function control of terminal 100. Specifically, other input devices 109 may include, but are not limited to, a physical keyboard, function keys (eg, volume control buttons, switch buttons, etc.), trackballs, mice, joysticks, light rats (light mice are touches that do not display visual output) a sensitive surface, or a touch-sensitive surface formed by a touch screen One or more of the extensions). Other input devices 109 may also include sensors built into the terminal 100, such as gravity sensors, acceleration sensors, etc., and the terminal 100 may also use parameters detected by the sensors as input data.

The display screen 110 can be used to display information input by the user or information provided to the user as well as various menus of the terminal 100, and can also accept user input. The display panel 112 can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. The touch panel 111 is also called a touch screen or a touch sensitive screen. Etc., the contact or non-contact operation of the user on or near the user may be collected (for example, the user may use any suitable object or accessory such as a finger or a stylus on the touch panel 111 or in the vicinity of the touch panel 111, or Including the somatosensory operation; the operation includes a single point control operation, a multi-point control operation and the like, and drives the corresponding connection device according to a preset program. It should be noted that the touch panel 111 may further include two parts: a touch detection device and a touch controller. Wherein, the touch detection device detects the touch orientation and posture of the user, and detects a signal brought by the touch operation, and transmits a signal to the touch controller; the touch controller receives the touch information from the touch detection device, and converts the signal into the processor 101. The information that can be processed is transmitted to the processor 101, and the commands sent from the processor 101 can also be received and executed. In addition, the touch panel 111 can be implemented by using various types such as resistive, capacitive, infrared, and surface acoustic waves, and the touch panel 111 can be implemented by any technology developed in the future. In general, the touch panel 111 can cover the display panel 112, and the user can cover the display panel 112 according to the content displayed by the display panel 112 (including but not limited to a soft keyboard, a virtual mouse, a virtual button, an icon, etc.). The touch panel 111 operates on or near the touch panel 111. After detecting the operation thereon or nearby, the touch panel 111 transmits to the processor 101 to determine the user input, and then the processor 101 provides the display panel 112 according to the user input. Corresponding visual output. Although the touch panel 111 and the display panel 112 are used as two independent components to implement the input and output functions of the terminal 100 in FIG. 1, in some embodiments, the touch panel 111 may be integrated with the display panel 112. To implement the input and output functions of the terminal 100.

The RF circuit 104, the speaker 106, and the microphone 107 provide an audio interface between the user and the terminal 100. The audio circuit 105 can transmit the converted audio data to the speaker 106 for conversion to the sound signal output. On the other hand, the microphone 107 can convert the collected sound signal into a signal, which is received by the audio circuit 105. The audio data is then converted to audio data, and the audio data is output to the RF circuit 104 for transmission to a device such as another terminal, or the audio data is output to the memory 102 for the processor 101 to perform further processing in conjunction with the content stored in the memory 102. In addition, the camera 103 can acquire image frames in real time and transmit them to the processor 101 for processing, and store the processed results to the memory 102 and/or present the processed results to the user via the display panel 112.

The processor 101 is the control center of the terminal 100, connecting various portions of the entire terminal 100 using various interfaces and lines, by running or executing software programs and/or modules stored in the memory 102, and recalling data stored in the memory 102. The various functions and processing data of the terminal 100 are executed to perform overall monitoring of the terminal 100. It should be noted that the processor 101 may include one or more processing units; the processor 101 may further integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface (User Interface, UI) And the application, etc., the modem processor mainly handles wireless communication. It can be understood that the above modem processor may not be integrated into the processor 101.

The terminal 100 may also include a power source 114 (eg, a battery) that supplies power to the various components, implemented in the present invention. In an example, the power source 114 can be logically coupled to the processor 101 through a power management system to manage functions such as charging, discharging, and power consumption through the power management system.

In addition, there is also a component that is not shown in FIG. 1 . For example, the terminal 100 may further include a Bluetooth module and the like, and details are not described herein again.

In the following, the technical solution of the embodiment of the present invention is described by taking the terminal 100 shown in FIG. 1 as a mobile phone and an industrial camera as an example. Among them, the industrial camera can be regarded as a part of the test equipment. For the test equipment, the industrial camera can transmit the image of the photographed mobile phone to the test equipment, and the test equipment analyzes and processes the same. Similarly, the mobile phone can also generate itself. Or the detected data is transmitted to the test device, and the test device performs subsequent operations such as positioning the mobile phone screen.

In an embodiment of the invention, an industrial camera is used to capture an included image of the handset, which is then processed by an industrial camera and/or mobile phone to locate the screen area of the handset. It should be noted that the method is also applicable to the positioning of the peripheral contour of the mobile phone or the positioning of a specific area of the mobile phone screen, which is not limited herein.

As shown in FIG. 2, a schematic diagram of a process of photographing a mobile phone by an industrial camera. Wherein, the mobile phone 202 is placed in the placement area of the test device, and the plane of the lens 201 of the industrial camera is parallel to the plane of the mobile phone 202. It should be noted that, in the embodiment of the present invention, an industrial camera can be regarded as a part of a test device, that is, an industrial camera and a placement area, and other components together constitute a test device, and of course, the industrial camera can also be separately set with the test device, and then Industrial cameras can transmit the captured images to test equipment or other devices such as mobile phones for processing, or industrial cameras can transfer the processed images to test equipment or pass them to test equipment through other devices such as mobile phones. This is not limited.

As shown in Figure 3, an image of a cell phone shot for an industrial camera. Among them, the image obtained after the industrial camera is taken is a grayscale image, and the image is only used as a possible shooting example, and is not limited as an image obtained after the shooting.

In the actual shooting process, the angle of the mobile phone, the distance between the mobile phone and the industrial camera, the angle of view of the industrial camera relative to the mobile phone, etc., will affect the image, that is, the image of the mobile phone shot by the industrial camera is different according to the actual shooting. Different, it is not limited here. In other words, in the process of actually shooting a mobile phone image, it is likely that the industrial camera is not due to the unsmooth placement of the test device or the plane of the placement area and the plane of the lens of the industrial camera. The plane of the vertical mobile phone is photographed, so that there is a certain error in the screen area of the mobile phone. For example, the screen area of the mobile phone is rectangular in the case of normal shooting, and the current quadrangle due to distortion is irregular or regular. In addition, the graphics presented on the phone screen after distortion will also be affected by the actual shape of the screen area of the phone.

In the embodiment of the present invention, as shown in FIG. 4, a screen with a resolution equal to or proportional to the resolution of the screen of the mobile phone may be generated according to the resolution of the screen of the mobile phone, for example, the resolution of the screen of the mobile phone. For 1280*720, the resolution of the pattern can be 1280*720 or A*B, where A is the product of 1280 and the first ratio, and B is the product of 720 and the second ratio. A ratio and a second ratio may be the same or different. In the embodiment of the present invention, the first ratio and the second ratio are positive values, and the logo graphic in the pattern is distributed according to a certain rule and displayed on the mobile phone display screen. in. In the pattern, the rule is distributed with a plurality of logo patterns that are circular. The logo graphic includes a regular logo graphic and a special logo graphic. Wherein, the conventional logo graphic is completely identical, and the special logo graphic is used to distinguish the orientation of the mobile phone, and the area of the special logo graphic is obviously larger than that in the pattern. Regular logo graphics.

In the embodiment of the present invention, the equidistant equidistant spacing of each circular marking pattern in the pattern is distributed in a matrix according to rows and columns, that is, the spacing of every two adjacent centers in the lateral direction is the same, and the spacing of each two adjacent centers in the longitudinal direction is the same. In addition, the circle in the pattern is black, and the portion of the pattern other than the logo pattern is white. It should be noted that each of the logo patterns in the pattern may also be an ellipse or the like, and the shape of the center point is not affected even if the deformation occurs, and the shape of the logo pattern is not limited herein.

In addition, the rule distribution referred to in the embodiment of the present invention is not limited to the rule in which the spacing between the center points of two adjacent flag patterns in the same direction is equal, and may be in the same direction according to the variability. Step size, setting the position of each logo graphic, for example, in the same direction, the distance between the center point of the first logo graphic and the second logo graphic is X, and the second logo graphic and the third logo The spacing between the graphics is 2X, and so on, for each additional logo graphic, the spacing between the increased logo pattern and the adjacent previous logo graphic is increased by X; or, in the same direction, the first logo The distance between the graphic and the center point of the second logo graphic is X, and the distance between the second logo graphic and the third logo graphic is X+2, and so on. For each additional logo graphic, the increase is increased. The spacing between the logo pattern and the adjacent previous logo pattern is increased by 2, etc., and is not limited herein.

Of course, the variability step size may be a function in which an independent variable changes continuously, and may be other implementation forms, which is not limited herein. In addition, for the distribution of colors in the pattern, in the embodiment of the present invention, the pattern is presented in black and white, that is, the logo pattern is black, and the rest is white. Considering that the image obtained after the industrial camera is taken is an image with gray scale, if there are only two colors of black and white in the pattern, the imaging effect will be better. It can be seen that the color of the pattern is not limited, and the arrangement pattern of the other colors can be clearly distinguished. For example, the logo pattern is dark, and the rest is light color.

In addition, for the special logo graphics in the pattern, one or more may be set, the purpose of which is to distinguish the direction in which the mobile phone is placed. The specific method of differentiation will be presented later and will not be repeated here. In the embodiment of the present invention, the special logo pattern and other logo patterns are distinguished by adjusting the size of the logo graphic, which is not limited herein.

In addition, in the embodiment of the present invention, the process of generating a pattern may be performed by a mobile phone, or generated by an industrial camera, a test device, or other device according to the resolution of the mobile phone screen, and the generated pattern is transmitted to the mobile phone for full screen of the mobile phone. It is shown here that the position, manner, and the like of the generated pattern are not limited.

Taking a regularly distributed pattern as an example, if the resolution of the mobile phone screen is 1280*720, the generated pattern can be regarded as a picture with a resolution of 1280*720. For example, the distribution of the logo graphics in the pattern can be 15*15, that is, the horizontal and vertical distributions are 15 logo graphics, then the horizontal direction (the direction of the edge corresponding to 1280 pixels in the mobile phone screen) is 15 each of the 15 logo graphics. The distance between the center points of the logo graphics is 80, and the vertical distance from the center point of the logo pattern near the longitudinal edge of the screen of the mobile phone to the longitudinal edge is also 80, that is, 80 pixels, wherein 80 pixels are calculated as 1280/(15+1)=80, portrait (the direction of the edge corresponding to 720 pixels in the screen of the mobile phone) The spacing between the center points of each of the two logo images in the 15 logo images is 45, and is close to the lateral edge of the screen of the mobile phone. The vertical distance from the center point of the logo pattern to the lateral edge is also 45, which is 45 pixels. For example, taking the special logo graphic as an example, the vertical distance of the special logo graphic from the lateral edge of the screen is 45, and the vertical distance from the longitudinal edge of the screen is 80. The distance referred to herein refers to the distance from the center point to the edge of the logo graphic. If the location of the special logo graphic is closer to the coordinate origin of the mobile phone screen (a corner point of the mobile phone screen, that is, the intersection formed by the intersection of two adjacent sides of the mobile phone screen), and the corner point is taken as the coordinate origin, the horizontal side of the mobile phone screen On The direction away from the origin is the positive direction of the x-axis in the two-dimensional coordinate system, and the direction away from the origin on the longitudinal side of the screen of the mobile phone is the positive direction of the y-axis in the two-dimensional coordinate system, and the coordinates of the corner point as the origin are (0) , 0), the coordinates of the center point of the special logo graphic is (80, 45). By analogy, the coordinates of the center point of each logo graphic when the pattern is presented on the screen of the mobile phone can be obtained.

It can be seen that each of the logo patterns existing in the pattern can be regarded as a set of logo graphics, denoted as P={P1, P2, . . . , Pn}, where n is an integer greater than or equal to 3, and each of the logo graphics The center point coordinates can be regarded as a set of center points of the logo pattern on the screen pixel coordinates, denoted as Ps={Ps1, Ps2, ..., Psn}. It should be noted that, in view of the process of shooting an image of a mobile phone by an industrial camera, it is likely that the lens of the industrial camera is not completely parallel with the screen of the mobile phone, that is, the image is taken at a certain angle, and thus the obtained image may have a certain distortion, that is, the original image is present. Rectangular phone screens are likely to be regular or irregular quads. In general, two points can determine a straight line, and three points can determine a plane. In the embodiment of the present invention, if the image is distorted and an attempt is made to determine the area where the screen of the mobile phone is located in the image, it can be regarded as determining a plane. The process, so at least the position of 3 points needs to be determined before the plane can be derived. The position of each point may correspond to a logo pattern. Therefore, in the embodiment of the present invention, the number of the logo patterns in the pattern may be set to be greater than or equal to three. It should be noted that, in consideration of at least one special logo pattern in the pattern to distinguish the direction in which the mobile phone is placed, in the embodiment of the present invention, in order to ensure that the direction of the mobile phone can be effectively distinguished, the flag is determined as accurately as possible. The coordinate point of the center point of the graphic in the image captured by the mobile phone screen and the industrial camera respectively, and at least the center point coordinate pair of the two logo figures (that is, the same center point in the image taken by the mobile phone screen and the industrial camera respectively) Coordinates are calculated as initial parameters. It should be noted that the more the number of the logo patterns in the pattern, the more accurate the coordinate transformation relationship is obtained, and the fewer the number of the logo patterns in the pattern, the less resources are occupied for calculating the coordinate conversion relationship. Therefore, in the embodiment of the present invention, the accuracy of the resource occupancy and the coordinate conversion relationship can be weighed according to the current actual requirements, so as to determine how many flag patterns are configured in the current pattern to be used, which is not limited herein.

As shown in FIG. 5, the mobile phone screen displays a pattern of regular distribution determined according to the resolution, and the industrial camera is perpendicular to the mobile phone, and the mobile phone image obtained after the mobile phone is photographed. Wherein, in the captured grayscale image, not only the logo graphic but also the interference graphic may be included. In the embodiment of the present invention, the interference pattern may be an external environmental factor such as illumination, or a rule that the industrial camera lens is not clean, or the mobile phone display screen is not clean, or irregular regions or points. It can be known from the acquired images that all the logo patterns included in the pattern may not be collected by the industrial camera, for example, 205 shown in FIG. 5, which means that the image captured by the industrial camera may only include a part. Logo graphic. It is also because of this that the more logo patterns are included in the pattern, the more accurate the processing results obtained after processing the collected images.

Based on the image shown in FIG. 5, based on the resolution of the mobile phone screen, a two-dimensional coordinate system is established to determine the center point coordinates of each of the logo patterns in the image. It should be noted that, since there is an interference pattern in the image, it is likely that some of the flag patterns cannot be accurately determined, and the center point coordinates of the individual flag patterns that cannot be accurately determined are also difficult to determine from the collected images. In the embodiment of the present invention, the center point coordinates of the missing or incomplete flag pattern can be calculated by combining the coordinates of the center point of the existing logo pattern, and then the center point set of the logo pattern on the pixel coordinates of the image is obtained, which is recorded as Pc. ={Pc1,Pc2,...,Pcn}.

The way to calculate the coordinates of the center point of a missing or incomplete marker graphic can be as follows:

Calculate the adjacent two targets according to the horizontal, vertical, or diagonal coordinates of the center point of the identified logo pattern. The coordinates of the center point of the graph are increased or decreased in the horizontal and vertical directions, or the increment and decrement rules, and then the coordinates of the center point of the missing or incomplete logo pattern in the pattern are derived;

If the number of the horizontal logo patterns is an odd number, the center point coordinates of all the logo patterns located at the lateral center position in the longitudinal direction are determined, and the determined straight line of each center point coordinate is used as the symmetry axis of the two-part logo pattern in the lateral direction, through the symmetry The way of comparison determines the coordinates of the center point of the missing or incomplete mark graphic in the pattern.

Wherein, if the number of the horizontal logo patterns is an odd number, the center point coordinates of all the logo patterns located at the lateral center position in the longitudinal direction are determined, that is, the center point coordinates of the two logo patterns are determined in each horizontal line, and then the two lines belonging to the same horizontal line are The center point coordinates of the logo patterns are used as a group to determine the midpoint of the line segment between the center point coordinates of the two logo patterns belonging to the same horizontal line, and the straight line of all the midpoints obtained thereafter is used as the horizontal two-part logo pattern. The axis of symmetry also determines the coordinates of the center point of the missing or incomplete mark pattern in the pattern by symmetric alignment.

It should be noted that the above implementation manners are also applicable to the vertical and oblique items. The specific implementation manner may refer to the horizontal determination manner, and details are not described herein. In addition, the manner of calculating the coordinates of the center point of the missing or incomplete mark pattern is not limited to the above-mentioned several examples, and may be other calculation methods based on the pattern distribution rule, and will not be described herein.

Considering that the pattern may be distorted, missing or incomplete after being photographed, in order to ensure that the orientation of the mobile phone can be accurately determined, the number of special logo graphics can be set to multiple, or directly on the screen of the mobile phone. The origin of the coordinate is the starting point, and the parameters such as the area or radius of the logo graphic are incremented or decremented in a diffused manner, so that although there is no unique or multiple special logo graphics in the pattern, the graphics in the pattern are intuitively displayed. The change rule is seen, so that it is convenient to distinguish the position of the coordinate origin of the mobile phone screen in the image, which is not limited herein.

It should be noted that the operation of determining the central point set of the logo pattern on the image pixel coordinates may be performed on a mobile phone, an industrial camera, a test device, or other devices, and is not limited herein.

After that, it can be set by any one of the mobile phone, the industrial camera, the test device and other devices according to the central point set of the logo graphic on the pixel coordinates of the image, the central point set of the logo graphic on the pixel coordinates of the screen, and the two coordinate systems. The position of the origin of the central coordinate, the coordinates of each center point in the image and the coordinates of each center point in the pattern are in one-to-one correspondence, thereby determining the point coordinate conversion relationship of each of the logo patterns in the different coordinate systems in the pattern, that is, the function relationship, to obtain the mobile phone The position coordinates of the corner point of the screen in the image.

For example, the resolution of the mobile phone screen is 1280*720, and the coordinates of the four corner points of the mobile phone screen may be (0, 0), (1280, 0), (0, 720), and (1280, 720), respectively. Then, according to the calculated function relationship, the position coordinates of the above four corner points in the image are determined, thereby locating the positions of the four corner points of the mobile phone screen in the image, and then establishing the connection between the adjacent two corner points. Line to determine the location of the phone screen in the image. Similarly, through the point coordinate conversion relationship obtained by the above calculation, the position of any point in the screen of the mobile phone can be converted into the position corresponding to the arbitrary point in the image, thereby realizing the subsequent detection of the sensitivity of the screen of the mobile phone, the click operation, and the like. Not limited.

The solution to be described in the embodiments of the present invention will be further described below in conjunction with an example.

In the process of performing screen detection on a terminal such as a mobile phone, the content to be recognized is often a specific icon or text, and therefore the requirement for recognition accuracy is not high. Moreover, for the user's use process, it is difficult for the user to ensure that the click operation can accurately fall to the center point position of the application icon. But for the phone The accuracy of the touch screen, etc., requires higher precision, and compared with the recognition process of the application icon, the accuracy test often does not have a specific icon template to match, so how to take a device with a shooting function such as an industrial camera In the captured images, it is particularly important to realize the recognition of the specific position of the mobile phone screen.

As shown in FIG. 6, the test for the sensitivity of the touch screen, that is, the touch test on the screen of the mobile phone, that is, whether there is a difference between the position of the finger press and the point actually sensed by the mobile phone, thereby judging whether the touch screen is good or bad. In the embodiment of the present invention, the dot positions 1 to 9 shown in FIG. 6 are distances with respect to the boundary line of the mobile phone screen. It can be seen that the determination accuracy of the screen boundary is very high.

In addition, for the test equipment, the position of the mobile phone on the test equipment may be the same or different each time, and on the same type of test equipment, the screen size of the mobile phone placed each time may be the same or different. For example, if the mobile phone screen is non-directional, then for a mobile phone with 1080*1920 pixels, if the user tries to click (500,500) and the mobile phone is placed in the forward direction, the (500,500) point can be accurately obtained. If the phone is placed in the reverse position, the user should actually click on the point (1080-500, 1920-500), which will cause a large error between the actual click position and the position the user is trying to click. Therefore, in order to solve the problem of the screen positioning of the mobile phone, the technical solution provided by the embodiment of the present invention can also reduce the process of manually recognizing the screen of the mobile phone through visual. The manner of distinguishing the orientation of the screen of the mobile phone can be implemented by referring to the difference of the logo pattern in the drawing mentioned above, and details are not described herein.

After the mobile phone is placed in the placement area of the test device, if the mobile phone is in a bright state, the mobile phone is photographed by an industrial camera, and an image as shown in FIG. 3 is obtained. The border of the mobile phone screen is the edge 207 of the bright screen area of the mobile phone screen 206. As can be seen from FIG. 3, the mobile phone screen 206 includes a mobile phone bright screen area and a black border 208 surrounded by four mobile phone bright screen area edges 207, wherein the boundary between the mobile phone bright screen area and the black screen 208 of the mobile phone screen 206 is attached. Obviously, the four boundaries of the mobile phone screen can be fitted by visual algorithm fitting, and the intersections obtained by the four boundaries are the four corner points of the mobile phone screen.

In the process of visual algorithm fitting, the pixel shown in FIG. 3 needs to be enlarged, wherein the gray image shown in FIG. 7 can be obtained by taking the corner of the mobile phone screen at the lower right of FIG. 3 as an example. Then, by means of straight line fitting, a black and white image is obtained, thereby determining the boundary contour of the screen of the mobile phone, and then determining the position of the corner point.

It should be noted that the straight line fitting method is mainly to set a threshold value, and then the gray image is generated by referring to the set threshold value to generate a black and white image, and then combined with the irregular edge in the black and white image to fit a straight line. In addition, the difference in threshold size settings will also affect the final difference in the boundaries of the phone screen. As can be seen from Fig. 7, although the straight line fitting method can determine the position of the corner point, it is difficult to fit an accurate straight line. For example, for a mobile phone screen, it is possible to fit the rectangular screen area to no. A regular quadrilateral area that affects the positioning of the corner points.

In the embodiment of the present invention, the pattern is generated in combination with the pixels of the screen of the mobile phone in the manner mentioned above, and then the mobile phone is displayed in full screen when the mobile phone is photographed. The more the number of the logo patterns in the pattern, the stronger the robustness of the algorithm for determining the coordinate transformation relationship of other arbitrary points obtained by using the least squares method to calculate the conversion relationship.

In addition, during the pattern generation process, the device that generates the pattern can be connected to the mobile phone using a Universal Serial Bus (USB) to establish a communication connection between the mobile phone and the device that generates the pattern, thereby generating the pattern device. After determining the pixel of the mobile phone screen and generating the pattern, push the pattern to the mobile phone, and the mobile phone can The pattern is stored in the local storage area, after which the phone can automatically display the pattern in full screen to prepare for testing.

In the embodiment of the present invention, when an industrial camera captures an image, the image sensor of the industrial camera can be parallel to the screen of the mobile phone. As for the object distance of the industrial camera, the size of the field of view is adapted to the size of the mobile phone, which is not limited herein. The image sensor includes, but is not limited to, one of the following: a Charge Coupled Device (CCD), and a Complementary Metal-Oxide-Semiconductor (CMOS).

It should be noted that the image captured by the industrial camera is a grayscale image. In order to reduce the error caused by the subsequent grayscale image to the recognition process, in the embodiment of the present invention, the collected grayscale image may be processed in advance. For example, the maximum inter-class variance method (OSTU) is used to segment the image in combination with a preset or a currently automatically generated threshold to obtain a black and white image. The grayscale image shown in Fig. 5 taken by the industrial camera is processed to obtain a black and white image as shown in Fig. 8.

Then, the device for image processing, such as a mobile phone, an industrial camera, a test device, or other device, can acquire the screen area of the mobile phone by means of contour extraction. Considering the image obtained by the OSTU automatic threshold segmentation, the edge of the screen area of the mobile phone is likely to have irregularities such as burrs, so in order to ensure that the acquired screen area of the mobile phone completely includes the screen of the mobile phone, the front contour can be extracted. The resulting maximum contour is processed to find the smallest rectangle circumscribing the maximum contour to initially determine the extent of the screen area of the handset.

In order to facilitate the determination of the point coordinates of each of the logo patterns in the image, in the embodiment of the present invention, the range of the initially determined screen area of the mobile phone can be used as the range of each of the logo patterns in the search pattern to be collected from the image captured by the industrial camera. The outline information, that is, the outline information of each of the logo patterns in the image, is extracted in the initially determined area of the screen of the mobile phone. The contour information includes, but is not limited to, at least one of the following: an area, a radius, and the like of the logo pattern. Then, according to the contour information, the process of filtering out the invalid contour is performed on each of the flag patterns that can be displayed in the image, thereby filtering out the invalid flag pattern caused by illumination or the like.

It is then necessary to selectively recover missing or incomplete flag graphics to determine the center point coordinates of the missing or incomplete marker graphic. It should be noted that the manner of restoring the incomplete logo pattern may be based on the definition of the logo graphic itself. For example, if the current logo graphic is circular, it can be triggered from the definition of the circle, and the distance from the center point to any point in the periphery is completely equal. The figure is a logo pattern in the pattern of the embodiment of the present invention, so that the circular contour is complemented according to the principle. The specific implementation manner can refer to the prior art, and details are not described herein. To restore the missing logo pattern, the center point coordinates of the missing logo pattern can be determined according to the relationship between the center point coordinates of the existing logo pattern, and then the contour information of other logo patterns is combined to determine the missing The outline features of the logo graphic, such as the radius of the circle, etc., thereby restoring the missing logo pattern. Of course, in the embodiment of the present invention, the center point coordinates of the missing or incomplete flag graphic may be directly determined according to the relative relationship between the coordinates of the center points of the existing flag patterns. The complete logo graphic does not need to restore the missing logo graphic.

After the image is processed, each of the logo patterns can be further processed, for example, a random sample consistency algorithm (Random Sample Consensus, RANSAC) is used to perform round extraction, that is, to detect a circular contour. The position of each point from the center of the circle is filtered to remove the point with large deviation on the circular contour, and then the least squares fitting is performed according to the unfiltered point, thereby obtaining the finely processed flag graphic parameter such as radius. .

In the embodiment of the present invention, in order to effectively locate the positive direction of the screen of the mobile phone, if the pattern is the image shown in FIG. 4, the relative position of the largest circle and the other rounds may be determined in combination with the logo pattern parameter, thereby determining the mobile phone. The positive direction of the screen, according to the positive direction, the coordinates of the center point of each logo graphic in the image captured by the mobile phone screen and the industrial camera, the coordinates of each point on the two coordinate systems are one-to-one correspondence, and then the minimum two can be passed. Multiplication calculates the transformation relationship between the two point coordinate sets, that is, calculates the rotation translation scaling transformation matrix.

The above-mentioned algorithms, such as the least squares method, are prior art, and reference may be made to the implementation conditions, calculation methods, and the like in the prior art, and details are not described herein.

After accurately determining the conversion relationship, in the case of knowing the corner coordinates of the mobile phone screen, the conversion relationship can be combined to calculate the position of each mobile phone screen corner point in the image, and the positive direction of the mobile phone screen can be determined. Then, through the position of the four corner points, four boundary lines of the mobile phone screen can be fitted in the image to determine the area of the screen of the mobile phone. In addition, according to the conversion relationship, it is also possible to determine the position in the image after any conversion on the screen of the mobile phone.

In order to further ensure the reliability of the conversion relationship, in the embodiment of the present invention, the pattern may be generated according to the resolution of the screen of the mobile phone, and the implementation manner may be repeatedly executed, thereby obtaining another conversion relationship, and then comparing the two conversion relationships. Pairing, adjusting, and thus a more accurate conversion method, that is, the conversion relationship that is ultimately used to determine the correspondence between the two coordinate systems. It should be noted that as the number of times of re-determining the conversion relationship increases, the resulting conversion relationship is more accurate. Of course, it is also possible to detect one or more positions on the screen of the mobile phone by using the current conversion relationship, thereby judging whether the currently obtained conversion relationship is accurate, and fine-tuning the conversion relationship according to the actual situation, etc., Give a brief description.

It can be seen that the embodiment of the present invention can be effectively applied to the automatic positioning of the screen of the mobile phone, so that the dot-dash line similar to the designated position in the screen of the mobile phone, the accurate acquisition of the icon position, and the like can be more accurately positioned. In addition, since the pattern is generated based on the pixels of the screen of the mobile phone, the implementation adopted by the embodiment of the present invention can be effectively applied to the detection of the electronic device having the display screen, such as a mobile phone or a tablet computer of each model. Moreover, in the implementation manner provided by the embodiment of the present invention, the placement direction of the device such as the mobile phone waiting for testing on the testing device is not limited, which means that the probability of the detection process being affected by the environment is low, thereby Improve the accuracy of the entire inspection process.

Taking a mobile phone as an example, in general, the screen of the mobile phone is likely to be the origin of the coordinate system of the mobile phone screen due to the process error of the bonding, but the point coordinate conversion provided by the embodiment of the present invention is adopted. The implementation of the method can effectively avoid the errors caused by the above problems. In addition, the problem of the light leakage of the mobile phone screen, the quality of the captured image, and the like may affect the testing process more or less. However, the implementation scheme provided by the embodiment of the present invention excludes the influence of the processing process of the screen boundary of the mobile phone, and the pattern is The more the number of logo patterns in the more, the richer the information used to determine the conversion relationship, the more accurate the determined conversion relationship will be, thereby improving the point in the image that fits the position corresponding to the coordinate system of the mobile phone screen. The precision of the coordinates.

For example, after obtaining the conversion relationship, the mobile phone can switch the interface presented by the mobile phone screen, for example, closing the current full-screen display and presenting the main interface of the mobile phone, as shown in FIG.

If the current attempt to simulate the user's click on the calendar icon to detect the sensitivity of the area of the calendar icon in the mobile phone screen, the test device can pre-acquire the location that the user actually needs to click when clicking the calendar icon, that is, the calendar icon is on the mobile phone screen. The display position on the top. In the embodiment of the present invention, referring to the implementation manner of positioning the mobile phone screen, the corner point on the upper left side of the mobile phone screen shown in FIG. 9 is also a two-dimensional coordinate system. The origin, the two-dimensional coordinate system based on the screen of the mobile phone is established, wherein the established coordinate system is the same as the two-dimensional coordinate system established on the screen of the mobile phone when the screen of the mobile phone is positioned, and the resolution of the screen of the mobile phone is still 1280*720. According to the display position of the center point of the calendar icon in the screen of the mobile phone, and the established two-dimensional coordinate system, the coordinates of the center point of the calendar icon, that is, the point coordinates (288, 213) shown in FIG. 10 are obtained. In other words, if the user attempts to click the calendar icon during the actual operation, the user needs to generate a click operation on the corresponding point in the two-dimensional coordinate system (288, 213). This means that the above point coordinates (288, 213) are used as input to the test equipment.

As shown in FIG. 11, the black-and-white image obtained by processing the grayscale image of the mobile phone captured by the industrial camera is calculated based on the previously obtained conversion relationship, and the coordinates of the point (288, 213) are calculated in the two-dimensional coordinate system shown in FIG. Corresponding point coordinates, ie (1620, 532). In other words, the test device can determine the position of the click operation that the robot arm needs to perform in the image collected by the industrial camera according to the input point coordinates (288, 213) and the obtained conversion relationship. For the robot arm, after the test device completes the calculation of the above conversion relationship, the robot arm is controlled to perform a click operation at a point of coordinates (1620, 532).

It should be noted that since the conversion relationship has already taken into consideration that the direction in which the mobile phone is placed will change, the conversion relationship of the mobile phone has been completed in the obtained conversion relationship, which means that no matter what the mobile phone is taking. Positioning is performed. Once the conversion relationship is determined, the position of the mobile phone does not change, and the relative position of the mobile phone when the industrial camera captures the image does not change, then in the process of finding a point in the image according to a point on the screen of the mobile phone. , you can directly apply the existing conversion relationship without any other operations. In addition, in order to facilitate performing a click operation on each position on the screen of the mobile phone, and reducing data interaction between the mobile phone and the test device, in the embodiment of the present invention, the mobile phone may pre-set each icon presented in the interface that needs to be tested on the screen of the mobile phone. The coordinates are sent to the test device. When the device to be tested needs the coordinates of an icon on the screen of the mobile phone, the coordinates of the icon sent by the previously received mobile phone can be directly input, and the coordinate is obtained by means of coordinate conversion. The position of the icon in the image, and the coordinates indicating the position of the icon in the image are transmitted to the robot arm for the robot arm to perform a click operation on the corresponding coordinates in the image.

It can be seen that by using each of the logo patterns in the pattern, the conversion relationship of the coordinates of the point between the screen of the mobile phone and the image of the mobile phone taken by the industrial camera can be effectively determined, and the conversion relationship can not only locate the screen angle of the mobile phone in the image. The position of the point can further locate the area of the screen of the mobile phone. Similarly, according to the conversion relationship, the position of any point or multiple points in the screen of the mobile phone can be located in the image.

A positioning device of the terminal screen may be disposed in the terminal, and the positioning device of the terminal screen includes a hardware structure and/or a software module corresponding to each function in order to implement the above functions. Those skilled in the art will readily appreciate that the present invention can be implemented in a combination of hardware or hardware and computer software in combination with the elements and algorithm steps of the various examples described in the embodiments disclosed herein. Whether a function is implemented in hardware or computer software to drive hardware depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods for implementing the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present invention.

The embodiment of the present invention may divide the function module of the positioning device of the terminal screen according to the above method example. For example, each function module may be divided according to each function, or two or more functions may be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of the module in the embodiment of the present invention is schematic, and is only a logical function division, and the actual implementation may have another division manner.

As shown in FIG. 12, it is a possible structural diagram of the positioning device of the terminal screen involved in the above embodiment. The positioning device 30 of the terminal screen includes an acquisition module 31 and a determination module 32. The obtaining module 31 is configured to support the positioning device 30 of the terminal screen to complete the acquisition of the first image; the determining module 32 is configured to support the positioning device 30 of the terminal screen to determine the conversion relationship between the first coordinate and the second coordinate, and according to the conversion. The relationship implements an area in which the terminal screen is located in the first image. In the embodiment of the present invention, the positioning device 30 of the terminal screen may further include a storage module 33 for storing program codes and data of the terminal, and a communication module 34 for supporting the positioning device 30 of the terminal screen and the positioning device of the terminal screen. Terminal, for example, data interaction between modules in a test device, and/or support for communication between the test device and other devices such as cell phones, industrial cameras, and/or other processes for the techniques described herein .

The determining module 32 can be implemented as a processor or a controller, for example, a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), and an application specific integrated circuit (Application- Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It is possible to implement or carry out the various illustrative logical blocks, modules and circuits described in connection with the present disclosure. The processor may also be a combination of computing functions, for example, including one or more microprocessor combinations, a combination of a DSP and a microprocessor, and the like. The communication module 34 can be implemented as a transceiver, a transceiver circuit, or a communication interface or the like. The storage module 33 can be implemented as a memory.

If the determination module 32 is implemented as a processor, the communication module 34 is implemented as a transceiver, and the storage module 33 is implemented as a memory, as shown in FIG. 13, the terminal 40 includes a processor 41, a transceiver 42, a memory 43, and a bus 44. The processor 41, the transceiver 42 and the memory 43 are connected to each other through a bus 44. The bus 44 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. Wait. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is shown in FIG. 13, but it does not mean that there is only one bus or one type of bus.

The steps of a method or algorithm described in connection with the present disclosure may be implemented in a hardware, or may be implemented by a processor executing software instructions. The software instructions may be composed of corresponding software modules, which may be stored in a random access memory (RAM), a flash memory, a read only memory (ROM), an erasable programmable read only memory ( Erasable Programmable ROM (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Register, Hard Disk, Mobile Hard Disk, Compact Disc Read-Only Memory (CD-ROM), or any of those well known in the art. Other forms of storage media. An exemplary storage medium is coupled to the processor to enable the processor to read information from, and write information to, the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium may be deployed in the same device, or the processor and the storage medium may be deployed as separate components in different devices.

The specific embodiments of the present invention have been described in detail with reference to the embodiments of the present invention. The scope of the invention is to be construed as being included in the scope of the embodiments of the invention.

Claims (25)

A method for locating a terminal screen, the method comprising: Acquiring a first image, the first image is an image of a photographed terminal, the terminal is provided with a screen, the screen displays a pattern, and the pattern includes a regularly distributed logo graphic; Determining a conversion relationship between the first coordinate and the second coordinate, the first coordinate is used to indicate a position of the logo graphic on the screen, and the second coordinate is used to represent the logo graphic in the first a position in an image; An area of the screen in the first image is determined according to the conversion relationship. The method according to claim 1, wherein the determining the area of the screen in the first image according to the conversion relationship comprises: Determining, according to the conversion relationship and the third coordinate, a fourth coordinate, where the third coordinate is used to indicate a position of each corner point on the screen on the screen, and the fourth coordinate is used to represent the each corner Pointing a position in the first image; In the first image, an area formed by determining a position of each of the corner points corresponding to the fourth coordinate as an end point is an area of the screen in the first image. A method according to claim 1 or 2, wherein said pattern comprises at least three logo patterns. The method according to any one of claims 1 to 3, wherein the rule distribution comprises a center point equidistance of the logo pattern distributed in a matrix in rows and columns. The method according to any one of claims 1 to 4, wherein the logo pattern comprises a regular logo pattern and a special logo pattern, the conventional logo pattern being identical, the special logo pattern being different from the conventional A logo graphic, and the special logo graphic is used to distinguish a direction in which the terminal is placed at the time of shooting. The method according to any one of claims 1 to 5, wherein the logo pattern is black, and an area other than the logo pattern in the pattern is white. The method according to any one of claims 1 to 6, wherein the position of the center point of the logo pattern does not change after the logo pattern is deformed due to a change in the photographing angle. The method of claim 7 wherein said logo pattern is circular or elliptical. The method according to any one of claims 1 to 8, wherein the resolution of the pattern is the same as the resolution of the screen. A method for locating a terminal screen, the method comprising: The terminal displays a pattern through a screen, the pattern includes a regularly distributed logo graphic, and the first coordinate is used to indicate a position of the logo graphic on the screen; The industrial camera acquires a first image and determines a second coordinate, the second coordinate is used to indicate a position of the logo graphic in the first image, and the first image is an image of the photographed terminal; The testing device acquires the first coordinate and the second coordinate, and determines a conversion relationship between the first coordinate and the second coordinate; The test device determines an area of the screen in the first image according to the conversion relationship. The method of claim 10, wherein after the testing device determines a conversion relationship between the first coordinate and the second coordinate, the method further comprises: The testing device acquires a fifth coordinate, where the fifth coordinate is used to represent a location to be tested on the screen; The testing device converts the fifth coordinate according to the conversion relationship to obtain a sixth coordinate, and sends the sixth coordinate to the robot arm, where the sixth coordinate is used to indicate that the to-be-tested position is in the first image Position in The robotic arm performs an operation at a position of the sixth coordinate in the first image in response to a message transmitted by the test device. A positioning device for a terminal screen, characterized in that the device comprises: An acquiring module, configured to acquire a first image, where the first image is an image of a photographed terminal, the terminal is provided with a screen, and the screen displays a pattern, where the pattern includes a regularly distributed logo graphic; a determining module, configured to determine a conversion relationship between the first coordinate and the second coordinate, where the first coordinate is used to indicate a position of the logo graphic displayed on the screen by the display module, the second coordinate Means for indicating a position of the logo graphic in the first image acquired by the acquiring module; The determining module is further configured to determine an area of the screen in the first image according to the conversion relationship. The device according to claim 12, wherein the determining module is further configured to: Determining, according to the conversion relationship and the third coordinate, a fourth coordinate, where the third coordinate is used to indicate a position of each corner point on the screen on the screen, and the fourth coordinate is used to represent the each corner Pointing a position in the first image; In the first image, an area formed by determining a position of each of the corner points corresponding to the fourth coordinate as an end point is an area of the screen in the first image. Apparatus according to claim 12 or claim 13 wherein said pattern comprises at least three logo patterns. The apparatus according to any one of claims 12 to 14, wherein the rule distribution comprises a center point equidistant spacing of the logo pattern distributed in a matrix in rows and columns. The apparatus according to any one of claims 12 to 15, wherein said logo pattern comprises a regular logo pattern and a special logo pattern, said conventional logo pattern being identical, said special logo pattern being different from said conventional A logo graphic, and the special logo graphic is used to distinguish a direction in which the terminal is placed at the time of shooting. The apparatus according to any one of claims 12 to 16, wherein the logo pattern is black, and an area other than the logo pattern in the pattern is white. The apparatus according to any one of claims 12 to 17, wherein the position of the center point of the logo pattern does not change after the logo pattern is deformed due to a change in the photographing angle. The device according to claim 18, wherein said logo pattern is circular or elliptical. Apparatus according to any one of claims 12 to 19, wherein the resolution of the pattern is the same as the resolution of the screen. A terminal comprising a display screen, a memory, one or more processors, a plurality of applications, and one or more programs; wherein the one or more programs are stored in the memory; The one or more processors, when executing the one or more programs, cause the terminal to implement the method of any one of claims 1 to 9. A readable storage medium, wherein the readable storage medium stores instructions for causing the terminal to perform the method of any one of claims 1 to 9 when the instruction is run on a terminal method. A computer program product, characterized in that the computer program product comprises software code for performing the method of any of the preceding claims 1 to 9. A positioning system for a terminal screen, characterized in that the system comprises: a terminal, configured to display a pattern through the screen and determine a first coordinate, the pattern includes a regularly distributed logo graphic, where the first coordinate is used to indicate a position of the logo graphic on the screen; An industrial camera for acquiring a first image and determining a second coordinate, the second coordinate for indicating a position of the logo graphic in the first image, the first image being an image of the photographed terminal ; a testing device, configured to acquire the first coordinate and the second coordinate, and determine a conversion relationship between the first coordinate and the second coordinate; The test device is further configured to determine an area of the screen in the first image according to the conversion relationship. The system according to claim 24, wherein after the testing device determines a conversion relationship between the first coordinate and the second coordinate, the testing device is further configured to acquire a fifth coordinate, The fifth coordinate is used to represent a location to be tested on the screen; The testing device is further configured to convert the fifth coordinate according to the conversion relationship to obtain a sixth coordinate, and send the sixth coordinate to the robot arm, where the sixth coordinate is used to indicate that the to-be-tested location is in the The position in the first image; The robot arm is configured to perform an operation at a position of the sixth coordinate in the first image in response to a message sent by the test device.

Images