CN105814524A - Object detection in optical sensor systems - Google Patents

Abstract

Object detection techniques for use with optical sensors are described. In one or more implementations, multiple inputs are received, each of the inputs being received from a corresponding optical sensor in a plurality of optical sensors. Machine learning is used to classify each of the multiple inputs based on whether the input indicates an object detected by the corresponding optical sensor.

Description

Object Detection in Optical Sensor Systems

background

Computing devices may be configured to include touch functionality for detecting proximity of objects to initiate one or more actions. For example, touch functionality may be utilized to detect the proximity of a finger of a user's hand or other object to a display device as part of recognizing a gesture to initiate one or more functions of the computing device.

Various different types of sensors may be utilized to detect this proximity, one example of which includes the use of optical sensors. The effectiveness of optical sensors often depends on the ambient lighting conditions in which they operate. Accordingly, conventional use of optical sensors in such environments can fail, thereby impairing the user's experience and usability of the computing device as a whole, especially where touch functionality is configured as the primary input technology for use with the computing device. case.

overview

Object detection techniques for use in conjunction with optical sensors are described. In one or more implementations, a plurality of inputs are received, each of the inputs being received from a corresponding optical sensor of the plurality of optical sensors. Each of the plurality of inputs is classified using machine learning according to whether the input indicates that an object is detected by the corresponding optical sensor.

In one or more implementations, a system includes a plurality of optical sensors and one or more modules implemented at least in part in hardware. The one or more modules are configured to implement the first classifier, the second classifier and the object candidate module. The first classifier is configured to generate a first probability map describing a likelihood of the object being detected by a corresponding one of the plurality of optical sensors. The probability map is generated by taking as input an image including both infrared light and ambient light, and an image with ambient light subtracted from infrared light. The second classifier is configured to generate a second probability map describing a likelihood of the object being detected by a corresponding optical sensor of the plurality of optical sensors based at least in part on the input having the image including both infrared light and ambient light. The object candidate module is configured to use the first and second probability maps to determine whether an object has been detected.

In one or more implementations, one or more computer-readable storage media include instructions stored thereon that, in response to execution by a computing device, cause the computing device to perform operations. The operations include generating a first probability map describing a likelihood of an object being detected by a corresponding one of the plurality of optical sensors by taking an image including both infrared light and ambient light and having the ambient light subtracted from the infrared light The image of is taken as input to generate. Operations also include generating a second probability map describing a likelihood of the object being detected by a corresponding optical sensor of the plurality of optical sensors based at least in part on the input having the image including both infrared light and ambient light. Operations also include determining whether the object has been detected using the first and second probability maps.

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

Brief description of the drawings

The detailed description is described with reference to the accompanying drawings. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different instances in the specification and drawings may indicate similar or identical items.

1 is an illustration of an environment in an example implementation that may be used to employ object detection techniques for use in an optical sensor system.

FIG. 2 diagrammatically shows an example of an image captured using the optical sensor of FIG. 1 .

FIG. 3 diagrammatically shows an example of an image captured using the optical sensor of FIG. 1 and processed using a local extrema based method.

FIG. 4 illustrates an example system showing the optical sensor system of FIG. 1 in more detail.

5 depicts an example of a touch probability map.

Figure 6 depicts an example of a cascaded filtering method.

7 is a flowchart depicting a process in an example implementation in which inputs detected via optical sensors are classified using machine learning.

8 is a flow diagram depicting a process in an example implementation in which a classifier is utilized to generate a probability map that can be used to determine whether an object is detected using multiple touch sensors.

9 illustrates various components of an example device that may be implemented as any type of portable and/or computer device as described with reference to FIGS. 1-8 to implement embodiments of the object detection techniques described herein.

A detailed description

overview

The accuracy of conventional use of optical sensors for object detection often depends on the lighting conditions of the environment in which the optical sensors are placed. For example, ambient lighting conditions may have an impact on a device's ability to distinguish one object (eg, the fingertips of a user's hand) from the device's surroundings.

Object detection techniques for optical sensor systems are described. Optical sensors can be configured in various ways to detect the proximity of objects, such as incorporated in a pixel sensor design as part of a display device. Images collected by the sensors may then be processed to detect whether an object is in proximity to a corresponding one of the sensors. A variety of different techniques can be used to perform this processing. For example, machine learning techniques can be utilized to answer the question "is an object detected" for each of multiple sensors, which can be expressed as a probability map. In another example, multiple classifiers may be utilized to perform the process. For example, a first classifier may process an image including both infrared light and ambient light and an image with ambient light subtracted from infrared light to generate a probability map. A second classifier may separately process images including both infrared light and ambient light to generate another probability map. These maps can then be utilized to detect the likelihood of whether an object was detected and where that object was detected. This object detection can then be leveraged to support a variety of different functions, such as recognizing gestures, identifying specific objects, and more. Further discussion of these and other techniques can be found in the referenced sections below.

In the following discussion, an example environment that may be used to employ the object detection techniques described herein is first described. An example illustration of the techniques and processes of the present invention, which may be employed in this example environment as well as in other environments, is described next. Accordingly, the example environment is not limited to performing the example techniques and procedures. Likewise, the example techniques and procedures are not limited to implementation in the example environment.

example environment

FIG. 1 is an illustration of an environment 100 in an example implementation that may be used to employ object detection techniques for use in an optical sensor system. The illustrated environment 100 includes an example of a computing device 102 that may be configured in various ways. For example, computing device 102 may be configured as a conventional computer (e.g., a desktop personal computer, etc.), a mobile communication device (e.g., a tablet as shown, a mobile phone, a portable gaming device, a portable music device, or otherwise configured to be One or more of the user's hands holds a mobile device), an entertainment device, a set-top box communicatively coupled to a television, a wireless phone, a netbook, a game console, and those as further described with respect to FIG. 9 . As such, computing devices 102 may range from full-resource devices with sufficient memory and processor resources (e.g., personal computers, game consoles) to low-resource devices with limited memory and/or processing resources (e.g., conventional set-top boxes, handheld gaming console). Computing device 102 can also relate to software that causes computing device 102 to perform one or more operations, as well as to combinations of devices, eg, a gesture capture device and a game console, a set-top box and a remote control, and the like.

Computing device 102 is shown including input/output module 104 . Input/output module 104 represents functionality related to input to computing device 102 . For example, the input/output module 104 may be configured to receive input from a keyboard, mouse to identify gestures and cause operations corresponding to the gestures to be performed, and so forth. Inputs may be identified by input/output module 104 in a variety of different ways.

For example, input/output module 104 may be configured to recognize input received via touchscreen functionality of display device 106 to detect, for example, an object proximate to display device 106 , such as a finger of a user's hand 108 proximate to display device 106 of computing device 102 , Stylus, etc. This input may take various forms, such as to recognize movement of a finger of the user's hand 108 on the display device 106, such as tapping, drawing a line, and the like.

In various implementations, these inputs may be recognized by gesture module 110 as gestures. Various different types of gestures may be recognized by gesture module 110, such as gestures recognized from a single type of input (eg, touch gestures) as well as gestures involving multiple types of input. For example, computing device 102 may be configured to detect and distinguish between inputs based on which object was used to perform a gesture (eg, the stylus or finger described above). Additionally, while a touch input is described, identifying an object as approaching display device 106 may be made without contact with display device 106 (eg, as a "hover").

Additionally, while the following discussion may describe specific examples of input, in each instance, the type of input may be switched (e.g., touch may be used instead of a stylus, hover may be used instead of physical contact) without departing from its spirit and scope. Furthermore, while gestures are shown in the examples in the following discussion as being input using touchscreen functionality, gestures may be input by a variety of different devices using a variety of different techniques to detect the proximity of an object.

One such example that may be utilized to detect the proximity of objects is shown as optical sensor system 112 . Optical sensor system 112 includes a sensor processing module 114 that represents functionality for making, for each of optical sensors 116 , a determination as to whether an object is placed proximate to the sensor.

For example, the optical sensor 116 may be configured as part of the display device 106 as a sensor array embedded with corresponding pixels to detect object proximity as a sensor-in-pixel (SIP) panel. For example, optical sensor 116 may be configured as an infrared sensor configured to detect infrared (IR) light to support an optical mode of interaction with computing device 102 . Optical sensor 116 in this IR configuration is embedded in display device 106 to capture an IR image of the environment surrounding display device 106 and even computing device 102 as a whole, especially when an object touches the display device (e.g., a user). touch the screen).

Optical detection by optical sensor 116 and subsequent processing by sensor processing module 114 allows optical sensor system 112 to map object position and motion to gestures that can be recognized as gestures by gesture module 110 and/or support other interactions such as object identification, etc. )Actions.

In the illustrated example, the sensor processing module 114 is shown as including first and second classifier modules 118 , 120 and an object candidate module 122 . The first and second classifier modules 118, 120 represent functionality for classifying whether the corresponding optical sensor 116 detects the likelihood of an object such as a proximity sensor (eg, a finger of the user's hand 108). This can be performed in various ways, such as to process images captured by optical sensor 116 to generate a probability map, as described below. The probability map may then be processed by object candidate module 122 to use the probability map to determine whether an object has been detected. An example of a system 400 employing the first and second classifier modules 118 , 120 and the object candidate module 122 is described in more detail in conjunction with FIG. 4 .

Traditional touch detection methods assume that infrared (IR) light is reflected back by the finger and forms a relatively bright spot on the SIP image captured by the sensor, where the background (non-finger area) is relatively dark. Therefore, these traditional methods are based on local extrema in the intensity landscape captured by the sensor. However, in practice, the IR map is strongly dependent on the ambient light lighting conditions in the environment. As shown in example 200 of FIG. 2, the background may be bright and there may be shadows cast by hands.

There are different types of images that can be read out directly from the optical sensor 116 . For example, a first type may involve images with both IR light and ambient light. The second type is an image that includes only ambient light. These two types of images are denoted below using "Field_0" and "Field_1", respectively.

In theory, Field_0 minus Field_1 could be performed to generate the IR component, which may be denoted Field_IR in the remainder of this discussion. The Field_IR image is (in theory) expected to be invariant to ambient lighting conditions. However, there are practical problems with this environmental offset technique. First, Field_0 and Field_1 are obtained at the same time. Therefore, when the object is moving, pixels near the object boundary can be bright in Field_IR, which makes conventional local extrema based methods fail, as shown in the example 300 shown in FIG. 3 .

Second, the measured Field_0 and Field_1 images can be noisy both spatially and temporally. Such noise is content dependent and thus may be difficult to remove by traditional filtering methods. Furthermore, pixel intensity values are not linear with respect to sensor integration time, and thus ambient offset is not straightforward. Therefore, motion artifacts, sensor noise, and nonlinear response of the optical sensor make speckle detection on Field_IR unreliable and lead to poor detection accuracy, which is also shown in the example 300 of FIG. 3 .

Accordingly, optical sensor system 112 may be configured to employ machine learning classification techniques to robustly detect object locations from input received from optical sensor 116 in real time under a wide variety of lighting conditions. A machine learning classifier can be utilized to answer the per-sensor (e.g., per-pixel sensor) question "Is this optical sensor detecting an object adjacent to this sensor?" Various techniques such as Randomized Decision Forests (RDF ) classifiers or other types of machine learning classifiers. In the following, techniques are described that include using machine learning classifiers to classify pixels when detecting objects, using multiple classifiers trained on different signals from the optical sensor 116 to increase detection accuracy, using multiple types of Classify split functions to increase detection accuracy, and use split functions. While touch input is described below as an example, it should be readily apparent that touch input may also be possible without involving actual physical contact between the object (e.g., a finger of the user's hand 108) and a surface associated with the display device 106 or the optical sensor 116. These inputs are detected in the absence of

In order to efficiently and robustly detect object position by optical sensor 116 under a range of lighting conditions, a machine learning discriminative classification is utilized below to answer each optical sensor question as described above, namely: "Is this optical sensor detecting Is there an object near this sensor?"

Given a set of classes "C={c1,c2,...,ck}", a discriminative classifier may be employed by the sensor processing module 114 configured as an algorithm that, for a given input "X", returns Discrete probability distribution for the class "C" conditioned on the input "X". In the case of a difficult problem involving the above problem, the relevant class sets discussed below are "touch" and "no touch", and the input to the classifier used is the sensor image patch around the pixel location. The classifier examines for each optical sensor 116 input, e.g., each pixel in the sensor image captured by that optical sensor 116, and returns for each optical sensor 116 its probability of being touched by (e.g., in contact with) a fingertip .

For example, an RDF classifier may be utilized by the sensor processing module 114 due to its efficiency and classification performance. RDF classifiers can be configured to utilize randomized decision tree (RDT) ensembles. The output of this RDF for a given input "X" can be computed by utilizing the output of each of its RDTs for the input "X".

RDTs are binary decision trees in which each internal (i.e., non-leaf) node has an associated "split" binary function that, when applied to an input, returns the input to be routed to Whether the node's "right" child or "left" child is determined. Each leaf node in the RDT is associated with a discrete probability distribution over the class set "C".

The classification process in an RDT for a given input "X" starts in the root node of the tree to process "X" by applying the root's associated split function. Processing of the input continues recursively on the child nodes corresponding to the results of the split function. When the process reaches a leaf node, the probability distribution associated with such leaf node is returned as output.

FIG. 4 depicts an example implementation of an overview of a system 400 employing an RDF classifier. As shown, two RDF classifiers are used in this example. The first classifier takes as input the Field_IR and Field_0 sensor images and takes as output a dense touch probability map "Prob". Each pixel value in the probability map "Prob" indicates the probability that the corresponding optical sensor (eg, sensor pixel) is in contact with an object (eg, fingertip).

As can be seen in the example implementation 500 shown in FIG. 5 , not only is this touch probability map clearer than Field_IR, but the appearance of the "touch area" (i.e., the fingertip-panel contact area) is more uniform, as shown in the figure Cyclic symmetry bright spots are shown. In general, the touch area in "Prob" is easier to describe than in Field_IR.

Returning again to the system 400 of FIG. 4 , in a subsequent step, the candidate touch location is assumed by blob detection to be at the geometric center of the high touch probability blob in "Prob". For example, a "difference of Gaussian" (DOG) based blob detector can be utilized to locate touch candidates. The probability map "Prob" may be convolved with a set of Gaussian filters at different blur scales, as shown in the example 600 of the cascaded filtering method in FIG. 6 .

A DOG image at scale "σ" can be computed as follows:

$\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&CircleTimes;</span>\mathrm{<span\; class="notranslate">\; PR</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>$

Subsequently, a pixel that is a scale-space local extremum in “D(x,y,σ)” is detected as a touch candidate.

In one or more implementations, each sample point is compared to its 8 neighbors in the current image and 9 neighbors in scales above and below, respectively. If the sample point is larger than each of the 26 neighbors, then the sample point is selected, as shown in example 600 of FIG. 6 . Use the blob contrast value "D(x,y,σ)" of the touch candidate "(x,y)". If the contrast is greater than a predefined threshold "T" (eg, predefined by the user), then pixel "(x,y)" is reported as a final touch. The detection accuracy mainly depends on the threshold "T". For example, if "T" is too small, an increased number of true touches can be located, but at the same time, a large number of false positives can be introduced into the detection result. On the other hand, if "T" is large, fewer false positives may be obtained at the expense of an increased number of false negatives.

A set of touch candidates, each touch candidate associated with a contrast value "D(x,y,σ)", is obtained as output of the blob detection step. To better distinguish between touch and non-touch image data, a second classifier trained only on Field_0 image data can be included in the system 400 that takes as input the list of touch location candidates returned by the blob detector. This second classifier can be tuned to improve the touch and non-touch data separation performance achieved by the first pixel-wise touch classifier which may be limited by ambiguity in its input signal. For example, as can be observed in Figure 3, while motion artifacts and noise can look very similar to actual touch signals in Field_IR, the appearance of touch and non-touch signals in the Field_0 image is significantly different.

The output of the second classifier can be configured as a dense probability map "Prob", which can be smoothed by convolving it with a Gaussian filter with a kernel size slightly larger than the diameter of the average fingertip touch area.

Finally, each touch candidate location "(x,y)" can be associated with a 2-D measurement as follows:

DP(x,y)=(D(x,y,σ),Prob'(x,y))

This expression includes the corresponding touch probability "Prob'(x,y)" and the contrast value "D(x,y,σ)" from the output of the blob detector. If the 2D point "DP(x,y)" is larger than the pre-defined 2D decision boundary, the touch candidate "(x,y)" is considered to be an actual touch. For example, the decision boundary may be configured as a specific gamma curve whose parameters are selected based on the training data and the desired trade-off between FP and FN classification rates.

For example, it can be observed that in general true touches have high contrast or high probability values. Thus, the touch probability "Prob'(x,y)" can serve as a cue for rejecting false touches without the number of false positives increasing significantly. Thus, classification using this 2D decision boundary can be used to better distinguish between real and fake touches, rather than using a single-dimensional decision boundary used with the contrast dimension of the output from the first classifier. Although the use of two classifiers and a 2D decision boundary is shown in the example system 400 of FIG. 4 , the approach can be extended to additional dimensions, ultimately resulting in a classification framework with higher classification accuracy.

classifier feature function

The classifier uses six different types of segmentation functions. Let "I(x)" be the 2D position on the input field "I" The intensity of the pixel. order for a given For the following discussion, let these expressions remain:

I _x ((x,y)=I((x+1,y))–I((x–1,y))

I _y ((x,y)=I((x,y+1))–I((x,y+1))

$\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\sqrt{{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

The partition function can then be expressed as follows:

I) Intensity difference: $f_{\mathrm{\&theta;}}^1(I,x)=I(x+u)-I(x+v),$ where θ = (u, v) and

2) Raw strength: $f_{\mathrm{\&theta;}}^2(I,x)=I(x+u),$ where θ = (u), and

3) Focus: $f_{\mathrm{\&theta;}}^3(I,x)=\underset{v=\{-1,1\}2}{max}|I(x+u+v)-I(x+u)|,$ where θ = (u), and

4) Gradient amplitude: where θ = (u), and

5) Gradient direction:

in and

6) Gradient orientation difference:

where θ = (u, v) and

All six types of segmentation functions can be used in the first classifier, while the second classifier can be limited to using intensity difference segmentation functions. Naturally, various other examples are also contemplated without departing from the spirit and scope thereof.

train classifier

data collection

To train the classifier, an offline ground-truth data collection stage can be utilized in which sensor images are collected for various touch and non-touch events under a wide range of lighting conditions (i.e., Field_0 and Field_IR) for a sufficiently large number of video sequences. To collect these positive (touch) and negative (non-touch) samples, the SIP panel or other arrangement of optical sensors 116 can be touched by a finger with different pressures, postures and orientations. Naturally, other objects may also be utilized, as well as samples that do not involve physical contact.

A second offline manual data labeling step can also be utilized in which, for each touch in each collected image, the pixel at the center of each fingertip touch (i.e., the "touch location") is labeled . Positive touch pixel samples can then be generated for each pixel inside a small radius disk centered at each touch location. A fixed radius that does not exceed the radius of the average touch area may be used.

Negative training samples can be generated by randomly selecting pixels outside the touch area. Each pixel in a small width circle around each touch location but with a radius slightly larger than the touch area is also marked as a negative sample.

RDF training process

In one or more implementations, the maximum height of the RDT may be set at a value of approximately 20. RDF classifiers are trained by independently training each of its RDTs one node at a time, starting from its root node. Each node is trained using the input training set. Initially, the entire training set is the input set used to train the root node.

Given an input training set "T" for a node "n", the node is trained by sampling "N" times the square root of the parameter space of each type of partition function, where "N" is the partition The cardinality of the function's parameter space. This sampling is done for each input image (ie, Field_IR or Field_0). Furthermore, for each sampled parameter, the number of possible thresholds is also sampled.

For a given split combination "Σ=(field,splitF,θ,τ)" of input field "field", split function type "splitF", split function parameterization "θ" and threshold "τ", each input "x ∈T" is split according to whether the value of "splitF _θ (x)" is less than, greater than or equal to the threshold "τ".

Let "Σ" be the partition combination that achieves the maximum information gain "IG" over partitions of each element in the node input set "T" over each sampled partition combination. If "IG" is too small or if the node is at the maximum preselected height (eg, 20), then the node is set to a leaf node and the touch probability associated with the leaf node is set to the number of touch samples equal to the node The ratio of the total number of samples in the data set "T". On the other hand, if "IG" is high enough, the split combination "Σ" is associated to node "n" whose input set "T" is split using "Σ" into two subsets " _TL " and "T _R ”, and “n” are assigned two child nodes, each node is recursively trained using the input sets “T _L ” and “T _R ”.

The dataset used to train each classifier

The first classifier "Pixelwise Touch Classifier" in Figure 4 can be trained on the Field_0 and Field_IR images using each of the manually labeled positive and negative samples. After the classifier is trained, each false positive of the first classifier on the training data is collected. Specifically, the entire training data is run through a first classifier and a blob detector applied to each corresponding output probability map using a small contrast threshold "T" (eg, T=5). A false positive may be considered for a reported touched pixel if the pixel is not close enough to the measured touch location.

The second classifier (ie, the pixel-wise FP filter in Figure 4) was trained on the Field_0 image using each of the manually labeled positive samples but as negative samples. Each of the detected false positive cases of the training data by the first classifier is employed. Thus, using multiple input fields together with six segmentation functions for the first classifier can be observed to achieve a significant performance improvement over using only the Field_IR domain and the dense difference segmentation function.

example process

The following discussion describes object detection techniques that may be implemented using the previously described systems and devices. Aspects of each of these processes may be implemented in hardware, firmware, software, or a combination thereof. Processes are shown as a set of blocks that specify operations performed by one or more devices, and are not necessarily limited to the order shown for performing operations by the corresponding blocks. In portions of the following discussion, reference will be made to Figures 1-6.

FIG. 7 depicts a flowchart of a process 700 in an example implementation in which inputs detected via optical sensors are classified using machine learning. A plurality of inputs are received, each of the inputs being received from a corresponding optical sensor of the plurality of optical sensors (block 702). Computing device 102 may, for example, include a plurality of optical sensors 116 arranged in an array as part of a pixel sensor configuration of display device 106 .

Each of the plurality of inputs is classified using machine learning with respect to whether the inputs indicate that an object is detected by the corresponding optical sensor (block 704 ). The sensor processing module 114 of the optical sensor system 112 may, for example, employ classifiers that perform machine learning to generate probability maps that describe the relative likelihood that the corresponding optical sensors depicted in the figure detect the proximity of objects.

The results of the classification are used to determine the location of the object (block 706). Continuing with the previous example, one or more locations of objects can then be detected from the probability map, which can be exploited to support a wide range of functionality. An example of such functionality is shown in FIG. 1 as gesture module 110 configured to initiate (eg, through an operating system, application, etc.) operation of a computing device in response to recognizing a corresponding gesture. Various other examples are also contemplated, such as identifying a particular type of object (eg, stylus vs. touch), and the like.

8 depicts a flow diagram of a process 800 in an example implementation in which a classifier is utilized to generate a probability map that can be used to determine whether an object is detected using multiple touch sensors. A first probability map is generated describing a likelihood of an object being detected by a corresponding one of the plurality of optical sensors. The probability map is generated by taking as input an image including both infrared light and ambient light and an image with ambient light subtracted from infrared light (block 802). As shown in FIG. 4, for example, the first classifier module 118 of FIG. 1 may be implemented as a "Pixel-wise Touch Classifier (RDF)" which takes the Field-IR image and the FIELD_0 image as input.

The second classifier is configured to generate a second probability map (box 804). Continuing with the previous example, the second classifier module 120 of FIG. 1 may be implemented as a "pixel-wise FP filter (RDF)" that takes the FIELD_0 image as input.

It is determined whether the object has been detected using the first and second probability maps (block 806). For example, the object candidate module 122 of FIG. 1 may be implemented as shown in FIG. 4 to support blob detection and two-dimensional thresholding to form an output including the coordinates of an object if detected. Various other examples are also contemplated as described above.

Example Systems and Devices

FIG. 9 illustrates generally at 900 an example system including an example computing device 902 , which represents one or more computing systems and/or devices that may implement various techniques described herein. Computing device 902 may be, for example, a server of a service provider, a device associated with a client (eg, a client device), a system on a chip, and/or any other suitable computing device or computing system.

The illustrated example computing device 902 includes a processing system 904, one or more computer-readable media 906, and one or more I/O interfaces 908 communicatively coupled to one another. Although not shown, computing device 902 may further include a system bus or other data and command transport system coupling the various components to each other. A system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or local bus utilizing any of a variety of bus architectures . Various other examples are also contemplated, such as control and data lines.

Processing system 904 represents functionality to perform one or more operations using hardware. Accordingly, processing system 904 is shown to include hardware elements 910 that may be configured as processors, functional blocks, or the like. This may include implementation in hardware as an application specific integrated circuit or other logic device constructed using one or more semiconductors. Hardware elements 910 are not limited by the materials from which they are formed or the processing mechanisms utilized therein. For example, a processor may be formed from semiconductors and/or transistors (eg, electronic integrated circuits (ICs)). In this context, processor-executable instructions may be instructions that are electronically executable.

Computer readable storage media 906 is shown including memory/storage 912 . Memory/storage 912 represents memory/storage capacity associated with one or more computer-readable media. Memory/storage component 912 may include volatile media such as random access memory (RAM) and/or nonvolatile media such as read only memory (ROM), flash memory, optical disks, magnetic disks, and the like. The memory/storage component 912 may include fixed media (eg, RAM, ROM, fixed hard drives, etc.) as well as removable media (eg, flash memory, removable hard drives, optical disks, etc.). Computer readable medium 906 can be configured in various ways as described further below.

Input/output interface 908 represents functionality that allows a user to enter commands and information into computing device 902 and also allows information to be presented to the user and/or other components or devices using various input/output devices. Examples of input devices include keyboards, cursor control devices (e.g., mice), microphones, scanners, touch capabilities (e.g., capacitive or other sensors configured to detect physical touch), cameras (e.g., invisible wavelengths of infrared frequencies to recognize movement as gestures that do not involve touch), and so on. Examples of output devices include display devices (eg, monitors or projectors), speakers, printers, network cards, haptic response devices, and the like. Accordingly, computing device 902 may be configured in various ways as described further below to support user interaction.

Various techniques may be described herein in the general context of software, hardware elements or program modules. Generally, such modules include routines, programs, objects, elements, components, data structures, etc. that perform particular tasks or implement particular abstract data types. As used herein, the terms "module," "function," and "component" generally represent software, firmware, hardware, or a combination thereof. Features of the technology described herein are platform independent, meaning that the technology can be implemented on a variety of commercial computing platforms with a variety of processors.

An implementation of the described modules and techniques may be stored on or transmitted across some form of computer readable media. Computer-readable media can include a variety of media that can be accessed by computing device 902 . By way of example, and not limitation, computer readable media may include "computer readable storage media" and "computer readable signal media."

A "computer-readable storage medium" may refer to a medium and/or device that enables persistent and/or non-transitory storage of information, as opposed to merely a signal transmission, a carrier wave, or the signal itself. Thus, computer-readable storage media refers to non-signal bearing media. Computer-readable storage media include storage devices such as volatile and non-volatile, removable and non-removable media and/or storage device hardware. Examples of the computer-readable storage medium include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, hard disk, magnetic tape cartridge, magnetic tape, magnetic disk storage or other magnetic storage device, or other storage device, tangible medium, or article of manufacture suitable for storing the desired information and accessible by a computer.

"Computer-readable signal medium" may refer to a signal-bearing medium configured to transmit instructions to hardware of computing device 902, such as via a network. Signal media typically embody computer readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave, data signal, or other transport mechanism. Signal media also include any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

As previously described, hardware elements 910 and computer-readable media 906 represent modules implemented in hardware, programmable device logic, and/or fixed device logic that may be employed by certain embodiments to implement the techniques described herein At least some aspects of , such as executing one or more instructions. Hardware may include integrated circuits or systems on a chip, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and components implemented in silicon or other hardware. In this context, hardware is operable as a processing device that performs program tasks through instructions and/or logic implemented by hardware, as well as hardware used to store instructions for execution (such as the computer-readable storage medium described above) .

Combinations of the foregoing may also be employed to implement the various techniques described herein. Thus, software, hardware, or executable modules may be implemented as one or more instructions and/or logic on some form of computer-readable storage medium and/or by one or more hardware elements 910 . Computing device 902 may be configured to implement specific instructions and/or functions corresponding to software and/or hardware modules. Accordingly, implementation of modules executable as software by computing device 902 may be at least partially implemented in hardware, eg, through the use of computer-readable storage media and/or hardware elements 910 of processing system 904 . The instructions and/or functions may be executable/operable by one or more articles of manufacture (eg, one or more computing devices 902 and/or processing system 904 ) to implement the techniques, modules, and examples described herein.

As further shown in FIG. 9 , example system 900 enables a ubiquitous environment for a seamless user experience when running applications on personal computers (PCs), television devices, and/or mobile devices. Services and applications run substantially similarly in all three environments for a common user experience when transitioning from one device to the next when using applications, playing video games, watching videos, and so on.

In example system 900, multiple devices are interconnected through a central computing device. The central computing facility may be local to the plurality of devices, or may be located remotely from the plurality of devices. In one embodiment, the central computing device may be a cloud of one or more server computers connected to multiple devices through a network, the Internet, or other data communication links.

In one embodiment, the interconnection architecture enables functionality to be delivered across multiple devices to provide a common and seamless experience to users of the multiple devices. Each of the multiple devices may have different physical requirements and capabilities, and the central computing device uses a platform to enable an experience that is customized for the device and yet common to all devices can be delivered to the device. In one embodiment, a class of target devices is created and experiences are adapted to the general class of devices. A device class may be defined by the physical characteristics of the device, type of use, or other common characteristics.

In various implementations, computing device 902 can take on a variety of different configurations, such as for computer 914 , mobile device 916 , and television 918 purposes. Each of these configurations includes devices that may have generally different configurations and capabilities, and thus computing device 902 may be configured according to one or more of the different device classes. For example, computing device 902 may be implemented as a computer 914 class of device, which class includes personal computers, desktop computers, multi-screen computers, laptop computers, netbooks, and the like.

Computing device 902 may also be implemented as a device of the mobile device 916 class, which includes mobile devices such as mobile phones, portable music players, portable gaming devices, tablet computers, multi-screen computers, and the like. Computing device 902 may also be implemented as a television 918 class of device, which includes devices that have or are connected to generally larger screens in a casual viewing environment. These devices include televisions, set-top boxes, game consoles, and more.

The techniques described herein may be supported by these various configurations of computing device 902 and are not limited to the specific examples described herein. This functionality may also be implemented in whole or in part through the use of a distributed system, such as through a "cloud" 920 via a platform 922 as described below.

Cloud 920 includes and/or represents a platform 922 of resources 924 . Platform 922 abstracts the underlying functionality of the hardware (eg, servers) and software resources of cloud 920 . Resources 924 may include applications and/or data that may be used when computer processing is performed on a server located remotely from computing device 902 . Resources 924 may also include services provided on the Internet and/or over a subscriber network, such as a cellular or Wi-Fi network.

Platform 922 can abstract resources and functionality to connect computing device 902 with other computing devices. Platform 922 can also be used to abstract scaling of resources to provide corresponding scaling levels to requirements encountered by resources 924 implemented via platform 922 . Accordingly, implementation of the functionality described herein may be distributed across system 900 in an interconnected device embodiment. For example, the functionality may be implemented partially on computing device 902 and via platform 922 that abstracts the functionality of cloud 920 .

epilogue

Although the invention has been described in language specific to structural features and/or methodological acts, it is to be understood that the invention defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed invention.

Claims (10)

1. one kind for using the input obtained from multiple optical pickocffs to detect the side of the adjacency of object Method, described method is realized by one or more calculating, and includes:

Receiving multiple input, it is corresponding that each in described input is received from the plurality of optical pickocff Optical pickocff；And

Detecting described object respective sensor whether in the plurality of sensor, described detection includes making With machine learning, according to the plurality of input, whether denoted object is detected by corresponding described optical pickocff Each in the plurality of input is classified.

2. the method for claim 1, it is characterised in that farther include to use the knot of described classification Fruit determines the position of described object, and can be used for initiating to set described calculating based on the described result identification determined The posture of standby operation.

3. the method for claim 1, it is characterised in that described classification is to use multiple graders Perform.

4. method as claimed in claim 3, it is characterised in that:

Grader described in first performs described classification in the following manner: will include infrared light and ambient light two The image of person and have and deduct the image of ambient light from infrared light and be taken as input, and generate the described object of description The probability graph detected by the respective optical sensor in the plurality of optical pickocff；And

Grader described in second is based at least partially on has the image including infrared light and ambient light Input performs described classification.

5. method as claimed in claim 4, it is characterised in that the input that grader described in second is taken is One or more speckle-detection technique is used to process.

6. the method for claim 1, it is characterised in that the plurality of optical pickocff is arranged to Forming array, described array is as a part for the element sensor function of the display device of the equipment of calculating.

7. the method for claim 1, it is characterised in that described classification includes distinguishing formula grader Use, this distinguishes that formula grader returns and detects that the discrete probabilistic of class set of described object divides as instruction The result of cloth.

8. the method for claim 1, it is characterised in that described classification include use use one or Described input is classified by the randomized decision forest (RDF) of multiple randomized decision trees (RDT).

9. the input obtained from multiple optical pickocffs for use to detect the adjacency of object is System, described system includes:

The plurality of optical pickocff；And

At least partially with hard-wired one or more modules, the one or more module is configured to Detect described object respective optical sensor whether in the plurality of optical pickocff, one or Multiple modules are configured to realize:

First grader, it is the plurality of that described first grader is configured to generate the described object of description First probability graph of the probability that the respective optical sensor in optical pickocff detects, this probability graph By the image of infrared light and ambient light will be included and has and deduct ambient light from infrared light Image is taken as inputting and generates；

Second grader, described second grader be configured to be based at least partially on have include infrared The input of the image of light and ambient light generates description object by the plurality of optical pickocff The second probability graph of probability of detecting of respective optical sensor；And

Object candidates module, described object candidates module is configured to use described first and second probability Figure determines that object is detected the most.

10. system as claimed in claim 9, it is characterised in that described second probability graph is by described second Grader deducts the image of ambient light having of not using that described first grader used from infrared light In the case of generate.