DE10313019A1 - Single handed operation of a mobile terminal by gesture recognition, whereby movement of the terminal relative to the user is measured and used to control the positioning of input selection means - Google Patents

Abstract

Mobile terminal has a user interface with positionable selection means, means for measuring the movement of the terminal (1) relative to a user (3) and means for positioning the selection means on the user interface based on the measured relative movement. An independent claim is made for a method for operating a mobile terminal in which the movement of the terminal relative to a user is measured and used for controlling input means. The method uses face detection and tracking using an integral camera of the mobile terminal.

Description

With the miniaturization of mobile communication terminals Keyboards and other haptic input options in favor of the device size Minimum reduced. In future terminals there will even be no need for keypads or input keyboards. In order to have a practicable and comfortable interaction option to ensure, new operating concepts become necessary. Restrictive limit due to the However, the existing hardware is usually the most expensive must be used intelligently for this.

Devices without Keypad or with virtual keypads have so far either been through Voice recognition or operated by pen input, such as common with PDAs is. The voice input alone is especially in very noisy Environments difficult. On the other hand, pen entry allows no one-hand operation for mobile devices often desired becomes.

since have been mobile devices for some time available, in which a camera is integrated.

From that starting from the object of the invention is an input option for mobile terminals to create a one-handed Operation allowed and no keyboard required.

This Object is achieved by those specified in the independent claims Inventions solved. Advantageous refinements result from the dependent claims.

Accordingly has a mobile device one that can be displayed, for example, on a display, in particular optical and more preferably graphical user interface. The user interface let yourself for example in the form of, for example, on a display, in particular an LCD or LED display, represented user interface or user interface realize or as a very simple visual display with e.g. differently illuminated digits, with the not selected Digits green light up and selected red. On the graphical user interface is a virtual and / or real selection tool for selection of individual functional areas of the user interface, the functionalities in each case are assigned, in particular positionable by moving. Still has the mobile device via means for measuring a relative movement of the mobile device against one Mobile device users. Furthermore assigns the mobile device also means for positioning the selection means on the User interface depending the measured relative movement.

So the user can simply put the mobile device in, for example hold his hand and move towards yourself to open the selection agent a desired one Moving area of the user interface. Is the means of selection in this way, for example, on a desired (virtual) actuation button (Button) of the user interface has been moved, so the user by speaking a command or by pressing an input button on mobile device the one with the virtual push button trigger connected action of the mobile device.

The Means for measuring the relative movement preferably have means to recognize the position of the lips, an eye and / or the face of the User on. Then the user only needs the mobile device to align and move on his face while doing that Track the movement of the selection mark on the user interface.

At the simplest leaves measure the relative movement when the mobile terminal an optical sensor, in particular a camera, for recording of a respective image of the user. From the recorded episode of pictures then determine the relative movement.

On The graphical user interface can be used to control the relative movement as feedback the current picture of the user are output from which the user then goes to his current Position opposite the terminal conclude can.

For one use in the dark, the mobile terminal can have lighting means, for the user to illuminate and also use the optical sensor to enable.

Preferably the mobile device has one Evaluation device through which the user's image can be analyzed is by specifying segments in the captured image, the brightness-specific Features have a positional relationship of the set Segments to each other is checked and a presence of a face in the image is deduced if a selection of fixed segments a certain positional relationship having.

The selection means can be displayed as a pointing means on the graphical user interface the. However, it is also possible to signal the position of the selection means to the user by not displaying the selection means itself, but rather only displaying the element currently selected on the user interface as selected.

Around a flowing The mobile shows how to ensure movement of the selection medium terminal advantageous stabilizing agent for the movement of the selection agent on.

In a method for operating a mobile terminal measured a relative movement of the mobile terminal with respect to a user and a selection means on a user interface of the mobile terminal in dependency of measured relative movement positioned. Advantageous configurations of the method result analogously to the advantageous configurations of the mobile device and vice versa.

So is preferably used, for example, to measure the relative movement from the mobile device a camera captured an image of the user. The user's picture can then be analyzed by specifying segments in the image that have brightness-specific characteristics, a positional relationship the specified segments are checked against each other and the existence of the user, in particular a face of the user, in the captured Image is derived when a selection of specified segments is a certain positional relationship having.

The In particular, method for analyzing the image of the user includes specifying segments in the captured image that are brightness specific Features. As will be explained in more detail below, the brightness-specific features, for example light-dark transitions and / or Include dark-light transitions. Then will a positional relationship of fixed segments checked against each other, being present of a user, especially a (human) face, in the captured image is derived when a selection of fixed Segments has a specific positional relationship. The means by the method just described can only by Analysis of certain areas of the captured image, namely the Segments with brightness-specific characteristics, more precisely by checking the positional relationship the specified segments on the existence of the user, especially his Facial, and its position closed to the mobile device become. Thus, the implementation the method requires only a small amount of computing power, so a resource reduction or resource optimization of a data processing system to perform of the procedure becomes.

According to one An advantageous embodiment is the captured image an image captured by an electronic camera that is electronic is processable. In particular, the captured image is made up of individual Pixels composed, which respective grayscale values assigned are.

According to one In another advantageous embodiment, segments are recorded in the Image set where the brightness-specific characteristics sharp or abrupt changes in brightness, for example from dark to light or from light to dark. such Find (sharp) brightness transitions for example in a person's face, in particular at the transition from the forehead to the eyebrows or (for people with light hair color) at the transition from the forehead to the shadows of the eye sockets. Find such (sharp) brightness transitions but also during the transition from the upper lip area or lip area to the mouth opening or from the mouth opening to the lip area of the lower lip or to the lower lip area. On further brightness transition stands between the lower lip and the chin area, more precisely said as a shadow area (depending on the lighting conditions or light incidence) based on a slight protrusion the lower lip, a. As explained in more detail below will it is possible based on these brightness transitions or based on the positional relationship a selection of the brightness transitions the presence of a human face or in particular to conclude the position of a mouth part.

According to an advantageous embodiment, the step of defining segments in the captured image includes processing the image using a gradient filter, in particular a horizontal gradient filter, in order in particular to achieve horizontal (sharp) brightness transitions, such as those on the eyebrows, on the eyes, or on the mouth make visible. The processing by means of the gradient filter produces a first processed image, in which, at the location of a brightness transition in the captured image, corresponding pixels are provided in the first processed image with a grayscale value, which depend on the sharpness of the brightness transition in the captured image. This means that when the brightness changes within a few pixels from a low brightness value (low grayscale value) to a high brightness value (high grayscale value), corresponding information is entered in the first processed image. It is assumed that the first edited image is shown in the form of a grayscale image If, for example, a positive horizontal gradient filter is applied to the captured image from top to bottom, areas of the image in which there are no brightness transitions can be displayed in medium grayscale values, while light-dark transitions (from above downwards) depending on the sharpness of the transitions receive a correspondingly higher gray value than the mean gray value and dark-light transitions (from top to bottom) receive a correspondingly lower gray value depending on the sharpness of the respective transition.

According to one Another advantageous embodiment can be the step of setting of segments in the captured image binarization processing the first edited image include. A second can be edited Image are generated at the corresponding pixels, their grayscale value a first predetermined threshold (which is dependent on the brightness of the captured image is selected), a binary "1" value is assigned while the remaining pixels a binary "0" value is assigned. In other words, can the pixels whose grayscale value exceeds the predetermined first threshold value assigned the value "white" while the rest Pixels the value "black" is assigned. Thus, only brightness transitions are left in the second processed image a certain pronounced sharpness to recognize.

Contiguous pixels, which a white value can be assigned can be seen as a unit of pixels for the following evaluation inscribed in a geometric shape, such as a rectangle become. This geometric shape can then be used as a fixed segment To be defined. Since it is necessary from the captured in the Image captured scene biometric features, especially a face regarding a human being, filtering out or extracting, to be able to conclude that there is a human face it is advantageous to identify certain areas or structures in the Image that, for example, is not biometric due to its size or shape Characteristics regarding of a face.

According to one advantageous areas are certain areas of second processed image with contiguous pixels on the ground of predetermined parameters regarding size and / or shape of the area or discarding the geometrical form that describes it and in particular, the pixels assigned to the areas are highlighted set a black value. A criterion regarding the shape of areas related Pixels with a white value can be, for example, that the height of this area or this Structure too great a value in contrast to the width of the structure. That means that relationship from height too wide is too big. Such a definition becomes clear when - as already mentioned above - segments based on brightness-specific characteristics of horizontally running Eyebrows or horizontal lips or mouth be determined. This in turn means that the previously captured Image with a large Number of pixels on only a small number of fixed Segments is returned, from their positional relationship to each other now the presence and position of a face can be derived.

According to an advantageous embodiment, in the step of checking the positional relationship in a first examination step according to a first detection mechanism, each of the defined segments is examined to determine whether there is a second defined segment for a segment to be examined, which segment is on a horizontal line or one in the Is essentially horizontal line to the currently examined fixed segment. Starting from a captured image, consisting of a plurality of pixels, the second segment does not necessarily have to lie on a horizontal line of pixels encompassed by the segment to be examined, it can also be higher or lower by a predetermined small amount of pixels with respect to the horizontal or horizontal line. If a second defined horizontal segment is found, then a third defined segment is searched for, which is located below the examined and the second defined segment and for which it applies that a distance from the examined to the second defined segment and a distance of a connecting path between the examined and the second defined segment to the third defined segment has a first predetermined ratio. In particular, a normal to the connecting path between the examined and the second defined segment can be defined, the distance from the third segment (along the normal) to the connecting path between the examined and the second defined segment being included in the first predetermined ratio. The distance between the examined and the second segment can be defined in such a way that the distance from a respective center of the defined segments is used. A center point can easily be determined, in particular when using segments with the geometric shape of a rectangle. The first examination step just described can thus be used to infer the presence of a face by determining the positional relationship between three defined segments. It is assumed here that the examined and the second defined segment represent a respective eyebrow section in the face of a human being, which normally has a striking or sharp light-darkness-brightness transition from top to bottom and is therefore clearly recognizable in the second processed image. The third defined segment represents a segment of a mouth area or the border area forming a shadow between the upper lip and lower lip. It has been found that, in order to determine a human face, the first predetermined ratio between the distance between examined and second defined segments to the distance of the connecting section of the two segments to the third fixed segment with a value from 1: 1 to 1: 1.7, in particular from 1: 1.2 to 1: 1.5 is advantageous. In addition to the possibility of using eyebrows as distinctive segments with brightness-specific characteristics, it is also possible to use shadow-forming areas of the eye sockets or the eyes or the iris itself instead of the eyebrows.

According to one Another advantageous embodiment is in the step of checking the positional relationship in a second investigation step (corresponding to a second Detection mechanism) of each of the specified segments examines whether a fourth fixed segment for this segment exists that is below, especially along a vertical, the examined determined segment and for which that the horizontal length of the examined fixed segment and the distance of the fourth fixed Segments to examine a predetermined segment Ratio. This second investigation step assumes that in particular in individuals who wear glasses or sunglasses, the individual eyebrows are not recognizable, but because of the strong contours only one segment or structure is recognizable, the geometrical of the length of the two eyebrows together. So that stands too Length of this the segment comprising two eyebrows again in a specific segment relationship to remove the mouth from the user's face. So be two defined segments, in which the length of the first segment to the distance of the second segment a second predetermined relationship according to the second examination step the presence of a human face. As a valid one Area for the second predetermined ratio can again a value from 1: 1 to 1: 1.7, but in particular from 1: 1.2 to 1: 1.5 can be assumed.

According to one advantageous embodiment is in the step of checking the positional relationship first the first step of the investigation was carried out, especially according to faces to be looked for by people who are not wearing glasses. Shows the first exam step no result for the respective segments examined If no second and third segments have been found, the The second examination step is then carried out. It is also possible first to carry out the second and then the first examination step.

In the previously described method and the advantageous refinements thereof, the positional relationship of three defined segments to one another or a first predetermined ratio of the spacing of the segments to one another, in particular in the first examination step, and the positional relationship of (two) segments becomes dependent in the second examination step the size and / or shape, in particular the (horizontal) length, of the first segment can be used as a criterion for deriving the presence of a face in the captured image of a scene. To improve the process, ie to increase the likelihood that it is actually. a face is involved, according to a further advantageous embodiment, biometric features relating to the mouth area or lip area can be extracted from an analysis area around a found third or fourth defined segment in a further method step. This means that in this additional step further tests should be carried out to determine whether the third or fourth defined segment representing a mouth area actually represents a mouth area or a mouth area. Light-dark and dark-light transitions can be determined in the area of the captured image in a region that is at the position of the third or fourth defined segment and includes the respective segment, in order to determine the presence of an upper lip and a lower lip Derived mouth part. The analysis area of the captured image, the area with the suspected mouth area, is advantageously subjected to processing with a horizontally oriented gradient filter in order to obtain a third processed image. The third processed image, viewed from top to bottom, can then represent a sharp light-dark transition from the upper lip area to the “intermediate lip area” or to the mouth opening area, corresponding to the first processed image, it being possible to infer the shape of the upper lip on the basis of this transition. This determined shape can be subjected to a first test step, which is to use the shape of the upper lip line to determine whether it is actually an upper lip or some other structure. Does the upper lip have specific geometric properties for it, such as a certain (horizontal) length or one or more specific curvatures, it can be concluded that the actual presence of a mouth area and thus also the presence of a human face. Thus, the method described above, including its advantageous refinements, in which the presence of a face is inferred based on the positional relationship of various defined segments, can be improved by feature extraction, in this case the extraction of geometric properties of the upper lip in the first test step become. This means that after this first section of the feature extraction within the scope of the first test step, the probability can already be increased that the third or fourth defined segment or the analysis area is an image area in which a mouth part of a face is located ,

Fulfill the certain geometric properties (for example with regard to length or curvature) for one Upper lip specific criteria, so the feature extraction pass into a second extraction section or a second test step, in which a lower lip or lower lip line is now extracted should. A lower lip can, for example, by means of a vertical Search algorithm can be recognized or extracted, in which the Gray scale values of the upper lip are determined and then corresponding gray scale values be searched in a section that is vertical located below the area of the upper lip. That way it possible very bright areas, such as the teeth, and very dark areas, such as the throat area, and skip one regarding the Identify the brightness of the upper lip as the lower lip. A another way Extracting the lower lip relies on the fact that at greater Choice of the analysis area, in which a part of the chin area with is included, especially in the third edited image of viewed from the bottom up, a dark area (shadow area) below the lower lip, which is due to the fact that the lower lip normally around the face small amount curved forward is. Thus, according to this procedure based on the lowest and most distinctive brightness transition viewed from dark to light from bottom to top, to the lower lip or lower lip line are closed.

are now identified upper lip and lower lip, so it is now possible to Viseme or extract mouth features by the geometric properties the upper lip and lower lip are examined separately.

It can thus be summarized that a method according to the present invention it first allows you to use a picture in one run to adjust the brightness Segments to help you check their positional relationship the presence and position of a (human) face close.

On Program product for a data processing system that contains code sections with which one of the described processes on the data processing system accomplished can be through suitable implementation of the method in a programming language and translation into Execute code executable by the data processing system. The Sections of code are saved for this. It is under a program product understood the program as a tradable product. It can be in any Form, for example on paper, a computer-readable data carrier or via a Network distributed.

Further Advantages and features of the invention result from the description of an embodiment based on the drawing. It shows:

1 a user with a mobile terminal;

2 a graphical user interface with control of a selection means by moving the head of the user;

3 a representation of a selection means as a mouse pointer;

4 a schematic arrangement of the essential components for analyzing an image of a face according to a preferred embodiment;

5 a schematic arrangement of the method steps according to a primary detection mechanism for locating a face, in particular a mouth area, in an image;

6 an image of a user in the raw state and after two specific processing steps;

7 an image of a user with a marked segment of a mouth area;

8th an enlarged view of the in 5 shown mouth area with marked upper lip and lower lip lines;

9 an image of a user in the Rohzu stood and after a plurality of processing steps.

The Introduction of camera functionalities into mobile devices the third generation allows new ways of interacting with the Devices. Facial detection and tracking can be carried out by the camera recorded movement of the user directly converted into movement data become. This movement data can be evaluated in such a way that the positioning for the selection of buttons displayed on a graphical user interface be used.

As in the 1 and 2 shown, can be done by moving a mobile device 1 and / or the head 2 of a user 3 a selection medium (selection marker) is moved to a desired button. In the illustrated embodiment, this is the button 9 on the graphical user interface 4 , Then either by acoustic input, in particular voice input, "yes", whistle, tongue-in-cheek, by optical input, in particular winking, opening the mouth, by pressing a key or by lingering the selection means on the selected object in the form of the button 9 the function of the mobile terminal associated with this object is triggered.

The means of selection 5 and thus its position on the graphical user interface 4 can also like in 3 represented optically, in particular in the form of a mouse pointer. Analogous to optical mice, the mouse pointer is triggered by the movement of the device and is moved according to the detected movement estimate. The mouse pointer movement is no longer measured and calculated as with an optical mouse via the relative movement to the underlying surface, but rather from the relative movement of the mobile device to the user. An intuitive operation is given by the mouse concept already known to the user.

Thereby can with mobile devices, especially with mobile devices with less than 24 or less than 10 keys, for mobile phones and / or for PDAs, with camera functionality a very comfortable one-hand operation without additional hardware become. When implementing it as a moving mouse pointer, the user can the mouse operating concept known to him on the movement of his Transfer terminal devices.

It can continue stabilization process for smooth movement of the mouse pointer button selection.

The image of the head can be used for optical feedback 2 of the user as in 2 shown, imported, or the procedure can be as in 3 shown can be used without image recording.

For an operation in a dark environment, the mobile device can be equipped with lighting devices such as infrared light emitting diodes.

The mobile device and the proposed method can be done in many ways used or applied. This includes entering phone numbers, with the appropriate size of the ad or a soft scrolling also a selection of letters or special characters, for example for SMS, a menu control of the device and / or an application control.

To the Measuring the relative movement, i.e. to localize the user or his face the terminal, the method described below can be used.

It is initially on 4 referenced, in which a schematic representation of the components or mechanisms contained in the mobile terminal for capturing and analyzing an image are shown. As in 4 It can be seen that a camera K is provided as an optical sensor, which is able to capture a scene (a real environment with certain things and / or people). The camera can be designed, for example, as a CCD camera. The camera K is connected to an optical evaluation section OAA, in which various mechanisms or methods for analyzing the scene captured by the camera can run. The optical evaluation section OAA is able to carry out a first or primary detection mechanism PLD for locating a face or a part of the mouth in the captured image of the scene. Accordingly, the optical evaluation section can carry out an alternative or additional second or secondary detection mechanism SLD for locating a face or a mouth area. After the position of the face has been found according to the primary or secondary detection mechanism, an improved plausibility check can be carried out on the one hand by means of a feature extraction ME as to whether it is actually the found face, in particular its mouth parity. The results obtained from the mechanisms PLD, SLD and ME, namely that a face of a person or a mouth area is present in the image, can be forwarded to a control device SE.

In the case of the preferred embodiment explained below, the in 4 Darge made components in a data processing system in the form of a mobile phone (other realizations of a data processing system are of course also possible, in which the in 4 components can be provided) arranged, the camera K can be permanently integrated in the mobile phone or can be provided as an exchangeable peripheral component. Especially in mostly small mobile devices such as mobile phones, it is important that applications, such as the analysis of a scene described below, have resource-saving algorithms that are minimized in terms of memory requirements, such as computing time and computing power, to the resources of the devices to be used as far as possible, but without requiring additional processor or memory expansion.

in the The following are now those in the optical evaluation section OAA feasible Mechanisms or individual procedures are presented in detail.

Primary detection mechanism

It is now on 5 referenced, in which a schematic representation of the components or individual method steps for realizing the primary detection mechanism are shown. An image captured by the camera K is first fed to an optional image contrast improvement BKV by means of a histogram-based gray level compensation (histogram equalization) per KV of the individual pixels of the captured image. The image thus improved with the contrast is then processed by a horizontally oriented positive gradient filter GF, by means of which brightness-specific horizontal contours, ie distinctive or sharp light-dark transitions or dark-light transitions, can be extracted. The first processed image obtained in this way is fed to a segmentation algorithm SA in which all the pixels with a gray value above a certain threshold value are assigned the gray level value "white", while the other image points are assigned the gray level value "black". In this way, for example, only certain sharp horizontal light-dark transitions can be made visible. In a second processed image obtained in this way, the "white" areas which now remain can be examined with regard to their area, position or geometry. The segments defined in the segmentation algorithm SA can then be subjected to a first examination step US1, by using the positional relationship of correspondingly defined segments or based on the positional relationship and the shape or extent of the segments, including biogeometric criteria for a face, for the presence of a face and the position of the face or the corresponding mouth area can be concluded.

A primary detection mechanism described in this way is now to be described using the 6A to 6C be presented in detail. It is initially in 4A the image of the user captured by the camera is displayed in a vehicle and is in the form of a grayscale image, ie the image consists of individual image points to which a respective specific grayscale value is assigned. The optional image contrast enhancement BKV (cf. 5 ), the captured image is then subjected to processing by a horizontally oriented positive gradient filter in order to determine horizontal structures, ie striking transitions in brightness. Such a (first) processed image is in 6B can be seen in which light-dark transitions viewed from top to bottom, starting from a medium grayscale value, in accordance with the sharpness of the transition to an increased grayscale value (towards "white"), while dark-light-light transitions from top to bottom considered, starting from a medium grayscale value to a lower grayscale value (in the "black" direction). By applying a threshold value formation (the threshold value being selected as a function of the brightness of the captured image, for example on the basis of a histogram representation of the gray level values of the individual image points in the captured image), image elements whose gray level value is above the brightness-dependent threshold value are assigned the value “white”, while the other pixels are assigned the value "black". In this way, a large number can be removed in a (second) edited image for the analysis of the scene of unimportant details. In the segmentation algorithm SA, white areas obtained whose size or shape do not correspond to certain criteria, in particular biogeometric criteria with regard to a face, can now also be removed. After discarding these too small or too large white areas or areas that are not relevant in form and by assigning the value "black" to the corresponding pixels, the actual segments relevant for the later examination are determined. For better analysis, these segments are inscribed in rectangles, as described in 6C can be seen. In the first examination step US1, the defined remaining segments are examined in order for their geometric or positional relationship to one another. The investigation is based on the fact that two horizontally arranged segments that form the eyebrows (see the eyebrow areas ABB in 4A . 4B ) or to show the contours of glasses in a human face, a third segment below these first two segments is sought, which represents a possible segment of the mouth (cf. the mouth area MB in 6A . 6B ) based on the brightness transition from the upper lip area to the intermediate lip area. If a first and a second segment lying horizontally to one another are found, such as the segments S1 and S2, their distance from one another, in this case the distance b, is determined on the basis of the respective center points of the segments. Subsequently, a third segment is sought below the two segments S1 and S2, whose distance, the distance a, from the first two segments or a connecting line thereof is in a first biogeometric relationship with respect to a human face. Such a ratio of a to b can have a value between 1.0: 1 to 1.7: 1, but in particular a value of 1.2: 1 to 1.5: 1. Depending on the use of the scope for the first ratio of a to b, in 6C the segment S3 and S4 as a possible third segment (potential mouth segment) come into question. According to an advantageous embodiment, it is therefore possible in this case to search for a fourth relevant segment which is located immediately below the third segment. Based on the fact that the lower lip of a mouth segment is normally slightly curved forward with respect to the face and thus a shadow area is created below the lower lip, this shadow area can be used as a fourth segment in a kind of plausibility check, which must be located immediately below the third segment. serve. Based on this check, segment S4 can then be identified as a potential mouth segment, segment S5 representing the shadow below the lower lip. Now an area in the captured image of 6A stored, which corresponds to the size and position of the area identified as a potential mouth segment (segment S4). This area, which will later be referred to as the analysis area, is used for the following feature extraction.

It it should be noted that the entire process of the primary detection mechanism, as just described, in a single pass carried out can be, whereby at the end of the run the originally recorded Image can be discarded and only the saved pixel data of the analysis area is required become. As can also be seen, within this single pass the image data of the captured image with very simple algorithms significantly reduced, which is the requirement to minimize memory requirements, Computing time and lead time takes into account.

feature extraction

The following feature extraction is used, for example, to check the plausibility of whether there is really a face with the mouth or lips in the analysis area found. First up 7 referenced in which the camera K in 5 captured scene accordingly 6A is shown, wherein the region AB of the mouth area which is now to be analyzed and which was stored during the implementation of the first detection mechanism is marked. This area is enlarged in 8th shown. According to the in 5 shown with a horizontally oriented gradient filter is now the in 8th Analysis area AB shown enlarged is subjected to processing with a horizontally oriented positive gradient filter. It should be noted that before processing with the gradient filter, an image contrast improvement corresponding to the image contrast improvement BKV in 5 can be carried out. Processing with the horizontally oriented positive gradient filter now creates a third processed image in which (horizontal) light-dark transitions, in particular from the upper lip area to the intermediate lip area, or in other words extracted from the area of the upper lip to the area below the upper lip become. In this way it is possible to obtain individual segments of the upper lip which can be connected to one another and can be smoothed. The upper lip line that has now been found or smoothed can now be evaluated with regard to its geometric properties (length, curvature, etc.), ie whether it corresponds to biogeometric characteristics of an actual upper lip or upper lip line. It is now possible that there are several light-dark transitions in the analysis area AB, which can represent an upper lip line. If a first upper lip line or a combination of several segments identified as the upper lip line, as just described, is assessed negatively, a further light-dark transition or a further section composed of several light-dark transitions, which can be combined to form an upper lip line examines the presence of certain geometric properties according to an actual upper lip. Experiments have shown that a repetition of the examination with a third-most likely light-dark transition area is possible, but mostly not successful, which is why, according to this embodiment, the feature extraction for the secondary detection mechanism explained in more detail below to find the position of a face or a mouth part is passed over.

If a found upper lip line is assessed as positive (corresponding to an actual upper lip or upper lip line), a second test step or extraction step is carried out in which the lower lip or lower lip line is to be identified. Since the exact extraction of the lower lip due to generally weaker contouring of the Lower lip using a filter, such as the gradient filter described above, is difficult in this case, a vertical search algorithm is used, starting from the relatively well recognizable upper lip or upper lip line, which is indicated by the upper white squares OL in 8th is shown, looking for the first color-like or grayscale-like pixel value down along the vertical. This makes it possible to skip bright areas, such as the teeth, as well as dark areas, such as the throat area, and a lower lip area as close as possible, which 8th identified by the line formed by white squares UL. The corresponding upper and lower lip lines OL and UL can then be examined with regard to their geometric properties (length and curvature).

Secondary detection mechanism

How above regarding of feature extraction mentioned has been, is it possible that after performing of the primary Detection mechanism 'and after a plausibility check in a first test step the feature extraction that found by the primary detection mechanism Analysis area no actual Mouth part can have. In this case, the secondary detection mechanism used to find a mouth area analysis area. It should be mentioned that the secondary Detection mechanism already algorithmic when running through the primary detection mechanism considered can be d. H. after capturing the first image of the scene. It is also possible that the second detection mechanism is only in a second Run on a second captured image of the scene can be performed. Will the secondary detection mechanism already performed when going through the primary detection mechanism, so can the segments identified or defined with it only then at Call the secondary Detection mechanism can be evaluated.

The reason for the failure of the primary detection mechanism can be that in the case of strongly contoured glasses, sunglasses or continuous eyebrows, there are no two separate eyebrow segments available for facial geometry assessment, which is why a different detection mechanism must be used in this case. In the second detection mechanism, the same processing steps as that in FIG 5 steps BKV, GF and SA explained. Instead of the first examination step US1, however, a second examination step US2 is used here, in which the facial geometry is used again to find the lips within the image captured by the camera K, a segment being sought here instead of the first and the second segment, that can geometrically correspond to the length of the two eyebrows together.

Starting from an in 9A shown image, which has the face of a user (a person in a vehicle) who has strongly contoured glasses, the face or the mouth area of this face is to be determined here. After performing the in 5 explained processing steps BKV, GF, SA with respect to the image of 9A , an in 9B shown (second) processed image (accordingly 6C ) in which there are only striking light-dark transitions of certain relevant size and shape. First, the horizontal length b 'of each of these existing segments is determined, as shown in segment S6. Now, for each segment, shown here by way of example on segment S6, a segment located below is sought, which has a certain distance a 'from it, the ratio a' to b 'having to have a second biogeometric ratio with respect to a face. This second ratio of a to b is in the range from 1: 1 to 1.7: 1, but in particular between 1.2: 1 to 1.5: 1. Like it in 9B can be seen, a further segment S7 is found for segment S6, by means of which the conditions of the second relationship can be fulfilled. If there are several segments by means of which the second ratio can be fulfilled, it is possible to find further test criteria for this plurality of segments. For example, the length of the segment below the segment representing the eyebrows, which has to lie within predetermined parameters, can be checked. If, for example, the length of the further segment is too large (for example greater than the length b), this cannot be identified as a potential mouth segment.

is now by means of the secondary Detection mechanism for a first one corresponding to the eyebrows Segment found a second segment corresponding to the mouth area this area can be found in terms of position and size in the captured image can be saved as the analysis area and the one above feature extraction shown are supplied.

In summary, it can thus be stated that the methods carried out in the optical evaluation section OAA for analyzing a scene or for locating, tracking and analyzing a mouth area are advantageous, in particular for data processing systems with limited resources. Through the use of facial geometrical segmentation processes, the visual or optical data volume can be reduced very quickly to small segment quantities for further processing be reduced. Built-in plausibility and robustness checks (for feature extraction) also allow a high degree of reliability and prevent incorrect detection.

All Procedure can advantageous for determining the position of the user relative to the mobile terminal used by from the position of the face, in particular the mouth, in the picture the position of the user is calculated. there it’s not even necessary to determine the absolute position, but only the relative in successive images that make up the relative movement towards the user the mobile device can be calculated. The selection medium then becomes corresponding to this relative movement moved on the graphical user interface.

Claims (24)

Mobile device with - a user interface ( 4 ) with a selection means that can be positioned on the user interface ( 5 ), - means for measuring a relative movement of the mobile terminal ( 1 ) towards a user ( 3 ), - means for positioning the selection means ( 5 ) on the user interface ( 4 ) depending on the measured relative movement. Mobile terminal according to claim 1, characterized in that the means for measuring the relative movement means for recognizing the position of the face ( 2 ) of the user ( 3 ) having. Mobile terminal according to claim 1 or 2, characterized in that the means for measuring the relative movement means for recognizing the position of the lips and / or an eye of the user ( 3 ) exhibit. Mobile terminal according to one of the preceding claims, characterized in that the mobile terminal for measuring the relative movement has an optical sensor (K) for recording an image of the user ( 3 ) having. Mobile terminal according to claim 4, characterized in that on the user interface ( 4 ) to control the relative movement as feedback the currently recorded image of the user ( 3 ) can be displayed. Mobile terminal according to one of claims 4 or 5, characterized in that the mobile terminal ( 1 ) Has lighting means to the user ( 3 ) to illuminate. Mobile terminal according to one of claims 4 to 6, characterized in that the mobile terminal ( 1 ) has an evaluation device (OAA, SE) through which the image of the user ( 3 ) can be analyzed with the following steps: - Definition of segments (S1 - S7) in the image that have brightness-specific features; - Checking a positional relationship (a, b) of the defined segments to each other; - Deriving a presence and / or a position of the user, in particular a face of the user, in the image if a selection of defined segments (S1, S2, S4; S6, S7) has a specific positional relationship. Mobile terminal according to one of the preceding claims, characterized in that the selection means ( 5 ) on the user interface ( 4 ) can be displayed. Mobile terminal according to one of the preceding claims, characterized in that the mobile terminal ( 1 ) Has stabilizing means to ensure a smooth movement of the selection means ( 5 ) to ensure. Method for operating a mobile device, in which - a relative movement of the mobile device ( 1 ) towards his user ( 3 ) is measured, - a selection agent ( 5 ) on a user interface ( 4 ) of the mobile device ( 1 ) is positioned depending on the measured relative movement. The method of claim 10, wherein an image of the user ( 3 ) is recorded. The method of claim 11, wherein the image of the user ( 3 ) is analyzed with the following steps: - definition of segments (S1 - S7) in the image which have brightness-specific features; - Checking a positional relationship (a, b) of the defined segments to each other; - Deriving a presence and / or a position of the user, in particular a face of the user, in the image if a selection of defined segments (S1, S2, S4; S6, S7) has a specific positional relationship. The method of claim 12, wherein the sensed Image is composed of individual pixels, which one Grayscale values and / or color values are assigned. The method of claim 12 or 13, wherein the brightness-specific features sharp brightness transitions and / or have sharp color transitions. The method of claim 14, wherein the The step of defining segments in the captured image includes processing (GF) the image using a gradient filter, in particular a positive horizontal gradient filter, in order to generate a first processed image in which a brightness transition and / or a color transition occurs in the captured image Corresponding pixels are provided in the first processed image with a grayscale value that depends on the sharpness of the brightness transition in the captured image. The method of claim 15, wherein the step binarization processing of specifying segments in the captured image (SA) of the first processed image includes a second one generate processed image at the corresponding image points, whose grayscale value exceeds a predetermined threshold value, assigned a "white" value will while the remaining pixels assigned a "black" value becomes. The method of claim 16, wherein the step inscribing segments in the captured image (SA) an area of the second processed image with contiguous Pixels to which a "white" value is assigned is in a geometric shape, in particular a rectangle, as a fixed one Segment includes. The method of claim 16 or 17, wherein certain Areas of the second processed image with contiguous "white" pixels Due to certain parameters regarding the size and / or shape of the areas or the geometrical form that inscribes this is rejected. Method according to one of claims 12 to 18, in which in the Step of reviewing the positional relationship in a first examination step of each of the defined segments it is examined whether a second one (S2) has been defined for this Segment exists that is essentially horizontal to the one under investigation fixed segment (S1), being in the presence of the second defined segment after a third defined segment (S4) is sought, which is below the examined and the second Segments, and for that is, a distance (b) from the examined to the second fixed segment, and a distance (a) of a link between the examined and the second defined segment to the third fixed segment has a first predetermined ratio. Method according to one of claims 12 to 19, in which in the Step of reviewing the positional relationship in a second examination step of each of the defined segments it is examined whether a fourth fixed one for this Segment (S7) exists that is defined below the one examined Segment (S6), and for this is true that the horizontal length (b ') of the examined specified segment and the distance (a ') of the fourth specified segment a second predetermined segment for the examined segment relationship exhibit. Method according to claim 19 or 20, in which the first / or the second predetermined ratio is one Has value from 1: 1 to 1: 1.7, in particular from 1: 1.2 to 1: 1.5. The method of any one of claims 19 to 21, further comprising Step of extracting biometric features with respect to a Mouth part from an analysis area around the found third (S4) or fourth (S7) defined segment includes. The method of claim 22, wherein in the detected Image in an analysis area (AB) located at the position of the third or fourth specified segment, and that the each segment comprises, light-dark and / or dark-light-brightness transitions determined to the presence of an upper lip and lower lip one Derived mouth part. Program product that, if it is on a data processing system loaded and executed on it is, a method according to any one of claims 10 to 23 or an apparatus according to one of the claims 1 to 9 takes effect.