KR100492765B1 - Apparatus and method for controlling an airbag in a vehicle by optimized human presence detection - Google Patents

Abstract

The human presence detection system 50 uses a frame discrimination technique to subtract background interference from the image produced by the system. The system 50 includes an infrared source 28 for generating an infrared radiation beam, and an infrared detector 72 for receiving infrared radiation reflected from an object in the path of the beam. Facial recognition software is used to determine the presence of the person 16 from the reflected radiation. The infrared source 28 is pulsed on and off, and the detector 72 is blocked at the same time with the pulse so that the image frame is generated at different times, where one frame includes reflected radiation and background radiation, and the other frame Include only background radiation. The frame is subtracted to separate the background radiation. In one embodiment, detector 72 includes a pixel array of photodiodes 90 and first and second capacitor storage sites 94 and 98 for storing image frames on a single CMOS chip 70. do. Storage sites 94 and 98 are subtracted in summing device 102 that is also present on CMOS chip 70. In alternative embodiments, frames are stored and subtracted at off-chip sites.

Description

Apparatus and method for controlling vehicle airbag by optimized human presence detection system {APPARATUS AND METHOD FOR CONTROLLING AN AIRBAG IN A VEHICLE BY OPTIMIZED HUMAN PRESENCE DETECTION}

FIELD OF THE INVENTION The present invention relates generally to frame differentiation techniques for removing background interference in human presence detection systems, and more particularly to background interference in human presence detection systems used in vehicles for proper airbag deployment. A frame discrimination technique for removing.

The National Highway Traffic Safety Administration requires passenger airbags for all passenger cars manufactured after 1998. Airbags can save thousands of lives, but safer airbag deployment systems can save even more lives. Proposals requiring improved airbags that are safer and more effective have been submitted by the Federal Road Traffic Safety Administration. Accordingly, there is a need in the art for an improved airbag deployment system that determines airbag deployment as it detects, identifies, and tracks people in a passenger seat of a vehicle.

Future airbag deployment systems will be able to identify and track people in the passenger seat of the vehicle. Based on this information, the system inflates the airbag in the event of a crash, depending on whether the person is seated in the passenger seat, the person's body shape and the person's position relative to the airbag deployment door, and at a low speed. It will provide a signal as to whether to inflate the airbag or inflate it at normal high speed. For example, if a person is detected in the passenger seat and is at a somewhat narrow distance (eg, within 3 inches) from the airbag door, the airbag does not inflate during a crash. If a person is detected in the passenger seat and close to the airbag door (eg between 3 inches and 8 inches), the airbag inflates at a lower speed during a crash. If a person is detected in the passenger seat and far enough away from the airbag door (eg greater than 8 inches), the airbag will normally inflate at high speed in the event of a crash.

Current occupant detection systems generally include ultrasonic sensors, mass sensors, infrared sensors and / or electromagnetic sensors to detect occupants for airbag deployment. The ultrasonic sensing system senses the movement of an object within the passenger seat area and determines whether the object moves closer or farther from the position of the sensor. However, the ultrasonic sensor does not identify the property of the object, and therefore cannot know whether the object is a human or other object such as a grocery bag. Similar to ultrasonic sensors, microwave sensors used in active doppler radar systems can track objects, but cannot identify human presence. The mass sensor identifies objects in the passenger seat based on the pressure applied but does not consider the occupant's position with respect to the airbag. Passive IR sensors capture the thermal image of the passenger seat, but these systems are very expensive. Current active IR sensors detect the passenger's relative position with respect to the airbag, but cannot identify the presence of a person. Electromagnetic systems include LC resonant circuits in which body capacitance is used to detect the presence and identify an object, but such a system cannot track the object.

U.S. Patent No. 5,835,613, issued to Breed et al. On November 10, 1998, discloses an in-vehicle surveillance claiming to identify, locate, and monitor a person in the occupant's area of the vehicle. Start the system. The monitoring system uses an infrared emitter that illuminates the interior of the vehicle and a charge couple device (CCD) array that detects the radiation wave. Output from the CCD array is analyzed by a computing device, which uses pattern recognition algorithms to classify, identify or locate content or objects in the passenger seat. The pattern recognition system for determining a vehicle occupant disclosed in the above-mentioned US patent (No. 5,835,613) uses complicated software that needs to know the shape of a person in all kinds of lighting situations under various conditions. Moreover, using pattern recognition in this way is limited in the ability to track a person because he moves in the passenger seat. Moreover, the ability to identify and track humans by general pattern recognition is unsuspectingly unreliable. The pattern recognition cannot identify the person itself, only the shape of the object.

There is a need for improved airbag deployment sensors and systems that can effectively identify and track people in the passenger seat of a vehicle. It is therefore an object of the present invention to provide such a sensing system for airbag deployment of a vehicle occupant.

In accordance with the teachings of the present invention, a human presence detection system is disclosed that uses frame discrimination techniques to subtract background interference from an image produced by the system. The system includes an infrared source for generating an infrared radiation beam, and an infrared detector for receiving infrared radiation waves reflected from an object in the path of the beam. Facial recognition software is used to determine the presence of a person from reflected radiation. The infrared source is pulsed and the detector is blocked synchronously with the pulses so that the image frames are generated at different times, where one frame contains reflected radiation and background radiation, and the other frame is just background radiation. Includes only. The frame is subtracted to separate the background radiation wave.

In one embodiment, the detector includes a pixel array of photodiodes and first and second capacitor storage sites (also referred to as 'memory') for storing image frames on a single CMOS chip. The storage site is subtracted from the summing device that also exists on the CMOS chip. In an alternative embodiment, the frame is stored at the off-chip site and subtracted.

The following discussion of a preferred embodiment of an airbag deployment system using human face region recognition is merely exemplary in nature, and therefore does not limit the present invention or its application or mode of use at all.

In accordance with one embodiment of the present invention, a technique for detecting, identifying, and tracking a person in the passenger seat of a vehicle is disclosed. As described above, depending on whether the person is in the passenger seat of the vehicle, whether the person is adjacent to the airbag door, and the person's body type, the airbag at the occupant's side of the vehicle should not be deployed at low speed or in the event of a crash. It is sometimes desirable not to. According to the invention, identification and tracking of a person is determined by facial recognition software, in particular software that recognizes and tracks the eye and other facial parts of the person. Tracking can occur that way at various head angles and postures. The software algorithm uses feature separation of the calibrated face and appearance to estimate the range. Tracking of localized areas on the human face allows more image frames per second to be obtained, allowing for more frequent tracking of people. Tracking the face area more frequently prevents obtaining a blurred image acquisition, because the image is updated more often.

Various software is known in the art to process data from video data patterns received from an object to be analyzed and to determine whether the object has a face. For example, such software includes Visionics Face-It software, which is well known to those skilled in the art. Although the present invention is not limited to any particular face region mapping function, it is suitable for the purposes described herein for recognizing face regions, whether in two or three dimensions, and in addition to defining a range of functions. It is emphasized that it can include any known algorithm that can be used. Moreover, in accordance with the present invention, delimiting algorithms is used with known face recognition software.

As will be discussed in detail later, the present invention uses infrared radiation waves reflected from an object on the occupant's side of the occupant's area of the vehicle and received by an electronic video camera. The video camera generates electronic signals and images used by facial recognition software to determine the identification and tracking of the presence of a person.

1 is a side view, partly cut away, of the zone 10 on the occupant side of the vehicle 12. In this figure, the person 16 is shown as being in the passenger seat 18, where the person 16 moves forward in the event of a crash. The airbag pillow 20 is shown to be deployed through the airbag door 24 housed in the instrument panel 22 in the event of a crash.

The video camera and IR LED illuminator unit 26 are mounted on the instrument panel 22 in a position suitable for the purposes described herein. 2 is a schematic diagram of a unit 26 detached from the vehicle 12. Unit 26 includes a cluster 28 of IR LEDs 30. A plurality of LEDs 30 are provided to generate the brightness needed for daytime operation. The cluster 28 emits an IR radiation beam towards the person 16, which is reflected from the person and returns back to the unit 26. Video camera 34 is provided to unit 26 to receive the radiation waves reflected from person 16. The video camera 34 is used as a non-limiting example in that any detector that detects infrared radiation waves suitable for the purposes described herein can be used.

Filter 36 is provided on camera 34 to filter out radiation waves that are not within the desired infrared range. Filter 36 may be any filter suitable for the purposes described herein, such as a TiO ₂ filter or a polarizing filter. The IR image is delivered to the detector, but the layer and thickness of the filter can be selected to reflect the image of visible light coming from the detector. Polarizing filters can be used to reduce visible light to the detector using electro-optical polarization that passes IR wavelengths but strongly attenuates non-IR wavelengths. 3 shows a luminance curve for sunlight, where filter 36 passes infrared radiation in a window of 40 nm bandwidth. The filter 36 provides a kind of protection against sunlight that can affect the operation of the airbag deployment system and the perception of the face of the person 16.

In this embodiment of the invention, one camera is used to capture and monitor the range of the person 16. The software used to perform this function uses two separate locations on the occupant's face to provide scoping. In a preferred embodiment, the human eye is detected to provide triangulation for the specification of the operating range. However, as will be appreciated by those skilled in the art, other facial portions of the person 16, such as the ear of a person, may also be used. In addition, software algorithms allow the head size of a person to be determined, so that once the person is captured, there is no need to examine both eyes to track the person. Furthermore, the software may be used to examine other parts of the body of the person, such as the torso of the person, in connection with the detection of facial areas or head size.

Because human face parts are unique, databases can be used to store specific information about a person, such as eye-to-eye separation, so that the software can specifically identify that person. Can be. In one example, this is important, as required by government mandates, to allow the system to identify children and 20% of women and prevent these people from inflating airbags. In addition, being able to identify a person in particular improves the accuracy of the operating range of the system, since the system recognizes the separation interval of the eye of the person or other specific facial areas.

Acquisition and tracking software needs to be calibrated for specific seat positions, airbag positions, vehicle structures, and the like. 4 is a diagram of camera orientation and position parameters for the center and normal of the airbag door 24. After the camera and the Visionics Face-It software have been calibrated, the three parameters that must be estimated in the vehicle to determine the operating range according to the invention include two position offsets and one angular offset. The two position offsets are the lateral offsets of the camera 34 relative to the center of the airbag, measured laterally and perpendicularly to the normal vector at the center of the airbag door 24. ) And the front or rear offset of the camera 34 relative to the center of the airbag door 24, measured along an axis parallel to the normal at the center of the airbag door 24 ( )to be. The angle parameter is the azimuth offset between the optical axis of the camera 34 and the normal vector emerging from the center of the airbag 20. )to be. Only two of three parameters, namely And Is used in the modified monocular distance equation.

The calibration procedure described below , And Can be used to determine. Measuring a plurality of calibration points, interpolating linearly between the plurality of calibration points, or measuring a few calibration points, and nonlinear interpolation between the few calibration points Each one is trade off. In theory, calibrating to multiple points does not require the estimation model described herein. The cost of making many measurements necessary for the correction of brute force, and the possibility that the operation of the camera 34 or the software may be out of range and neglected to the correction of the oblique correction, should be considered before using such an approach. do. The approach used herein uses a few calibration tests to form a model for interpolation.

From Fig. 4, the following equation can be written to correlate the measurements made in the camera lens reference frame xyz with the measurements made in the airbag door reference frame xyz. Only the position (x> 0) of the face in front of the airbag door is considered in the following equation. Ρ, θ (pitch or elevation) defined in the camera's cylindrical coordinate system, and For yaw or azimuth, the equation for displacement (x and y) in the airbag coordinate system is:

this is Is considered to be fixed during calibration. Rearranging Equations 1 and 2 yields the following equations:

Fixed Taking a reading of the eye coordinates at and taking the slope of the tangent with respect to y, yields:

By subtracting the result of x and the right side of Equation 4, Can be determined. By knowing Equation 3 and the measured data Can be used to determine. Then, , , By using Equation 3 and its data Can be determined. By using Equation 3 and Equation 5 below,

The modified monocular equation is according to equation (6) to define the object parameter (x), or the distance between the airbag and the eye.

The calibration table consists of a lookup table of the coordinates of the eyeballs determined by the SDK as pixel values linked to the associated ray slope when viewed in the camera's coordinate system. Equation 6 can be simplified to the slope of the ray (single angle tangent) term so that the calibration table can be used directly to determine the distance between the eyeball and the airbag from monocular operation. Equation 7 below uses the trigonometric identity of tangent,

Applying this identity to Equation 6 yields a modified monocular equation that can be used to obtain the tangent / tilt directly from the calibration lookup table. This equation is given by Equation 8 below.

5 is a block diagram of an image system 50 of the present invention that includes a digital signal processor (DSP) 52. The DSP 52 includes face recognition software and an operating range designation function that perform analysis on the image generated by the camera 34. Clock generator 60 provides timing for various digital devices in system 50, and power management system 62 provides power. The DSP 52 is connected to a CMOS chip 70 that includes a pixel array 72 representing an IR detector, such as a camera 34. In this example, pixel array 72 includes 256 x 256 pixels to provide the desired level of resolution. CMOS chip 70 includes state matching logic circuit 74, clock / timing circuit 76, analog conditioning circuit 78, register / buffer 80, on-chip for timing and control purposes. Also included are various devices including (on-chip) programmable logic circuits 82, and the like. Moreover, analog-to-digital converter 84 is also provided to convert analog signals from pixel array 72 into typical digital signals. On-chip SRAM memory 86 is shown for storage purposes, but may also be off-chip. The operation of these devices in the systems described herein is apparent to those skilled in the art.

In one embodiment, the infrared LED 30 is continuously on to provide reflected radiation waves received by the camera 34. However, some sort of filtering or signal processing is generally done to correct the problems caused by the direct sunlight coming on the camera 34, which is done through the filter 36. In particular, the system 50 needs to be able to distinguish between the shadows caused by sunlight and the actual edges for the appearance of the occupant 16. According to the present invention, a frame differentiation technique is used, which pulses the LED 30 synchronously for a predetermined time period and for a predetermined number of video data frames, and then the same number. Pulse off for a predetermined time period over the video data frame The frames of data are then subtracted from each other so that frames without IR illumination can be subtracted from frames with IR illumination and the background can be removed. The detector is electrically shuttered synchronously with the pulse to provide exposure control. The frame differentiation technique described herein is used in connection with infrared pulse operation to achieve the desired result. In other words, the frame discrimination operation is synchronized with the infrared pulse.

The concept of frame differentiation is a technique for storing images using only natural light and also images using both natural and added infrared light at a time-aperture, on a pixel-level basis. This frame discrimination subtracts these images to mitigate the effects of strong visible light illumination. This set-up includes a neutral density filter that sets the IR illumination in addition to the worst case background to maximize the analog / digital converter input. Face recognition requires that the worst case analog / digital range for the distinct image is between 5 and 6 bits. Visible light fits within the remaining range allowed by the analog-to-digital converter. Image discrimination is performed in the analog region in which two pixel level capacitors are charged at each illumination level, or in the digital region where RAM memory of the digitized pixel output is taken in each illumination. Frame discrimination techniques work by subtracting the lighting effect of visible light illumination and increasing its image contrast. Frame differentiation can be performed in acquisition mode / high bandwidth generation mode, or in narrow bandwidth tracking mode using pulsed LED illumination. The number of electrons from the pulsed IR light source should be 10 times greater than the photon noise of the ambient light. Here, the noise of the ambient light is the square root of the value that is twice the number of electrons in the intensity of sunlight ( This is because two image frames are acquired for every one IR image received.

6 is a diagram of how a frame discrimination technique is performed in the camera 34, in accordance with an embodiment of the present invention. 7 is a signal timing line showing the operation of the frame discrimination technique of the present invention. Pixel array 90 in camera 34 receives radiation waves from the scene in a predetermined time period 10 ms during the on period of the IR from LED 30. At this time, the pixel array 90 receives ambient light and infrared light. The charge stored by each pixel, ie photodiode 92, in array 90 is then transferred to charge storage site 94, which is composed of a plurality of capacitors with one capacitor 96 for each pixel 92. do. After about 10 ms, when the pulse from the cluster 28 is off, the pixel array 90 detects only ambient light for the same time period. The charge received at the pixel array 90 in this time period is stored at the capacitor storage site 98 with the capacitor 100. An electronic shutter is used in the detector to open and close at an appropriate time synchronously with a pulse of IR radiation (also called a 'pulse') for the operations described herein.

The two storage sites 94 and 98 are summed in a summation amplifier 102. The difference between the two storage sites 94 and 98 then represents a frame of data that has been digitized by the analog-to-digital converter 104 and ambient light has been removed. The data read takes about 10 ms and then, in the next time period, the next pulse from cluster 28 occurs. The complete frame discrimination process can be performed on a single chip of CMOS where the pixel array 90 and storage sites 94 and 98 are present together. In alternative embodiments, the frame discrimination technique is performed at different time periods at off-chip sites, where storage sites 94 and 98 are RAM.

The frame discrimination technique of the present invention can be described in the following manner. Variables are L when ambient light {I (x, y)}, direct ambient light {T (x, y)}, scene reflectance {R (x, y)}, and on and off. Is limited to a source that is modulated to zero when The response of camera 34 is proportional to the product of reflectance and illumination.

S (x, y, OFF) = k ^* {I (x, y) ^* R (x, y)}

S ^* (x, y, ON) = k ^* {(L + I (x, y)) ^* R (x, y)} + T (x, y)

D (x, y) = S (x, y, ON) -S (x, y, OFF) = KL ^* R (x, y)

The scene of this difference has much less dynamic range than a simple image {S (x, y, OFF)}. The same benefit can be derived by reading the frame under the LED, then reading the frame without the LED and subtracting the frame outside of the camera 34. The disadvantage is the increased dynamic range required to avoid saturation.

L must be much larger than the photon noise for I. Thus, I is made as little as possible by using a narrowband filter that is aligned to L at frequency. The raw sampling rate should be twice the requirement set by object tracking, because two frames are distinguished in order to obtain one frame for feeding facial recognition software. The LED 30 should be much faster. The IR radiation wave source must be modulated such that all emissions occur for the time when all the detectors are activated. If the integration time of all the pixels in the detector is not aligned, the possible time that the source can be turned on is reduced by the worst case misalignment.

The foregoing discussion discloses and describes only exemplary embodiments of the invention. For example, pulsed laser diodes can be used in place of LEDs. Those skilled in the art will readily recognize from such discussion and from the accompanying drawings, and claims, and such various changes, modifications, and changes, may be made without departing from the spirit and scope of the invention as defined in the following claims.

As described above, the present invention is generally used in a frame differentiation technique for eliminating background interference in a human presence detection system.

1 is a cut away side view of a person in a passenger seat of a vehicle relating to an image sensing system for airbag deployment in accordance with one embodiment of the present invention;

2 is a schematic diagram of a video camera and LED illuminator used in the airbag deployment system of the present invention.

3 is a graph of wavelength on the horizontal axis and luminous energy on the vertical axis, showing a luminance curve for sunlight.

4 shows the camera orientation and position parameters with respect to the center and normal of the airbag door.

5 is a block diagram of an airbag deployment system of the present invention.

6 is a schematic diagram illustrating a frame discrimination technique used in the airbag deployment system of the present invention.

7 is a time diagram for the frame discrimination technique of the present invention.

<Explanation of symbols for the main parts of the drawings>

16: person 28: infrared source

50: human presence detection system 72: infrared detector (pixel array)

70: single CMOS chip 90: photodiode

94, 98: storage site 102: summing device

Claims (42)

delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete In a vehicular device, the device comprises: As an imaging system, An infrared source for generating an infrared beam, comprising: an infrared source pulsed to be on and off at predetermined associated time intervals; An infrared detector responsive to an infrared radiation wave reflected from an object in the path of the infrared beam, the infrared detector generating an image signal; And A processor responsive to the image signal from the detector, the processor generating one first image in one first time period when the infrared source is on and one second when the infrared source is off Generating a second image in a time period, the processor subtracting the second image from the first image to produce a filtered image subtracted from ambient light from the image; A processor using a pattern recognition algorithm that uses the filtered image to recognize a pattern indicating the presence of a person. An imaging system comprising; And Operable Air Bag Wherein the processor controls the operation of the airbag in response to the recognized pattern, Wherein the number of electrons emitted from the infrared source of the imaging system is at least ten times the photon noise contained in the ambient light, Vehicle device. 23. The system of claim 22, wherein the imaging system comprises: a first memory for storing the image signal of the first image, a second memory for storing the image signal of the second image, and the first memory; And a subtraction device for subtracting the image signal stored in the second memory from an image signal. delete 23. The apparatus of claim 22, wherein the imaging system comprises a filter having a narrow passband to reduce the photon noise of the ambient light detected by the infrared detector. 23. The vehicle apparatus of claim 22, wherein the infrared source of the imaging system is a cluster of infrared light emitting diodes that collectively emit the infrared beam. 23. The apparatus of claim 22, wherein the imaging system includes means for determining a distance between the location where the air bag is stored and the person. In a vehicular system, the vehicular system comprises: As an imaging system, An infrared source for generating an infrared beam, comprising: an infrared source pulsed to be turned on and off at predetermined associated time intervals; An infrared detector responsive to an infrared radiation wave reflected from an object in the path of the infrared beam, the infrared detector generating an image signal; And A processor responsive to the image signal from the detector, the processor generating one first image in one first time period when the infrared source is on and one second when the infrared source is off Create a second image in a time period, the processor subtracting the second image from the first image to produce a filtered image with ambient light subtracted from the image, and the processor subtracting the second image from the human face A processor using a face recognition algorithm that uses the filtered image to recognize presence An imaging system comprising; And Operable Air Bag Wherein the processor controls the operation of the airbag in response to recognition of the human face, Wherein the number of electrons emitted from the infrared source of the imaging system is at least ten times the photon noise contained in the ambient light, Automotive systems. delete 29. The vehicle system of claim 28, wherein the imaging system includes a filter having a narrow passband to reduce the photon noise of the ambient light detected by the infrared detector. 29. The vehicle system of claim 28, wherein the infrared source is a cluster of infrared light emitting diodes that collectively emit the infrared beam. 29. The apparatus of claim 28, wherein said imaging system includes means for determining a distance between a location where said airbag is stored and said face of said person when recognized, said distance determining means determining said distance when determining said distance. A vehicle system using two points on the phase. 33. The system of claim 32, wherein the two points on the face of the person comprise a position of a first eye and a position of a second eye. A method for controlling inflatable airbag operation of a vehicle, the airbag operation control method comprising: Generating an infrared radiation beam; Pulsing the infrared radiation beam to be on and off at predetermined associated time intervals; Detecting infrared radiation reflected from an object in the path of the radiation beam; Generating and storing one first image from the reflected infrared radiation in one first time period when the beam is on; Generating and storing one second image in one second time period when the beam is off; Subtracting the second image from the first image to remove ambient light and produce a filtered image; Using a pattern recognition algorithm using the filtered image to recognize a pattern indicative of the presence of a person in a vehicle; And Controlling the operation of the air bag in response to the recognized pattern Including, Wherein the generating an infrared radiation beam comprises emitting an electron number that is at least ten times the amount of photon noise of the ambient light, How to control the airbag operation of the vehicle. delete 35. The method of claim 34, further comprising filtering the ambient light prior to detecting the reflected infrared radiation to reduce photon noise of the ambient light in the detected infrared radiation. . 35. The method of claim 34, further comprising determining a distance between the stored position of the airbag and the recognized pattern. A method for controlling inflatable airbag operation of a vehicle, the airbag operation control method comprising: Generating an infrared radiation beam; Pulsing the infrared radiation beam to be on and off at predetermined associated time intervals; Detecting infrared radiation reflected from an object in the path of the radiation beam; Generating and storing one first image from the reflected infrared radiation in one first time period when the beam is on; Generating and storing one second image in one second time period when the beam is off; Subtracting the second image from the first image to remove ambient light and produce a filtered image; Using a face recognition algorithm that uses the filtered image to recognize the presence of a face of a person in a vehicle; And Controlling the operation of the air bag in response to the recognized face Including, Wherein the generating an infrared radiation beam comprises emitting an electron number that is at least ten times the amount of photon noise of the ambient light, How to control the airbag operation of the vehicle. delete 39. The method of claim 38, further comprising filtering the ambient light before detecting the reflected infrared radiation to reduce photon noise of the ambient light in the detected infrared radiation. . 39. The method of claim 38, further comprising determining a distance between the stored position of the airbag and the recognized face dead using two points on the recognized face. 42. The method of claim 41, wherein determining the distance between the stored position of the airbag and the recognized face using two points on the recognized face comprises: first one of the two eyes on the recognized face; using the position of the eye and using the position of the second eye as one of two points on the recognized face.