DE102007058943A1 - Multi-spectral video device for air-based surveillance and real time aerial photograph monitoring of uncrewed aircraft, has electronic mosaic/fusion device designed such that fusion device receives image signals of video scanner cameras - Google Patents

Abstract

The device (1) has two video scanner cameras (2, 3) with surface sensors and an electronic swiveling control device (4). The control device swivels lines of vision (5, 6) of cameras independently in the coordinated step-by-step operation such that images of recording sequences of each camera overlap partial edge. An electronic mosaic/fusion device (12) is designed in such a manner that fusion device receives image signals (13, 14) of cameras whose individual images are combined to a coherent image, and geometrically overlaps at a large extent congruently to a multi-spectral resulting image.

Description

The The invention relates to a multispectral video device for airborne observation, especially for the real-time aerial image surveillance from airplanes by radio transmission the Images to a bottom evaluation station.

Of the Term "video camera" refers to electronic analog or digital cameras that are capable of taking pictures in such a way faster episode record that a viewer from the presentation this sequence of movements in the scene. The pictures of such video sequences will be also called "video frames". One Practical guideline for the minimum video frame rate for one immediately visual traceable movement is about 5 Hz.

Video scanners cameras be for a long time in the airborne Observation used, but usually in the form of line scan cameras, so-called line scanner cameras. These can be used for periodic view direction swing perpendicular to the line progression and thus provided for periodic construction of a planar image with rotating mirrors be. Several simultaneous recording lines of different enable spectral sensitivities multispectral result images. Examples of such applications are for example, a daylight color camera with the spectral channels Red Green and blue, a daylight / thermal imaging camera optimized for the respective lighting situation for the Day and night use, a daylight camera with additional thermal information, d. H. Information about Object temperatures, daylight / near infrared cameras for detection of vegetation conditions based on the characteristic spectral profile of chlorophyll in the near infrared range or so-called hyperspectral cameras Image decomposition into several narrowband spectral bands.

The Panning the viewing directions of the aforementioned cameras is usually done perpendicular to the direction of flight of the camera-carrying aircraft, so that periodically following each other from the rotation mirror movement, transverse to the direction of flight oriented image strips revealed. The temporal Sequence of these image strips results with the help of trajectory movement of the aircraft or of the aircraft a coherent one Tapestry of the overflown and observed area along the trajectory track. The advantage of this Technology lies in its basic simplicity. The essential Disadvantage is that the attitude movements of the camera bearing Aircraft with high accuracy and must be logged in the following computer-aided Image composition from the recording lines are to be considered geometric fractures to avoid in the pictures obtained.

According to newer Technology come for airborne Image strip images instead of the aforementioned line scan area sensor cameras, too Focal plane array cameras called, used. The latter can Viewing swung be formed, the panning steadily or step by step ("Step and starlings ") can. But it is also a fix mounted row arrangement of several divergent looking cameras in fan arrangement possible, whose down looking fields overlapping each other and overlapping each other in the subsequent Assembling a picture strip revealed. The sequence of these image strips in the direction of flight leaves again during the Flight a tapestry of the overflown Area, the so-called footprint. The advantage of this Area sensor technique is that the attitude movements of the camera-carrying aircraft not with the for the aforementioned ones Line scan cameras required precision must be known. This is due to the fact that the in larger picture content area and not just in parts of a picture line overlapping video frames a geometric orientation of adjacent frames alone their picture content he possible.

Especially because of the size of highly sensitive Infrared video cameras, it is not possible or not technically or economically useful, such area sensor cameras as a fixed line arrangement of several divergent looking Cameras in fan arrangement to combine a multispectral camera for flight use, although multispectral result images are especially useful for observation use interesting.

combinations from daylight and thermal imaging cameras are mostly in shape in the form of so-called gimbal swivel systems from balls to cylinders, so-called camera turrets, although for the point observation However, these relatively large and heavy systems are known in Terms of their orientation for the use as a scanner too slow and therefore unsuitable.

Of the Invention is based on the object, a multispectral video device the aforementioned Way of creating, by means of a seamless, multispectral Tapestry in high quality form, d. H. in as possible low motion blur, allows is.

These The object is achieved by a multispectral video device with the features of the claim 1 solved.

Advantageous developments are counter stood the dependent claims.

According to the invention multispectral video device at least two video scanner cameras, which with area sensors stocked are and have different spectral sensitivities; Furthermore, an electronic swivel control device, by means of the viewing directions of the video cameras on the one hand independently, on the other hand in coordinated stepping operation are pivotable so that the pictures of picture sequences of each video camera at least partially Cover edge; and also an electronic mosaic / fusion device is formed such that it receives the image signals of the video cameras receives their single images too coherent Put pictures together and geometrically largely congruent with a multispectral one Overlay result image.

The respective frames of the video cameras, as mentioned above, also Called video frames. The term "mosaicking" is the merging of individual images to coherent Understood pictures. The term "fusion" includes the geometric largely congruent superpositions the coherent pictures the video cameras to a multispectral result image. This here used term "multispectral" includes the term "at least bispectral", z. B. visible Spectral range and additionally one Infrared spectral band.

The aforementioned, inventive multispectral Video device allows on the one hand the recording of a gapless Tapestry, on the other hand, high quality images of this Tapestry, as the video cameras are independent, on the other hand, are pivotable in coordinated step operation. It is possible, too, to produce such a multispectral video device in a small size, for themselves the use in usual Air observation aircraft including light, unmanned aerial vehicles Aircraft, suitable.

advantageously, and relevant to the application are the spectral sensitivities of the Video cameras in the visible or infrared range. In order to contains that overlaid Result image more information than the individual spectral image contributions and in particular such information as it is for desired the observation use are.

According to one Another development of the invention, the swivel control device preferably electromotive one or two axes movable mirrors for pivoting the viewing directions of the video cameras. Let that way due to the low moments of inertia The moving parts in start-stop operation, also called "Steg and Stare" operation, achieve higher cycle frequencies.

from that results in a reduction of motion blur by the image acquisition in the mirror standstill and a shorter one Recording cycle for the single video strip of each video camera than this is even more acceptable Motion blur through a steady gaze tilting would be achievable. from that it also emerges that the inevitable perspective differences between successive image strips due to the fast stripe sequence become minimal, with consequently simplified ratios for the subsequent electronic Mosaic and fusion.

According to one particularly preferred embodiment The invention relates to video cameras and swivel control devices so controllable that the image captures of the video cameras in a periodic Clock are coordinated and the pivoting of the gaze of the Video cameras in start / stop mode synchronized with this clock is that the line of sight of one of the at least two video cameras relative to the viewing direction of the at least one other Video camera is at a standstill and takes a picture while itself the viewing direction of at least one other video camera related on the viewing direction of the one of the at least two video cameras in the movement. In the case of only two video cameras This means that in each case a video camera is at a standstill and takes a picture while the other video camera her gaze changes, that is, is in motion. With the aid of this preferred embodiment of the invention will achieved that for the two video cameras alternately taken pictures without motion blur can be and at the highest possible Start / stop frequency and thus the highest possible Frequency of the stripe cycle. It follows that the means of the can be achieved according to the invention video device Result images are of high quality. This is possible in a conventional Video device with, for example, a total of two pivoted Video cameras with common optical axis and different Not reach spectral ranges. The latter are in fact pivoting Avoidance of motion blur either slowly, but then no complete tapestry is recordable, or impractical in so-called step-and-star operation, which results in a Gesamtverschwenkung the cameras only low cycles frequency achieve.

According to another embodiment of the invention, the fields of view of the video cameras are preferably variable by means of zoom or switching lenses, but have mutually equal solid angle sizes. In particular in connection With the temporal timing or nesting described above, the advantage in this embodiment is that the images of the various video cameras recorded temporally successively have minimally different visual fields and perspectives, whereby mosaicking and fusing can be carried out in a particularly simple manner. In this case, the temporal nesting has the particularly advantageous effect that in each case alternately the image center of a spectral component image coincides geometrically with the overlapping region of the second spectral component image, with the consequence of simplified and more precise mosaicing and fusing.

embodiments of the subject invention are described below with reference to the drawing, wherein all described and / or illustrated features alone or in any combination the subject of the present invention independently from their summary in the claims or their dependency form. Show it:

1 a schematic, partial perspective view of a multispectral video device for airborne observation according to a first embodiment;

2 a schematic representation in the temporal change of alternatively changing image sequences of two video cameras and the resulting video image sequence;

3 a schematic representation of two geometric image strips of in 2 illustrated alternatively changing image sequences;

4 a schematic representation of the two image strips according to 3 in simplified form, which image strips form the result image;

5 a schematic, partial perspective view of a multispectral video device for airborne observation according to a second embodiment with only one integrated for both video cameras mirror pivoting device, which performs the viewing direction pivoting for the two video cameras in rapid time change; and

6 a schematic, partial perspective view of a multispectral video device for the airborne observation according to another embodiment with a uniaxial, taking place for both video cameras in quick temporal change swivel drive for two coaxial fixedly connected, moving mirrors.

A multispectral video device 1 for airborne observation is shown in a schematic, partially perspective view in FIG 1 shown. The video device 1 has at least two video scanner cameras 2 . 3 , which are equipped with surface sensors not shown in detail. Furthermore, the mentioned video scanner cameras have 2 . 3 , also referred to as video cameras below, have different spectral sensitivities. According to a preferred embodiment of the invention, the spectral sensitivities of the video cameras are 2 . 3 in the visible range or in the infrared range.

Furthermore, the video device has 1 an electronic swivel control device 4 , by means of the viewing directions 5 . 6 the video cameras 2 . 3 on the one hand independently of each other, on the other hand in the coordinated step operation are pivotable so that the images A, B of recording sequences 10 . 11 cover each video camera at least partially edge.

Finally, the video device has 1 an electronic mosaicking / fusing device 12 , This is designed such that it receives the image signals 13 . 14 the video cameras 2 . 3 receives whose individual images A, B zusammenfügt to coherent images and geometrically largely congruent with a multi-spectral result image 15 superimposed.

According to a preferred and in 1 indicated embodiment of the invention, the swivel control device 4 preferably electromotively movable mirrors 16 . 17 to pivot the view directions 5 . 6 the video cameras 2 . 3 on. According to 1 are the mirrors 16 . 17 each by means of two electromotive adjustments 20 . 21 ; 22 . 23 adjustable, the two-axis mirror pivoting by means of the double arrows C, D and E, F are illustrated. This means an adjustment of the mirror 16 in the direction of the double arrow C, its pivoting about a horizontal axis and in the direction of the double arrow D about a vertical axis. Analogous versions apply to a pivoting of the mirror 17 in the direction of the double arrows E, F.

As in 1 are therefore located in front of the video cameras 2 . 3 Cardan adjustable rotary mirror systems 24 . 25 , wherein the electrical control of the displacements by the aforementioned pivot control device 4 he follows. These are the individual electromotive adjustments 20 to 23 via electrical lines 30 with the swivel control device 4 connected.

The in 1 illustrated case of biaxial pivoting allows the precise spatial orientation of the camera viewing directions under Be Consideration and compensation of the attitude movements of the aircraft resulting in simplified ratios for mosaicking and merging. Alternatively, according to another embodiment described in more detail below, mirror pivoting about one of the two axes, preferably about the vertical axis (see double arrows D and F in FIG 1 ), omitted, ie, only one-axis pivoting can be realized, with the result of reduced mechanical complexity, but increased effort in the subsequent mosaicking and fusing.

For transmitting the video camera output signals representing the image signals 13 . 14 included are the video cameras 2 . 3 over other electrical lines 31 . 32 with the mosaicizer / fuser 12 connected. There the video camera output signals become geometric stripes 33 and further to tapestries (not shown in detail) composed of numerous image strips, mosaicked and further to a multispectral result image 15 merged.

How closer in 2 shown are video cameras 2 . 3 and swivel control device 4 According to a particularly preferred embodiment of the invention controllable such that the image recordings A, B, the video cameras 2 . 3 are coordinated in a periodic clock. Furthermore, the pivoting of the gaze directions 5 . 6 the video cameras 2 . 3 synchronized in start-stop operation with this clock so that the line of sight 5 or 6 one of the at least two video cameras 2 or 3 related to that line of sight 6 or 5 the at least one other video camera 3 . 2 is at a standstill and takes a picture A or B while the line of sight 6 or 5 the at least one other video camera 3 or 2 related to that line of sight 5 . 6 one of the at least two video cameras 2 . 3 in the movement.

Standstill and taking a picture are in 2 marked by the reference numerals "A" and "B", whereas a movement, in particular a pivoting of the viewing direction, by the in 2 symbolized arrows G is symbolized. In 2 is thus the temporal change between movement on the one hand and standstill and recording on the other side clarified. In the case of the picture shown here by two video cameras 2 . 3 in a common video clock with the resulting consecutively according to clocks i - 1, i, i + 1, ...., i + 7 numbered pictures A, B, also called video frames, change motion (see illustration of the arrows G) and thus motion unsharp frame and standstill (and therefore taking the picture A, B and frame without motion blur) of the line of sight. This sequence takes place in the alternative change, so that in the mosaic / fusion device 12 the temporal sequence of the images A, B, A, B ... of the frames without motion blur is available for the construction of result images.

The picture strip 33 thus illustrates the temporal sequence of the individual images A, B, but not the geometric structure of a picture strip.

The fields of vision 34 . 35 the video cameras 2 . 3 are preferably variable by means not shown in detail zoom or switching lenses. Further preferably, the fields of view 34 . 35 however, mutually equal solid angle sizes.

As in 3 shown in more detail schematically, the successive viewing direction pivoting takes place of the exemplary two video cameras 2 . 3 such that the visual fields 34 . 35 of the two cameras in successive standstill phases, see Figure A (i), A (i + 2), A (i + 4), A (i + 6) ... and B (i - 1), B (i + 1), slightly overlap B (i + 3), B (i + 5), B (i + 7) .... This coverage area 36 For example, the individual images A and B are about 20% of the image width.

The sequence of the video images A (i), A (i + 2), A (i + 4)... Or B (i-1), B (i + 1)... thus forms two image strips 40 . 41 in different spectral ranges, but in geometrically largely similarity. These picture strips 40 . 41 with nearly identical image contents are summarized and simplified in 4 shown. It follows that in the picture strip 40 . 41 each pixel P, P 'correspond geometrically, so that, as previously mentioned, the image strips 40 . 41 represent, as it were, congruent layers of a multispectral image strip. As mentioned, the picture strips 40 . 41 in the electronic mosaic / fusion device 12 mosaicized geometrically to tapestries not shown in detail and further to a multispectral result image 15 merged. Such a result picture 15 arises according to 4 in that the picture strips 40 . 41 be connected to a multispectral result image.

Unless in another embodiment of the invention the visual fields 34 . 35 the video cameras 2 . 3 are adjustable, this adjustment is made simultaneously for all video cameras, so that the fields of view and perspectives successively recorded images of different video cameras always have minimal differences. In this case, the angular step sizes of the start / stop movement of the individual cameras are adapted in such a way that adjacent images in an image strip each have a coverage area 36 to allow image content related mosaicking and to avoid the occurrence of coverage gaps. As mentioned, the coverage areas are 36 in 3 hatched shown.

The Image recordings of the video cameras are, for example, in the television clock, z. In the CCIR / PAL standard, with 25 frames per second or Coordinates 50 fields per second.

In a practical application is when using only one Mirror swivel axis, preferably parallel to the longitudinal axis of the camera-carrying aircraft, an interactive object tracking, also called object tracking, by two-dimensional gaze tilting in the way possible, that a pivoting dimension by the pivoting of the camera, the second Panning dimension, however, by coordinated pitch and yaw position changes the camera carrying the aircraft is realized. Thus, that can Panning the viewing directions uniaxial, biaxial or triaxial corresponding to a pan around one, two or three Euler angles in a fixed to the camera-bearing aircraft coordinate system respectively.

The image signal connection from the video cameras 2 . 3 to the mosaicizer / fuser 12 Can be wired or wireless. It is irrelevant which transmission distance is bridged. The signal connection can be made aboard the camera-carrying aircraft or to a ground station. The visual fields of the video cameras can also be dimensioned differently. It is not necessary that the video camera assembly is fixedly connected to the camera-carrying airframe; Rather, the video camera arrangement can be designed to be movable as a whole in the aircraft, z. B. gimbaled to realize, for example, a spatial stabilization.

Regarding 4 It is pointed out that the pixels P, P 'represent any desired pair of points in the two image layers or image planes or image strips. For color images, the term "color separations" is used in the printing industry. These indicate the cyan, magenta, yellow and black levels. In this sense, the images A ... and B ... are "spectral extracts" of the same multispectral image, just as a color image is strictly speaking also a multispectral image as compared to a black and white image.

It should also be noted that in 3 to be detected displacement of the image strips is caused only by staggered recording times, the size of this temporal offset advantageously corresponds to about half the image recording clock. However, such a size of the offset is not mandatory and not advantageous in all applications. It remains the case that the picture strips 40 . 41 geometrically largely congruent, but include different spectral ranges.

5 shows a variation of in 1 illustrated embodiment in such a way that here the two mirrors 16 . 17 the first embodiment to a mirror pivoting device 42 are integrated. Their only mirror 43 is widened so far that he the beam paths of the two video cameras 2 and 3 reflected side by side. Furthermore, their electromotive adjustments work 20 . 21 in this embodiment, as a rotary actuators with increased angular velocity, so that the pivoting of the viewing directions 5 and 6 the video cameras 2 and 3 in the time alternating cycle with this one integrated mirror pivoting device 42 respectively. Adjacent plane deflection mirror 44 . 45 serve only the purpose, the beam paths of the cameras 2 . 3 regardless of the camera dimensions so close to each other that the both beam paths detected mirror 43 as small as possible can be dimensioned.

6 shows a uniaxially adjustable special version of the 1 and 5 shown embodiments in which the viewing directions 5 . 6 the video cameras 2 . 3 again guided in spatial separation, namely without the adjacent plan deflection mirror 44 . 45 about individual mirrors 16 and 17 , firmly on both sides extended axle parts 46 . 47 the only electromotive adjustment 20 are mounted in the form of a rotary actuator. The tilt of the camera's directions of view 5 . 6 again takes place in temporal change. This arrangement has the particular advantage of mechanical simplicity and robustness, but places increased demands on the subsequent mosaicking and fusing.

In order to is a multispectral video device created by means of a complete one Tapestry in high quality form is mapped.

Claims (7)

Multispectral video device for airborne observation, with at least two video-scanning cameras ( 2 . 3 ), which are equipped with surface sensors and have different spectral sensitivities, an electronic swivel control device ( 4 ), by means of which the viewing directions ( 5 . 6 ) of the video cameras ( 2 . 3 ) on the one hand independently of one another, on the other hand in coordinated stepping operation, are pivotable in such a way that images (A, B) of sequences of images (FIG. 10 . 11 ) of each video camera ( 2 . 3 ) at least partially obscure edge, and with an electronic mosaic / fusion device ( 12 ), which is designed such that it receives the image signals ( 13 . 14 ) of the video cameras ( 2 . 3 ) whose individual images (A, B) merge into coherent images and which are geometrically largely congruent with a multispectral result image (FIG. 15 ) superimposed. Multispectral video device according to claim 1, characterized in that the spectral sensitivities of the video cameras ( 2 . 3 ) are in the visible range or in the infrared range. Multispectral video device according to claim 1 or 2, characterized in that the swivel control device ( 4 ) preferably electromotively movable mirrors ( 16 . 17 ) for pivoting the viewing directions ( 5 . 6 ) of the video cameras ( 2 . 3 ) having. Multispectral video device according to one of claims 1 to 3, characterized in that video cameras ( 2 . 3 ) and swivel control device ( 4 ) are controllable such that the image recordings (A, B) of the video cameras ( 2 . 3 ) are coordinated in a periodic clock and the pivoting of the viewing directions ( 5 . 6 ) of the video cameras ( 2 . 3 ) is synchronized in start / stop operation with this clock such that the viewing direction ( 5 ; 6 ) one of the at least two video cameras ( 2 ; 3 ) in relation to the viewing direction ( 6 ; 5 ) the at least one other video camera ( 3 ; 2 ) is at a standstill and takes an image (A; B), while the viewing direction ( 6 ; 5 ) the at least one other video camera ( 3 ; 2 ) in relation to the viewing direction ( 5 ; 6 ) of one of the at least two video cameras ( 2 ; 3 ) is in motion. Multispectral video device according to one of the preceding claims, characterized in that the visual fields ( 34 . 35 ) of the video cameras ( 2 . 3 ) are preferably variable by means of zoom or switching lenses, however, have mutually equal solid angle sizes. Multispectral video device according to one of the preceding claims, characterized in that the mirrors ( 16 . 17 ) in a mirror pivoting device ( 42 ) are integrated, by means of which the independent pivoting of the viewing directions ( 5 . 6 ) of the video cameras ( 2 . 3 ) are carried out in the time change clock. Multispectral video device according to one of the preceding claims, characterized in that the mirrors ( 16 . 17 ) by a common pivot drive preferably uniaxially in temporal change are pivotally.