AU2018204554A1

AU2018204554A1 - Method, system and apparatus for determining velocity of an object in a scene

Info

Publication number: AU2018204554A1
Application number: AU2018204554A
Authority: AU
Inventors: Matthew Raphael Arnison; Peter Alleine Fletcher; Ruimin Pan
Original assignee: Canon Inc
Current assignee: Canon Inc
Priority date: 2018-06-22
Filing date: 2018-06-22
Publication date: 2020-01-16

Abstract

METHOD, SYSTEM AND APPARATUS FOR DETERMINING VELOCITY OF AN OBJECT IN A SCENE A method of determining velocity of an object in a scene imaged by an imaging device. A first frame and a second frame are captured using the imaging device, each frame being associated with a first image captured using a first optical configuration and a second image captured using a second optical configuration. A first shift is determined for a first pixel position between the first image associated with the first frame and the first image associated with the second frame. Velocity of the object moving in the scene is determined by compensating for distortion caused by a turbulence phase screen based on the first shift, a distance from the imaging device to the turbulence phase screen and a second shift affected by the turbulence phase screen. The second shift is determined from the second image associated with the first frame and the second image associated with the second frame. The second shift is subject to the turbulence phase screen. 17294453v1

Description

METHOD, SYSTEM AND APPARATUS FOR DETERMINING VELOCITY OF AN OBJECT IN A SCENE TECHNICAL FIELD

The present invention relates to a method and apparatus for determining velocity of an object in a scene imaged by an imaging device, and to a computer program product including a computer readable medium having recorded thereon a computer program for determining velocity of an object in a scene imaged by an imaging device.

BACKGROUND

Atmospheric turbulence is a well-known source of distortion that can degrade the quality of images and videos acquired by cameras viewing scenes from long distances. Images captured at high magnification through large volumes of atmosphere exhibit a shimmering effect, including blur and image shifts which vary randomly across the image and over time. The shimmering effect is particularly significant for astronomy and long-distance surveillance.

Atmospheric turbulence can be compensated for using optical methods or image processing. Optical compensation of turbulence is known as adaptive optics. In adaptive optics, a wavefront sensor is used to continuously detect the phase wavefront error due to turbulence, and a deformable mirror is continuously adjusted to compensate for the wavefront error in each image captured. However, the optical equipment required for adaptive optics is expensive, fragile and bulky. In addition, adaptive optics is only able to compensate for turbulence within a very small field of view. In astronomy, the very small field of view is acceptable, but in long-distance surveillance it is desirable to have a portable imaging system with a wider field of view.

Compensation of turbulence using image processing involves detecting the image shift from each frame of a captured video with respect to a reference frame, in each region of the image. The reference frame is an average of a large number of frames over in which the image shifts due to atmospheric turbulence average to zero. The detected image shifts can be used to warp the captured frame to align with the reference frame. The warping method can produce a significantly sharper and more stable video than the original captured video.

Surveillance is used to monitor changes in a scene, including identifying objects and tracking object activity and movement. It is desirable that images of moving objects are high quality. If the scene is being viewed through strong turbulence, such as in hot or windy weather or long distance imaging, then apparent motions in the image may be caused by turbulence or object motion. If an image processing method is used to compensate for turbulence, then the measured image shifts compared to a reference frame may be caused by turbulence or by object motion, and the compensated video will have the object appear to be stationary. However, information about the motion of objects is important for surveillance, so this type of compensation causes problems for the user.

In another turbulence compensation image processing method, object motion is estimated using adjacent video frames. Shifts which are larger than the image shift expected due to turbulence, or shifts which have a consistent direction over multiple frames, are classified as object motion. Regions with object motion can be processed differently from regions without object motion. The turbulence compensation can use fewer frames or be disabled for regions with object motion. However, using fewer frames may reduce the image quality for the moving objects. The method of estimating object motion using adjacent video frames relies on thresholds on the image shifts to detect motion. The detection of motion is unreliable for slowly moving objects, or objects that are accelerating. The inaccurate object velocity estimates further reduce the image quality of the turbulence compensated video for moving objects.

SUMMARY

It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

Disclosed are arrangements for determining object velocity for objects viewed through atmospheric turbulence and for compensating for a degradation effect due to atmospheric turbulence in image processing.

According to one aspect of the present disclosure there is provided a method of determining velocity of an object in a scene imaged by an imaging device, the method comprising: capturing a first frame and a second frame using the imaging device, each frame being associated with a first image captured using a first optical configuration and a second image captured using a second optical configuration; determining a first shift for a first pixel position between the first image associated with the first frame and the first image associated with the second frame; and determining velocity of the object moving in the scene by compensating for distortion caused by a turbulence phase screen based on the first shift, a distance from the imaging device to the turbulence phase screen and a second shift affected by the turbulence phase screen, wherein the second shift is determined from the second image associated with the first frame and the second image associated with the second frame, and wherein the second shift is subject to the turbulence phase screen.

According to another aspect of the present disclosure there is provided an apparatus for determining velocity of an object in a scene imaged by an imaging device, the apparatus comprising:

means for capturing a first frame and a second frame using the imaging device, each frame being associated with a first image captured using a first optical configuration and a second image captured using a second optical configuration;

means for determining a first shift for a first pixel position between the first image associated with the first frame and the first image associated with the second frame; and

means for determining velocity of the object moving in the scene by compensating for distortion caused by a turbulence phase screen based on the first shift, a distance from the imaging device to the turbulence phase screen and a second shift affected by the turbulence phase screen, wherein the second shift is determined from the second image associated with the first frame and the second image associated with the second frame, wherein the second shift is subject to the turbulence phase screen.

According to still another aspect of the present disclosure there is provided a system for determining velocity of an object in a scene imaged by an imaging device, the system comprising: a memory for storing data and a computer program; a processor coupled to the memory for executing the computer program, the computer program comprising instructions for: capturing a first frame and a second frame using the imaging device, each frame being associated with a first image captured using a first optical configuration and a second image captured using a second optical configuration; determining a first shift for a first pixel position between the first image associated with the first frame and the first image associated with the second frame; and determining velocity of the object moving in the scene by compensating for distortion caused by a turbulence phase screen based on the first shift, a distance from the imaging device to the turbulence phase screen and a second shift affected by the turbulence phase screen, wherein the second shift is determined from the second image associated with the first frame and the second image associated with the second frame, wherein the second shift is subject to the turbulence phase screen.

According to still another aspect of the present disclosure there is provided a computer readable medium having a computer program stored on the medium for determining velocity of an object in a scene imaged by an imaging device, the program comprising:

code for capturing a first frame and a second frame using the imaging device, each frame being associated with a first image captured using a first optical configuration and a second image captured using a second optical configuration;

code for determining a first shift for a first pixel position between thefirst image associated with the first frame and the first image associated with the second frame; and

code for determining velocity of the object moving in the scene by compensating for distortion caused by a turbulence phase screen based on the first shift, a distance from the imaging device to the turbulence phase screen and a second shift affected by the turbulence phase screen, wherein the second shift is determined from the second image associated with the first frame and the second image associated with the second frame, wherein the second shift is subject to the turbulence phase screen.

According to still another aspect of the present disclosure there is provided a method of determining velocity of an object in a scene, the method comprising:

capturing a first frame and a second frame, each frame being associated with a first image captured using a first optical configuration and a second image captured using a second optical configuration;

determining a first shift for a first pixel position between the first image associated with the first frame and the first image associated with the second frame;

determining a second pixel position in the second image of the first frame and the second image of the second frame which is subject to the turbulence phase screen as the first pixel position;

determining a second shift at the second pixel position between the second image of the first frame and the second image of the second frame; and

determining velocity of the object moving in the scene based on comparison of the first shift and the second shift.

According to still another aspect of the present disclosure there is provided an apparatus for determining velocity of an object in a scene, the apparatus comprising:

means for capturing a first frame and a second frame, each frame being associated with a first image captured using a first optical configuration and a second image captured using a second optical configuration;

means for determining a first shift for a first pixel position between the first image associated with the first frame and the first image associated with the second frame; means for determining a second pixel position in the second image of the first frame and the second image of the second frame which is subject to the turbulence phase screen as the first pixel position; means for determining a second shift at the second pixel position between the second image of the first frame and the second image of the second frame; and means for determining velocity of the object moving in the scene based on comparison of the first shift and the second shift.

According to still another aspect of the present disclosure there is provided a system for determining velocity of an object in a scene, the system comprising:

a memory for storing data and a computer program;

a processor coupled to the memory for executing the computer program, the computer program comprising instructions for:

According to still another aspect of the present disclosure there is provided a computer readable medium having a computer program stored on the medium for determining velocity of an object in a scene imaged, the program comprising:

code for capturing a first frame and a second frame, each frame being associated with a first image captured using a first optical configuration and a second image captured using a second optical configuration;

code for determining a first shift for a first pixel position between thefirst image associated with the first frame and the first image associated with the second frame;

code for determining a second pixel position in the second image of the first frame and the second image of the second frame which is subject to the turbulence phase screen as the first pixel position;

code for determining a second shift at the second pixel position between the second image of the first frame and the second image of the second frame; andcode for determining velocity of the object moving in the scene based on comparison of the first shift and the second shift.

According to still another aspect of the present disclosure there is provided a method of determining an undistorted image of a scene, said method comprising:

capturing a first frame and a second frame, each frame being associated with a first image of the scene captured using a first optical configuration and a second image of the scene captured using a second optical configuration;

determining velocity of an object moving in the scene based on comparison of the first image and the second image;

segmenting foreground regions of the captured frames containing moving objects using the determined velocity to create at least one undistorted foreground region; and

compositing the undistorted foreground region onto a background region of the captured frames to create the undistorted image.

Other aspects are also disclosed.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the invention will now be described with reference to the following drawings, in which:

Fig. 1A shows a camera capturing an image of a moving object through atmospheric turbulence;

Fig. 1B shows a phase screen placed between the camera and the moving object;

Fig. 2 is a diagram showing an object, phase screen and image plane geometry for capturing an image of a scene through turbulence using a stereo camera;

Fig. 3 shows ray tracing geometry for the wavefront error at the lens aperture for capturing an image of a scene through turbulence using a stereo camera;

Fig. 4 shows the ray tracing geometry for the object, phase screen, lens aperture and image planes for capturing an image of a scene through turbulence using a stereo camera;

Fig. 5 is a schematic flow diagram showing a method of compensating for turbulence in an image of an object;

Fig. 6 shows focal plane geometry for a single main pixel of a dual-pixel auto-focus sensor camera;

Fig. 7 shows an arrangement of sub-pixels and main pixels on a dual-pixel auto-focus sensor;

Fig. 8 is a schematic flow diagram showing a method of determining an image offset map, as used in the method of Fig. 5;

Fig. 9 is a schematic flow diagram showing a method of determining a stereo camera image offset, which may be alternatively used in the method of Fig. 5;

Fig. 10 is a schematic flow diagram showing a method of detemining an undistorted image as used in the method of Fig. 5;

Fig. 11 is a schematic flow diagram showing a method of determining velocity of a moving object, using a turbulence phase screen distance;

Fig. 12 is a schematic flow diagram showing a method of determining velocity of a moving object, using a pixel position offset; and

Figs. 13A and 13B collectively form a schematic block diagram representation of a camera upon which described arrangements can be practised.

DETAILED DESCRIPTION INCLUDING BEST MODE

Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.

Long-distance imaging through the atmosphere is subject to rapidly varying turbulence which significantly affects image quality, especially in hot or windy conditions. It is desirable to have high image quality in surveillance imaging, especially for moving objects. Image processing may be used to compensate for turbulence, including in video of moving objects which may be compensated for by tracking object motion.

Accurate turbulence compensation for images of moving objects requires determining the object motion independently from the apparent motion due to turbulence. A method of determining object velocity through turbulence is described below. The method may be used to improve the image quality of turbulence compensation for long-distance surveillance video of moving objects.

Long-distance imaging through turbulence will now be described by way of example with reference to Fig. 1A. In the example of Fig. 1A, a camera 100 captures images through the atmosphere 110 of a moving foreground object 120, such as a car, against a stationary background object 130, such as a building. The foreground and background together form the scene being imaged.

Random spatio-temporal variations in the temperature and density of the atmosphere 110 cause fluctuations in the refractive index in the air between the camera 100 and the scene. The fluctuations cause wavefront errors in the light propagating from the scene to the camera. The wavefront errors cause imaging artefacts including image shift and blur. The artefacts vary across the image and over time, with the amount of variation and the strength of the artefacts depending on the strength of the atmospheric turbulence. In general, atmospheric turbulence is stronger close to the ground and in hot or windy weather.

The effects of atmospheric turbulence can be approximated using a thin phase screen. Fig. 1B shows a phase screen 115 placed between the camera 100 and scene comprising the object 120 and the object 130 of Fig. 1A. The phase screen 115 describes the phase changes caused by fluctuations in refractive index between the camera 100 and the scene for each light ray passing from the scene to the camera 100. The phase screen approximation may be especially appropriate when there is a thin region in between the camera 100 and the scene with stronger turbulence, for example if the imaging path passes over the top of a ridge in the terrain then the turbulence will be stronger at the distance of the ridge, where the imaging path is closest to the ground, than over the rest of the imaging path.

Fig. 2 is a diagram showing a scene 210, phase screen 250 and image plane geometry for imaging through turbulence with a stereo camera (e.g., the camera 1301) which is not shown in Fig. 2. The stereo camera has a horizontal baseline 270 parallel with the x-axis, and consists of a left camera including a left lens 220 and a left image sensor 230, and a right camera including a right lens 225 and a right image sensor 235. The image sensors 230 and 235 are in the x-y plane, and the optical axes of the cameras are parallel to the z-axis. The ray passing from the centre of the lens aperture to a scene point is called a chief ray.

The phase screen 250 is an approximation for the effect of atmospheric turbulence which can be expressed as a two-dimensional (2D) function of position within a plane parallel to the camera image sensors 230 and 235. In Fig. 2, the phase screen 250 is at a distance zp from the stereo camera, in between the cameras and the scene, where the distance zp is determined by the atmospheric conditions which may vary slowly over time during the course of a day. The value of the phase screen at each position in the plane is the optical path length delay caused by fluctuations in refractive index caused by atmospheric turbulence, where the optical path length delay is relative to the optical path length for rays propagating through the atmosphere at the mean refractive index. The phase screen values vary rapidly and randomly with transverse position and over time. The temporal variation is on the order of milliseconds.

The ray tracing geometry for the wavefront error at the lens aperture is shown in Fig. 3. Considering a scene point A and the left camera lens 320, the wavefront error caused by the phase screen 350 may be represented as a 2D function of position across the left lens aperture 340. The amount of wavefront error at each position on the lens aperture is determined by the change in optical path length caused by the phase screen for the ray from scene point A to the lens aperture position. In this way, the wavefront error depends on distance 360 from the phase screen to the lens, the optical path length delay values of the phase screen, and the position of the scene point A. The amount of shift and blur in the left image of scene point A is determined from the wavefront error for the left lens.

Similarly, the wavefront error across the right lens aperture 345 in the right camera lens 325 is determined for scene point B at each lens aperture position by ray tracing from scene point B through the phase screen 350 to the lens aperture position. As shown in Fig. 3, if the left camera rays and the right camera rays have common intersection points with the phase screen for each lens aperture position, then the left camera lens wavefront error is the same as the right camera lens wavefront error. The common intersection will happen at a specific scene point B at a transverse distance from scene point A, where the transverse distance is called the field offset 300.

The shift and blur caused by the wavefront error for each lens may be represented using a point spread function (PSF), which is convolved with the scene point A to form an image AL. Spatio temporal variations in the phase screen produce variations in the PSF across the image and from frame to frame, such that the PSF may only be considered constant for a small regions within single frames.

Using the baseline 270, the distance between the camera and the phase screen 250, the distance between the camera and the scene, and the position of a scene point A, a second scene point B may be determined for which the chief ray of the left lens imaging scene point A intersects at the phase screen with the chief ray of the right lens imaging scene point B. If the left and right cameras have the same lens aperture diameter, then the left camera image AL of scene point A is affected by the same wavefront error as the right camera image BR of scene point B. The atmospheric turbulence will cause the same PSF and therefore the same shift and blur in the scene point images AL and BR. The transverse distance between scene points A and B is referred to as "field offset" 200, as seen in Fig. 2. The transverse distance between the image points AR and BR is referred to as "image offset" 260, where image point AR is the right camera image of scene point A.

As an example, the object at scene point A is stationary, such as a building in the background of the scene, and the object at scene point B is moving, such as a moving car. Video frames are captured using the left camera and the right camera at times ti and t2. The two (2) frames captured with the left camera include an image AL of scene point A which is shifted between the first frame and the second frame by the vector difference of the shifts caused by atmospheric turbulence at times t2 and ti. The two (2) frames captured with the right camera include an image BR of scene point B which is shifted between the first frame and the second frame by the vector difference of the shifts caused by atmospheric turbulence at times t2 and ti and the shift caused by object motion at scene point B.

The image shifts at AL and BR may be determined using image processing methods such as cross-correlation. By subtracting the image shift at BR from the image shift at AL, the turbulence shifts cancel out, because the turbulence shifts for AL and BR are the same. The remaining value in the image shift difference is the shift caused by object motion at scene point B. In this way, the object velocity may be determined independently of rapid spatial and temporal variation in the atmospheric turbulence.

Figs. 13A and 13B collectively form a schematic block diagram of an imaging device in the form of a camera 1301 including embedded components, upon which methods to be described are desirably practiced. The camera 1301 may be, for example, a video camera, a surveillance camera, or a mobile phone, in which processing resources are limited. Nevertheless, the methods to be described may also be performed on higher-level devices such as desktop computers, server computers, and other such devices with significantly larger processing resources.

As seen in Fig. 13A, the camera 1301 comprises an embedded controller 1302. Accordingly, the camera 1301 may be referred to as an "embedded device." In the present example, the controller 1302 has a processing unit (or processor) 1305 which is bi-directionally coupled to an internal storage module 1309. The storage module 1309 may be formed from non-volatile semiconductor read only memory (ROM) 1360 and semiconductor random access memory

(RAM) 1370, as seen in Fig. 13B. The RAM 1370 may be volatile, non-volatile or a combination of volatile and non-volatile memory.

The camera 1301 includes a display controller 1307, which is connected to a video display 1314, such as a liquid crystal display (LCD) panel or the like. The display controller 1307 is configured for displaying graphical images on the video display 1314 in accordance with instructions received from the embedded controller 1302, to which the display controller 1307 is connected.

The camera 1301 also includes user input devices 1313 which are typically formed by keys, a keypad or like controls. In some implementations, the user input devices 1313 may include a touch sensitive panel physically associated with the display 1314 to collectively form a touch screen. Such a touch-screen may thus operate as one form of graphical user interface (GUI) as opposed to a prompt or menu driven GUI typically used with keypad-display combinations. Other forms of user input devices may also be used, such as a microphone (not illustrated) for voice commands or a joystick/thumb wheel (not illustrated) for ease of navigation about menus.

As seen in Fig. 13A, the camera 1301 also comprises a portable memory interface 1306, which is coupled to the processor 1305 via a connection 1319. The portable memory interface 1306 allows a complementary portable memory device 1325 to be coupled to the camera 1301 to act as a source or destination of data or to supplement the internal storage module 1309. Examples of such interfaces permit coupling with portable memory devices such as Universal Serial Bus (USB) memory devices, Secure Digital (SD) cards, Personal Computer Memory Card International Association (PCMIA) cards, optical disks and magnetic disks.

The camera 1301 also has a communications interface 1308 to permit coupling of the device 1301 to a computer or communications network 1320 via a connection 1321. The connection 1321 may be wired or wireless. For example, the connection 1321 may be radio frequency or optical. An example of a wired connection includes Ethernet. Further, an example of wireless connection includes BluetoothTM type local interconnection, Wi-Fi (including protocols based on the standards of the IEEE 802.11 family), Infrared Data Association (IrDa) and the like.

Typically, the camera 1301 may be configured to perform some special function. The embedded controller 1302, possibly in conjunction with further special function components 1310, is provided to perform that special function. For example, the components 1310 may represent a lens, focus control and image sensor of the camera. The special function components 1310 is connected to the embedded controller 1302. As another example, the device 1301 may be a mobile telephone handset. In this instance, the components 1310 may represent those components required for communications in a cellular telephone environment. Where the device 1301 is a portable device, the special function components 1310 may represent a number of encoders and decoders of a type including Joint Photographic Experts Group (JPEG), (Moving Picture Experts Group) MPEG, MPEG-iAudio Layer 3 (MP3), and the like.

The methods described hereinafter may be implemented using the embedded controller 1302, where the processes of Figs. 5, 8, 9, 10, 11 and 12 may be implemented as one or more software application programs 1333 executable within the embedded controller 1302. The camera 1301 of Fig. 13A implements the described methods. In particular, with reference to Fig. 13B, the steps of the described methods are effected by instructions in the software 1333 that are carried out within the controller 1302. The software instructions may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules performs the described methods and a second part and the corresponding code modules manage a user interface between the first part and the user.

The software 1333 of the embedded controller 1302 is typically stored in the non-volatile ROM 1360 of the internal storage module 1309. The software 1333 stored in the ROM 1360 can be updated when required from a computer readable medium. The software 1333 can be loaded into and executed by the processor 1305. In some instances, the processor 1305 may execute software instructions that are located in RAM 1370. Software instructions may be loaded into the RAM 1370 by the processor 1305 initiating a copy of one or more code modules from ROM 1360 into RAM 1370. Alternatively, the software instructions of one or more code modules may be pre-installed in a non-volatile region of RAM 1370 by a manufacturer. After one or more code modules have been located in RAM 1370, the processor 1305 may execute software instructions of the one or more code modules.

The application program 1333 is typically pre-installed and stored in the ROM 1360 by a manufacturer, prior to distribution of the camera 1301. However, in some instances, the application programs 1333 may be supplied to the user encoded on one or more CD-ROM (not shown) and read via the portable memory interface 1306 of Fig. 13A prior to storage in the internal storage module 1309 or in the portable memory 1325. In another alternative, the software application program 1333 may be read by the processor 1305 from the network 1320, or loaded into the controller 1302 or the portable storage medium 1325 from other computer readable media. Computer readable storage media refers to any non-transitory tangible storage medium that participates in providing instructions and/or data to the controller 1302 for execution and/or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, flash memory, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the device 1301. Examples of transitory or non tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and/or data to the device 1301 include radio or infra red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like. A computer readable medium having such software or computer program recorded on it is a computer program product.

The second part of the application programs 1333 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display 1314 of Fig. 13A. Through manipulation of the user input device 1313 (e.g., the keypad), a user of the device 1301 and the application programs 1333 may manipulate the interface in a functionally adaptable manner to provide controlling commands and/or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing speech prompts output via loudspeakers (not illustrated) and user voice commands input via the microphone (not illustrated).

Fig. 13B illustrates in detail the embedded controller 1302 having the processor 1305 for executing the application programs 1333 and the internal storage 1309. The internal storage 1309 comprises read only memory (ROM) 1360 and random access memory (RAM) 1370. The processor 1305 is able to execute the application programs 1333 stored in one or both of the connected memories 1360 and 1370. When the camera 1301 is initially powered up, a system program resident in the ROM 1360 is executed. The application program 1333 permanently stored in the ROM 1360 is sometimes referred to as "firmware". Execution of the firmware by the processor 1305 may fulfil various functions, including processor management, memory management, device management, storage management and user interface.

The processor 1305 typically includes a number of functional modules including a control unit (CU) 1351, an arithmetic logic unit (ALU) 1352, a digital signal processor (DSP) 1353 and a local or internal memory comprising a set of registers 1354 which typically contain atomic data elements 1356, 1357, along with internal buffer or cache memory 1355. One or more internal buses 1359 interconnect these functional modules. The processor 1305 typically also has one or more interfaces 1358 for communicating with external devices via system bus 1381, using a connection 1361.

The application program 1333 includes a sequence of instructions 1362 though 1363 that may include conditional branch and loop instructions. The program 1333 may also include data, which is used in execution of the program 1333. This data may be stored as part of the instruction or in a separate location 1364 within the ROM 1360 or RAM 1370.

In general, the processor 1305 is given a set of instructions, which are executed therein. This set of instructions may be organised into blocks, which perform specific tasks or handle specific events that occur in the camera 1301. Typically, the application program 1333 waits for events and subsequently executes the block of code associated with that event. Events may be triggered in response to input from a user, via the user input devices 1313 of Fig. 13A, as detected by the processor 1305. Events may also be triggered in response to other sensors and interfaces in the camera 1301.

The execution of a set of the instructions may require numeric variables to be read and modified. Such numeric variables are stored in the RAM 1370. The disclosed method uses input variables 1371 that are stored in known locations 1372, 1373 in the memory 1370. The input variables 1371 are processed to produce output variables 1377 that are stored in known locations 1378, 1379 in the memory 1370. Intermediate variables 1374 may be stored in additional memory locations in locations 1375, 1376 of the memory 1370. Alternatively, some intermediate variables may only exist in the registers 1354 of the processor 1305.

The execution of a sequence of instructions is achieved in the processor 1305 by repeated application of a fetch-execute cycle. The control unit 1351 of the processor 1305 maintains a register called the program counter, which contains the address in ROM 1360 or RAM 1370 of the next instruction to be executed. At the start of the fetch execute cycle, the contents of the memory address indexed by the program counter is loaded into the control unit 1351. The instruction thus loaded controls the subsequent operation of the processor 1305, causing for example, data to be loaded from ROM memory 1360 into processor registers 1354, the contents of a register to be arithmetically combined with the contents of another register, the contents of a register to be written to the location stored in another register and so on. At the end of the fetch execute cycle the program counter is updated to point to the next instruction in the system program code. Depending on the instruction just executed this may involve incrementing the address contained in the program counter or loading the program counter with a new address in order to achieve a branch operation.

Each step or sub-process in the processes of the methods described below is associated with one or more segments of the application program 1333, and is performed by repeated execution of a fetch-execute cycle in the processor 1305 or similar programmatic operation of other independent processor blocks in the camera 1301.

Fig. 5 is a flow diagram showing a method 500 of compensating for turbulence in an image of an object, using the camera 1301, where the image of the object is affected by turbulence in the atmosphere between the camera 1301 and the object. The method 500 will be described by way of example where the camera 1301 is a stereo camera comprising a left camera and a right camera. The method 500 may be used for determining velocity of the object in the scene imaged by an imaging device in the form of the camera 1301. The method 500 may be implemented as one of more software code modules of the software application program 1333 resident in storage module 1309 and being controlled in its execution by the processor 1305.

In determining step 510, an image offset map P(x, y) is determined under execution of the processor 1305. The image offset map P(x, y) is a set of image offset values for each right camera image co-ordinate (x, y). Each image offset value represents the transverse distance (e.g., 260) from image point ARat co-ordinate (x, y) to image point BRfor which the turbulence phase screen is the same for image points AL and BR,where image point AL is the image of scene point A with the left camera, image point ARis the image of scene point A with the right camera, and image point BRis the image of scene point B with the right camera. As described above, this transverse distance (e.g., 260) is referred to as the image offset.

The co-ordinate of image point AL may be determined from the co-ordnate of image point AR

using the depth of the scene point A by ray tracing from image point AR through the centre of the right camera lens to the scene point A, and then ray tracing from scene point A through the centre of the right camera lens to the image point AL. The image offset map P(x, y) determined at step 510 may be stored in the storage module 1309. A method 800 of determining an image offset map, as executed at step 510, will be described in detail below with reference to Fig. 8.

In capturing step 520, first frame images IL (x, y, t1 ) and IR (x, y, t 1) are captured with the stereo camera 1301 at time ti, where IL (x, y, t) is the left camera image captured with a first optical configuration and IR (x, y, t 1 ) is the right camera image captured with a second optical configuration, and (x, y) are the transverse co-ordinates within the image. The first frame images

IL(x, y, t 1) and IR (x, y, t 1) are captured under execution of the processor 1305 and may be stored in the storage module 1309. The exposures are synchronised between the left and right camera, such that the start of the capture time ti and the exposure duration At is the same for both cameras. For example, capture of the first frame starts at ti = 0 milliseconds, and the exposure duration is At = 20 milliseconds.

In one arrangement, the stereo camera 1301 is a camera with a single main lens and a dual-pixel auto-focus (DAF) image sensor. The focal plane geometry for a single main pixel of a DAF sensor camera is shown in Fig. 6. The DAF image sensor has a microlens 640 and a pair of sub pixels for each main image pixel, where the left sub-pixel 660 captures light through the left half 620 of the main lens 610 forming afirst optical configuration, and the right sub-pixel 665 captures light through the right half 625 of the main lens 610 forming a second optical configuration. In the process of image formation by the camera lens and the microlens the left right direction in image space is flipped with respect to the left-right direction in object space. The chief ray 630 is a line from the scene point being imaged by the main pixel, through the centre of the main lens and then to the centre of the main pixel. The arrangement of sub-pixels and main pixels on a DAF sensor is shown in Fig. 7, where the main pixels 710 are square, with each main pixel divided into an equal size pair of rectangular left sub-pixels 720 and rectangular right sub-pixels 725 and the dividing line between the sub-pixels is parallel to the y axis.

The main pixel value is determined by summing the left sub-pixel and right sub-pixel, and the main pixels across the sensor form a standard image. The sub-pixels may also be used to form a stereo image pair with a baseline 635 of half the diameter of the main lens aperture, where the left sub-pixels across the sensor form a left camera image through the left half of the main lens, and the right sub-pixels form a right camera image through the right half of the main lens. Image points in the stereo image pair have a relative shift or disparity along the x axis according to the distance of the corresponding scene point from the best focus plane of the main camera lens. At best focus the disparity is zero (0) pixels, for scene points on the near side of best focus the disparity is negative, and for scene points on the far side of the best focus the disparity is positive. The disparity is normally used to estimate the distance of scene points for the purpose of controlling the camera auto-focus.

In another arrangement, the stereo camera is a pair of cameras of the same model with the same model of lens attached to both cameras, and with the same shooting parameters including focal length, focus distance, relative aperture, exposure gain and shutter speed. The pair of cameras may be synchronised so that for each frame the start time and duration of each image capture is the same. The left camera of the stereo pair forms a first optical configuration, and the right camera of the stereo pair forms a second optical configuration. The baseline is the distance between the centre of the left camera lens aperture and the centre of the right camera lens aperture. The disparity between the stereo image pair is zero for the scene plane parallel to the stereo camera baseline where the optical axes of the left camera lens and the right camera lens intersect, and the disparity varies according to the distance of scene points from the intersection plane to the centre of the stereo camera baseline.

In another arrangement, the pair of cameras may be different models or have different lens models attached, or have different capturing parameters, as long as the physical sizes and shapes of the lens apertures are approximately the same, and the exposure start and end times are the same.

The stereo camera images 'L(x, y, ti) and IR (x, y, ti) are affected by atmospheric turbulence between the camera 1301 and the scene. The turbulence creates distortion in the image, including spatio-temporal variations in image shift. The image shift due to turbulence can be described using a turbulence warp map Wtb (x, y, t) which varies over the image co-ordinates (x, y) and the time t, assuming a constant exposure duration At. The warp map has two (2) values at each spatio-temporal co-ordinate (x, y, t), where the two (2) values are the image shifts in the x and y directions relative to an undistorted image of the scene. The turbulence warp map for the first frame left camera image IL (x, y, ti) is WtbL (x, y, ti) and the turbulence warp map for the first frame right camera image IR (x, y, t1 ) is WtbR (X, y, t 1 ). If the left camera lens aperture shape is the same as the right camera lens aperture shape, and the exposure start and end times are the same for the left camera and right camera, then the left camera turbulence warp map is equal to the right camera turbulence warp map with the image offset map P(x, y) applied, in accordance with Equation (1) below:

WtbL (x, y, t) = WtbR + Px, y y,t] (1)

In capturing step 530, second frame images IL(x, y, t 2 ) and I (x, yt 2 ) are captured with the stereo camera at time t2, where IL (x, y, t 2 ) is the left camera image and IR (x, y, t 2 ) is the right camera image. The second frame images IL (x, y, t 2 ) and IR (x, y, t 2 ) are captured under execution of the processor 1305 and may be stored in the storage module 1309. The exposures are synchronised between the left and right camera, such that the start of the capture time t2 and the exposure duration At is the same for both cameras. For example, capture of the first frame starts at t2= 40 milliseconds, and the exposure duration is At = 20 milliseconds. Alternatively, the exposure duration of the second frame can be different from the exposure duration of the first frame, as long as the exposure duration is the same for both cameras for each captured frame. The cameras are mounted on a rigid platform, such as a tripod. If there is platform vibration which causes global image shift between frames, such as tripod movements due to windy conditions at the camera site, then the captured images are stabilised using optical stabilisation during capture, or using multi-frame image stabilisation after capture. The stabilised images are then used for subsequent processing.

The turbulence warp map for the second frame left camera image IL (x, y, t 2 ) is WtbL (x, yt 2 )

and the turbulence warp map for the second frame right camera image I (x, yt 2 ) is

WtbR(x,Yt2) If objects in the scene are moving, then the effect of object motion on the captured image frames may be described using an object motion warp map Wom(X, y, t1 , t 2 ), where the 2 values at each co-ordinate in the object motion warp map are the image shifts in the x and y direction caused by object motion but not including image shifts caused by turbulence.

The object motion warp map for the left camera image is WmL (X, y, t 1 , t 2 ), and the object motion warp map for the right camera image is WmR (x, y, t1 , t 2 ).

The temporal changes apparent image positions of objects in the stereo camera images may be described using a left camera temporal warp map WTL and right camera temporal warp map WTR.

The temporal warp maps include the effects of turbulence and image motion in accordance with Equations (2) and (3), below:

WTLft(x,y,tl),IL(x,y,t 2 )J= WtbL(X,Y,t2) WtbL(x,Y,tl) +Wom(X,Yt 1 ,t 2 ),

(2)

WTR UR (X, y, t1 ), IR (x, y, t 2 ))= WtbR xy, t2)- WtbR (X, y, t1 ) + Wom (X, y, t1 , t2 ). (3)

In determining step 540, the temporal image shift for each camera across the two (2) captured frames is determined. The temporal image shifts are determined in the form of temporal warp maps for the left camera WTL and the right camera WTR, in accordance with Equations (4) and (5), below:

WTL(x,y,t 1 ,t 2 ) = WT{I(x,y,t1),IL(x,y,t 2 )J, (4)

WTR (X,y,t1 , t2 ) = WTR R (X, y, t1 ), 'R (X, y, t 2 )). (5)

The temporal warp maps are determined by comparing local regions between the captured image from the first frame and the second frame, where the image shift is measured for each local region. For example, the local image shift may be measured using cross-correlation between an image patch at co-ordinate (x, y) from the first frame, and an image patch at the same (x, y) co ordinate in the second frame.

At determining step 550, the object velocity is determined using the image offset, under execution of the processor 1305, in accordance with Equation (6). An offset right camera temporal warp map is determined from the right camera temporal warp map by shifting the right camera temporal warp map by the image offset value of the image offset map P(x, y) at each co ordinate (x, y). A relative object velocity map AV(x, y, t 1 , t 2 ) is a discrete derivate of the object velocity with respect to the image offset and with respect to the time step between the first frame and the second frame. The relative object velocity map is determined by subtracting the image shift values of the left camera temporal warp map from the image shift values in the offset right camera temporal warp map, dividing by the image offset P(x, y) and dividing by the time step tz - ti between the first and second frame.

AV (x, y, t1 , t 2 ) = WTRL+P(X,y),y,t,t 2 ]-WTL(X,y,tl,t 2 ) (6) P(x,y)[tz -t1]

where the relative velocity is the difference between the object velocity at a co-ordinate (x, y) and the object velocity at an offset co-ordinate [x + P(x, y), y].

Using Equation (1) for the effect of the image offset on turbulence warp maps, and the Equations (2), (3), (4) and (5) for the temporal warp maps, it can be shown that shifting the right camera temporal warp map using the image offset map causes the turbulence warp maps to cancel out in the numerator of Equation (6) for the relative velocity in accordance with Equation (7), below. Equation (7) assumes that the left camera and the right camera have lens apertures with approximately the same shapes and sizes.

AVN(x, y, t 1, t 2) = WTR [X+ P(x, y), y,t 1 ,t 2 ] - WT L(x,y,t 1,t 2 )

=tWtbR [X+P(,y),y,t2 ] - WtbR [X+P(,y),y,t1 ]+Wom[x+P(x,y),y,t1 ,t2 ])

[WtbL (x, y, t2 ) - WtbL (x, y, t 1 ) + Wom(X, Y, t 1, t2 )] Wom [X+ P ((x,y),y,t 1, t 2 ] - Wom(XY, t1, t2 ),

(7)

where AVN (X, y, t1 , t2 ) is the numerator of the relative velocity in Equation (6). In this way the values of the relative object velocity map AV(x, y, t1 , t2 ) are the relative transverse velocities of objects in the scene between left camera image co-ordinate (x, y) and left camera image co ordinate [x + P(x, y), y]. The relative object velocity map AV has two (2) values for each co ordinate (x, y), with the first value being the relative object velocity in the x direction, and the second value being the relative object velocity in they direction. The relative velocity of objects with small velocities and accelerating objects may be accurately measured using two (2) frames with the method 500, because the relative object velocities are independent of the image distortions caused by atmospheric turbulence.

In a camera with a DAF sensor, the shape of the left camera lens aperture is a semi-circle with the straight edge on the right-hand side, and the shape of the right camera lens aperture is a semi circle with the straight edge on the left-hand side. This means that the lens aperture shapes and sizes are approximately the same. In an alternative arrangement, the main lens aperture is square, which means that the left camera lens aperture and the right camera lens aperture are rectangular with the same shape and size.

In an alternative arrangement, the camera 1301 may be a stereo camera comprising a pair of cameras with circular lens apertures, in which case the lens apertures may be configured to both have the same shape and size. For stereo cameras with circular apertures, the chief ray for each camera lens passes through the centre of the camera lens aperture, and the stereo baseline at each image point is the distance between the centres of the camera lens apertures. For a non-circular aperture, or an aperture affected by vignetting, the chief ray for each image point and object distance may be determined using ray tracing software, or using camera calibration. The stereo baseline for each image point is the distance between the intersection points of the chief rays with each of the corresponding lens apertures.

An object velocity map V(x, y, t 1 , t2 ) is determined at step 550 by horizontal line integration of the relative object velocity map multiplied by the image offset with respect to a relative velocity horizontal co-ordinate x', in accordance with Equation (8), below

V (x, y, t1 , t2 ) f AV (x', y)P(x, y)dx', (8)

where the lower limit of integration is the horizontal co-ordinate of the first pixel of each horizontal line and the upper limit of integration is the horizontal co-ordinate of the object velocity map co-ordinate for which the velocity is being determined. Horizontal line integration is performed using numeric integration for each x co-ordinate in the relative velocity map with the constant of integration set to zero assuming there are no moving objects at the left edge of the image. Alternatively, a vertical scene feature which is marked by the user as non-moving may be used to set the constant of integration for each row in the velocity map. Regularisation of the object velocity map is performed by spatial smoothing using a 2D Gaussian.

At determining step 560, an undistorted image is determined using the object velocity map, under execution of the processor 1305. The undistorted image may be stored in the storage module 1309. The undistorted image compensates for the effects of atmospheric turbulence, including spatio-temporal variation in image shift and blur, while preserving object motion and avoiding artefacts such as motion blur. A method 1000 of determining an undistorted image, as executed at step 560, will be described below with reference to Fig. 10.

Following step 560, the method 500 returns to step 530 and continues for each video frame in a multiple frame video. Each time a new frame is captured, the first frame is replaced by the second frame, and the new frame replaces the second frame. After all the video frames have been processed, the process is complete and the method 500 concludes. The undistorted image sequence may then be viewed by a human operator to identify and characterise moving objects in the scene being captured by the camera 1301. The velocity of the object in the scene may be determined using the undistorted image sequence. In one arrangement, the undistorted image sequence may be used as input to image analysis algorithms for object identification and tracking, where the image analysis algorithms do not require special modification for the distortion effects caused by atmospheric turbulence. The image analysis algorithms may be used for determining the velocity of the object in the scene imaged by the camera 1301.

The method 800 of determining the image offset map P(x, y), as executed at step 510, will now be described with reference to Fig. 8. The method 800 may be implemented as one or more software code modules of the software application program 1333 resident in the storage module 1309 and being controlled in its execution by the processor 1305. The method 800 will be described by way of example with reference to Fig. 3.

The method 800 begins at determining step 810, where the object distance 380 is determined under execution of the processor 1305. The object distance 380 determined at step 810 maybe stored in the storage module 1309. If the scene is all at the same distance and in focus, then the object distance is determined using the focus distance of the stereo camera lenses. Alternatively, if the object distance varies across the scene, then depth estimation methods may be used to determine a spatially varying depth map. The stereo disparity between the left camera image and the right camera image captured using a DAF sensor may be used to determine the axial distance of an object from the best focus plane. Alternatively, the distance of the scene may be set by the user.

In determining step 820, the phase screen distance 360 is determined under execution of the processor 1305. The phase screen distance 360 determined at step 360 may be stored in the storage module 1309. Depending on atmospheric conditions, the turbulence may be approximated using a single phase screen. The phase screen distance may be determined at step 820 using atmospheric turbulence characterisation methods. One such method of characterisation is to use SLOpe Detection And Ranging (SLODAR). A SLODAR device uses a Shack-Hartmann wavefront sensor to measure the wavefront error caused by turbulence for two (2) scene points. The wavefront errors are cross-correlated to determine the distance to the phase screen. If the cross-correlation produces a single peak, then a single phase screen is a good approximation for the atmospheric turbulence. Another method of determining the phase screen distance is to use known atmospheric and topographical conditions. For example, if the path between the camera 1301 and the scene passes over the top of a ridge, then a phase screen at the distance of the ridge may be a good approximation, and a user may input the distance to the ridge as the phase screen distance.

In determining step 830, the image offset map P(x, y) is determined using the ray tracing geometry shown in Fig. 4. For the purposes of illustration, Fig.4 is shown for scene points in the plane y = 0. However, an image offset value may be determined for scene points outside the plane y = 0 using the same principles but with 3D geometry. An image point AL atco-ordinate (x, y) on left image sensor 480 corresponds to scene point A. A left camera chief ray for image point AL 470 is a line from scene point A through the centre of the left camera lens aperture to image point AL. The left camera chief ray intersects with phase screen 450 at phase screen point ALP . Scene point B is determined by tracing a right camera chief ray 475 from the centre of the right camera lens aperture to the phase screen point ALP. The horizontal distance from the centre of left camera lens aperture 440 to the centre of right camera lens aperture 445 is stereo camera baseline 270. Scene point B may be found at the intersection between the right camera chief ray and object plane 410, and image point BR corresponding to scene point B may be found at the intersection between the right camera chief ray and right image sensor 485. The image point AR may be determined using a second right camera chief ray 478, which is a line from scene point A through the centre of the right camera lens aperture. The intersection of the second right camera chief ray with the right image sensor 485 is the image point AR. The horizontal distance from image point AR to image point BR is the image offset 260 as represented by the image offset value in the image offset map P(x, y) at the co-ordinate (x, y).

For example, for the geometry shown in Fig.4 with scene points in the plane y = 0, then the image offset value at the co-ordinate (x, y) is determined in accordance with Equation (9), below:

P(x,y)=b zi zi), (9) zy Zo;

where b is the length of the stereo camera baseline, z; is the axial distance from the lens to the sensor, zP is the axial distance from the lens to the phase screen, and z. is the axial distance from the lens to the object.

The stereo camera baseline 270 may be set and calibrated during manufacture of the stereo camera 1301. Alternatively, the stereo camera baseline may be calibrated by capturing stereo images of a test chart including a known physical length scale. The stereo camera baseline for a camera with a DAF sensor is half the diameter of the main lens aperture, which changes depending on the relative aperture lens setting used during image capture.

In an alternative arrangement, the stereo camera baseline has a non-horizontal orientation. For example, two (2) cameras may be mounted with a vertical or otherwise rotated baseline, or a DAF sensor camera may be operated in a portrait or otherwise rotated orientation, or a DAF sensor may have sub-pixels arranged with a vertical orientation. In the case of a different stereo camera baseline, the distance representing the image offset will be parallel to the baseline, and steps in the described methods which depend on the baseline orientation may need to be performed in ways consistent with the baseline orientation using any suitable method. For example, in the method 800, geometrical calculations may need to account for the baseline orientation. As another example, the step for creating an offset temporal warp map and the step for ID integration of the relative object velocity in step to determine the object velocity will both need to account for the baseline orientation.

In an alternative arrangement, the stereo camera 1301 may consist of four (4) cameras with both vertical and horizontal baselines, including a left camera, a right camera, a top camera, and a bottom camera. Using a four (4) camera system, the relative object velocity may be determined with both horizontal and vertical image offsets. Further, using a four (4) camera system, 2D integration may be used to determine the object velocity, which may be performed using spatial domain integration. Alternatively, 2D integration may be performed in the spatial frequency domain by representing the relative velocity vectors as complex numbers with the horizontal component forming the real part and the vertical component forming the imaginary part. The complex relative velocity map is Fourier transformed, multiplied by a spiral phase operator, then inverse Fourier transformed to produce the object velocity map. Boundary artefacts may be reduced by mirror reflection.

The image offset (e.g., 260) may vary over the image. For example, the object distance may vary or the phase screen distance may vary. The phase screen distance may vary over the image if the imaging path in one part of the scene passes over a ridge, while in another part of the scene the closest point to the ground along the imaging path is near the camera 1301. For example, in Fig.2 if a second phase screen 255 is at a different distance from the first phase screen 250, then for a scene point A viewed through the left camera through the second phase screen 255 forming an image at image point AL, then the image offset is the distance between the right camera image point AR and the right camera image point CR, where CR is the image point corresponding to scene point C, and the chief ray from scene point A to image point AL intersects with the chief ray from scene point C to image point CR, and the intersection point is at the phase screen distance corresponding to the second phase screen 255. In this example of a spatially varying phase screen distance, the scene point A viewed through the second phase screen 255 is a different scene point from the scene point A viewed through the first phase screen 250.

A spatially varying image offset may be determined using a spatially varying phase screen distance map by determining a spatially varying object distance map, and then for each image co-ordinate (x, y) determining an image offset P(x, y).

Fig. 9 is a flow diagram showing a method 900 of determining a stereo camera image offset. The method 900 may be used for determining the image offset at step 510 using captured stereo images from the camera 1301 where the camera 1301 includes a dual-pixel auto-focus (DAF) image sensor. The method 900 may use the same DAF sensor camera as is used for the main image capture. Further, specialised and expensive wavefront sensing equipment is not required in the method 900.

The method 900 may be implemented as one or more software code modules of the software application program 1333 resident in the storage module 1309 and being controlled in its execution by the processor 1305.

In capturing step 910, at least two (2) stereo image frames are captured using the camera 1301 under execution of the processor 1305. For example, a sequence of ten (10) frames may be captured with both the left camera and the right camera. The camera parameters, exposure settings and exposure timings are synchronised between the left camera and the right camera of the camera 1301. In step 920, temporal warp maps WTL (x, y, t1 , t 2 ) and WTR (x, y, t1 , t 2 ) are calculated for the left camera and the right camera for each frame after the first frame. For example, from ten (10) captured frames, a sequence of 9 temporal warp maps is calculated for each camera. The image frames captured at step 910 may be stored in the storage module 1309.

In determining step 930, a cross-warp map WC(x, y, t 1 , t 2 ) is determined for each frame after the first frame, under execution of the processor 1305. Due to random spatial variation in turbulence, the temporal warp maps WTL and WTR also have random spatial variation. If the atmospheric turbulence can be approximated using a single phase screen and there is no object motion, then it can be shown using equations (1), (2) and (3) that the left temporal warp map is equal to the right temporal warp map shifted in the horizontal direction by the value of the image offset, in accordance with Equation (10), below:

WTL(x, y,t1,t 2 ) = WTR [x+ P(x, y), y, t 1 , t 2 (10)

The temporal warp maps have the same random spatial variation except for a horizontal image shift, which means there is a strong correlation signal between the temporal warp maps. The cross-warp map WC (x, y, t 1, t 2 ) is determined for each co-ordinate (x, y) and for each frame captured at ti and t 2 by determining a spatially varying warp map between the left temporal warp map and the right temporal warp map, in accordance with Equation (11), below:

WC (X, y, t 1 , t 2 ) = W [WTL (x, y, t 1 , t2 ),WTR (X, y, t1,t2 ), (11)

where W(1 1 , 2) represents determination of the warp map from image 1i to image 12. As the temporal warp maps WTL and WTR consist of a 2-dimensional (2D) array of values, warp map determination methods which are normally applied to a pair of images may also be applied to a pair of warp maps. The warp maps have two (2) values (i.e., horizontal and vertical shifts) at each image co-ordinate (x, y), which may be configured as two (2) channels (red and green), with the third channel (blue) set to zero (0). The resulting 3-channel image may then be used as input to a warp map determination method which usually accepts a colour image with three (3) channels (i.e., red, green and blue). Alternatively, using a greyscale warp map determination method, a cross-warp map may be determined separately for the horizontal image shifts and the vertical image shifts, and then the two (2) cross-warp maps may be averaged to create a single cross-warp map.

Alternatively, using a warp map determination method which accepts complex numbers as input such as patch-based cross-correlation, the horizontal image shifts may be set as the real part and the vertical image shifts may be set as the imaginary part. The resulting complex images may be used as input to a warp map determination method to produce the cross-warp.

The cross-warp is determined at step 920 for each sequential pair of captured frames. For example, nine (9) cross-warp maps may be determined from ten (10) captured frames. The values of the cross-warp maps are then averaged across the frames at each co-ordinate (x, y) to form an average cross-warp map Wc (x,y).

In determining step 940, an image offset map P(x, y) is determined from the average cross-warp map. The value of the image offset for each image co-ordinate (x, y) is determined as the horizontal shift value at the same co-ordinate (x, y) in the cross-warp map.

In a long distance surveillance application, the value of the image offset is expected to vary slowly in comparison to the time between capture frames, because the value of the image offset depends on the object distance and phase screen distance which are expected to vary slowly. For example, phase screen distances may vary as the temperature changes over the course of a day in different parts of the ground below the imaging path between the camera 1301 and the scene, and so it may be sufficient to determine the image offset once per hour. Frame to frame capture times may be expected to be less than one (1) second in order to monitor a changing scene. For example, the frame to frame capture time may be forty (40) milliseconds for a frame speed of twenty-five (25) frames per second. The relatively slow variation in the value of the image offset also enables improving the accuracy of determining the image offset by averaging the cross-warp map over multiple frames.

At step 930, it is assumed that there is no object motion in the scene. The method 900 begins after the user checks the scene visually to ensure there is no object motion, and then inputs a turbulence calibration command. Alternatively, the temporal warp maps determined at step 920 may be averaged over multiple frames (e.g., ten (10) frames), and a small threshold may be applied to the image shift values in the averaged temporal warp map to determine if there is any object motion. By averaging the temporal warp over multiple frames, even slow moving objects may be detected. If object motion is not detected, then the method 900 proceeds to step 930. If object motion is detected at step 930, then the method 900 returns to step 910.

In one arrangement, the method 900 is used to determine the image offset for each captured frame after the first frame, for use in step 510.

In an alternative arrangement, the method 900 is used with an initial set of captured frames, and the image offset is stored in the storage module 1309. The method 500 may then be used with a following set of frames using the image offset from the storage module 1309 for step 510.

Image shift determination is used by the method 500 at step 540. Image shift determination is also used in the method 900 at steps 920 and 930. As described below, image shift determination is also used in method 1000 at steps 1020 and 1065, in method 1100 at steps 1130 and 1140, and in method 1200 at steps 1230 and 1240.

In one arrangement, image shift determination may be used to determine a warp map using covariance-based mutual information (CMI) warp map determination using a first input image and a second input image, where the warp map is an array of 2D vectors. In such a warp map, each vector describes the shift for a pixel from the first input image to the second input image.

The warp map may be determined using CMI by first creating an initial warp map. An initial CMI warp map is the identity mapping consisting of a set of (0, 0) vectors. Alternatively, where the input images are from different optical configurations, the initial CMI warp map may be determined using a disparity map estimated using multiple frames captured when there are no moving objects in the scene. CMI warp map determination then proceeds by assigning colour labels to each pixel in the input images, using colour clustering. The colour information of each input image is quantised independently into two hundred and fifty-six (256) colour clusters (e.g., by using the k-means algorithm). Each colour cluster is represented by a colour label (e.g., as a unique integer per colour cluster in that input image). A first pixel is selected in the first input image, and a second pixel is determined in the second input image by using the initial warp map. A set of third pixels is selected from the second input image, using a 3 x3 neighbourhood around the second pixel.

A covariance score is determined for each pixel in the set of third pixels, which determines the statistical dependence between the label of the first pixel and the labels of each of the third pixels. A covariance score (Cg,) for labels (ai, bj) is determined using the marginal and joint histograms determined using partial volume interpolation. According to partial volume interpolation, histograms involving the transformed second input image are instead determined by first transforming pixel positions (i.e., integer-valued coordinates) of the second input image onto the coordinate space of the first input image using the mapping. Then the label associated with each pixel of the second input image is copied using a spatial distribution to the pixel positions surrounding the associated transformed coordinate (i.e. in the coordinate space of the first image). The spatial distribution is controlled by a kernel of weights that sum to one (1), centred on the transformed coordinate. For example, a trilinear interpolation kernel or other spatial distribution kernels may be used to control the spatial distribution. Then histograms involving the transformed second image, namely the marginal histogram for the second image and the joint histogram, may instead be determined using the spatially distributed labels.

The covariance score is determined using Equation (12),

C - =,b) (12) 'A P (ai,bj) +P (at)P(b;)+E'

where P(ag, bj) is a joint probability determined for labels at and bj placed at corresponding positions of the first input image and the second input image determined based on the joint histogram of the first and second input images; P(a) is the probability determined for the label ai appearing in the first input image determined based on the marginal histogram of the first input image; and P(bi) is the probability determined for the label bj appearing in the second input image determined based on the histogram of the second input image. - is a regularization term to prevent a division-by-zero error, and may be an extremely small value. Corresponding positions for pixels in the first input image and the second input image may be determined using the initial CMI warp map. In Equation (12), the covariance score is a ratio, where the numerator of the ratio is the joint probability estimate, and the denominator of the ratio is the joint probability estimate added to the product of the marginal probability estimates added to the regularization term.

The covariance score has a value between zero (0) and one (1). The covariance score Cg t akes

on values similar to a probability. When the two labels appear in both input images, but rarely co-occur, Ci, approaches 0, i.e. P(ai, b) « P(a)P(b). C, is 0.5 where the two labels are

statistically independent, i.e. P(a, b) = P(ai)P(b). C 1 approaches 1.0 as the two labels co

occur more often than not, i.e. P(ai, b) » P(a)P(b).

Candidate shift vectors are calculated for each of the third pixels, where each candidate shift vector is the vector from the second pixel to one of the third pixels.

An adjustment shift vector is then calculated using a weighted sum of the candidate shift vectors for each of the third pixels, where the weight for each candidate shift vector is the covariance score for the corresponding third pixel. The adjustment shift vector is used to update the initial displacement field, so that the updated displacement field for the first pixel becomes a more accurate determination of the alignment between the first input image and the second input image. The process is repeated by selecting each first pixel in the first input image, and creating an updated warp map with increased accuracy.

The CMI warp map determination method then determines whether the alignment is completed based upon an estimate of convergence. Examples of suitable convergence completion tests are a predefined maximum iteration number, or a predefined threshold value which halts the iteration when the predefined threshold value is larger than the root-mean-square magnitude of the adjustment shift vectors corresponding to each vector in the warp map. An example threshold value is 0.001 pixels. In some implementations, the predefined maximum iteration number is set to one (1). In one arrangement, however, to achieve accurate registration, the maximum iteration number is set to at least ten (10). For smaller images (e.g. 64x64 pixels) the maximum iteration number may be set to one hundred (100). If the alignment is completed, then the updated warp map becomes the final warp map which is the output from CMI warp map determination.

In an alternative arrangement, warp map determination is performed using patch-based correlation. In the patch-based correlation method, the input images are divided into a set of small overlapping regions called patches. The patches cover the entire input images, and have the same sizes and positions in both input images. For example, the patch size may be 64x64 pixels and the patch positions are spaced thirty-two (32) pixels apart horizontally and vertically. A patch from the first input image is cross-correlated with the corresponding patch from the second input image, and the position offset of the peak relative to the centre of the patch in the cross-correlation output image is used to determine the image shift between the first input patch and the second input patch. Alternatively, phase correlation may be used instead of cross correlation. A window function, such as a Hann window, may be applied to both input patches before correlation to reduce boundary effects. The 2D image shift vector of the warp map at the position of the input patches is assigned to the value of the correlation peak offset. The correlation is repeated for all of the patches producing a warp map which is the same size as the number of patches. The warp map is then interpolated to produce an image shift vector for each pixel in the first input image.

In an alternative arrangement, optic flow is used to determine the warp map. Optic flow uses iterative gradient based shift estimation to determine a 2D image shift vector from the first input image to the second input image for each pixel in the first input image.

The method 1000 of determining an undistorted image as executed at step 560 of the method 500. The method 1000 may be implemented as one or more software code modules of the software application program 1333 resident in the storage module 1309 and being controlled in its execution by the processor 1305.

The method 1000 begins at segmenting step 1010, where background regions in the left camera image for the current frame are segmented under execution of the processor 1305. The segmented background regions may be stored in the storage module 1309.

A threshold on the object velocity map for each frame is used at step 1010 to segment background image regions which have no object motion in that frame, and foreground image regions which do have object motion in that frame. The threshold may be set to a small number to exclude false positives caused by noise in the object velocity map. For example, the threshold may be set to 0.05 pixels/frame. The boundaries of the foreground and background regions are pixel-accurate because the boundaries are based on the object velocity map which contains an accurate velocity for each pixel.

In determining step 1015, a left camera reference image is determined by determining the pixel wise median of the left camera images from multiple frames during a period when there is no motion in the scene. For example, a sequence of ten (10) frames may be selected in which the object velocity map is below a threshold of 0.05 pixels/frame. Alternatively, a time period with no scene motion may be indicated by the user. The image shift distortion caused by turbulence averages to zero over multiple frames. The reference frame is therefore a determination of the undistorted background image, and each captured frame may be aligned to the reference frame to give a consistent stabilised image throughout the sequence of frames in a video.

In determining step 1020, a temporal reference warp map is determined for the current frame by performing image shift determination between the left camera image from the current frame and the left camera reference image.

In determining step 1025, undistorted background image regions are determined for the current frame under execution of the processor 1305. The undistorted image regions are determined by determining the negative of the image shift values in the temporal reference warp map and then using the negative image shift values to apply a spatially varying warp to the background regions. The undistorted left camera background image regions from multiple frames are combined to increase the image contrast and reduce noise. In addition, image sharpening may be applied to reduce blur caused by turbulence. The undistorted background image regions determined at step 1025 may be stored in the storage module 1309.

In segmenting step 1050, the foreground regions in the left camera image for the current frame are segmented under execution of the processor 1305. The segmented foreground regions may be stored in the storage module 1309.

In stabilising step 1055, object motion is stabilised in the foreground regions using an object velocity map 1057, creating motion stabilised foreground image regions. In determining step 1060, for each motion stabilised foreground image region, a foreground reference image region is determined by determining the pixel-wise median over the corresponding motion stabilised foreground image regions from multiple frames. In determining step 1065, a foreground region reference temporal warp map is created for each foreground region in the current frame by determining the image shift between the motion stabilised foreground image region and the foreground reference image region.

In determining step 1070, undistorted foreground image regions are determined under execution of the processor 1305. The undistorted foreground image regions are determined by determining the negative of the image shift values in the foreground region temporal reference temporal warp maps and then using the negative image shift values to apply a spatially varying warp to the motion stabilised foreground image regions. In addition, image sharpening may be applied to reduce blur caused by turbulence.

In reapplying step 1075, the object motion is reapplied to each undistorted foreground image region using the object velocity map 1057 creating undistorted moving foreground image regions. Then in compositing step 1095, the undistorted moving foreground image regions are composited into the undistorted background image, under execution of the processor 1305, using the foreground segmentation, to create an undistorted left camera image for the current frame. The undistorted left camera images have reduced spatio-temporal variations in image shift and blur, while preserving object motion and avoiding artefacts such as motion blur and moving object boundary artefacts.

In an alternative arrangement, the right camera image is used in method 1000 to determine an undistorted image instead of the left camera image, in which case the relative object velocity map determined in step 550 by subtracting the image shift values of right camera temporal warp map from the image shift values in an offset left camera temporal warp map.

In an alternative arrangement, an undistorted main image is formed by determining a left camera undistorted image using the method 1000 and determining a right camera undistorted image using method 1000, and then pixel-wise adding the left camera undistorted image and the right camera undistorted image to create a main undistorted image.

In an alternative arrangement, the method 1000 is applied to a main image frame. The main image frame is the sum of the left camera image for the current frame and the right camera image for the current frame. When using a camera with a DAF sensor, the main camera image formed in this way is equivalent to an image formed with the whole main camera lens aperture. If the scene is at the best focus plane of the main camera lens, then the disparity between the left camera image and the right camera image is zero (0) pixels, and the object velocity map may be used directly in steps 1055 and 1075 to stabilise object motion and reapply object motion in foreground regions of the main camera image. If there are objects in the scene which are not at best focus, then the disparity may be used to adjust the co-ordinates between the left camera image, the right camera image and the main camera image, and the corresponding left and right camera warp maps and the left and right camera object velocity maps. For a scene point A, main image point Am of scene point A will be halfway between left camera image point AL and right camera image point AR. If there are no objects moving along the optical axis, then a spatially varying disparity map may be estimated from the average over multiple frames of the image shift between the left camera image and the right camera image. The distance between image point AL and image point AR is given by the value of the disparity map at image point AL.

Fig. 11 is a flow diagram showing a method 1100 of determining velocity of an object moving in a scene being imaged by the camera 1301, using a turbulence phase screen distance. The method 1100 may be implemented as one or more software code modules of the software application program 1333 resident in the storage module 1309 and being controlled in its execution by the processor 1305.

The method 1100 begins at capturing step 1110, where frame images are captured, under execution of the processor 1305, resulting in a first frame first image 1112 captured using a first optical configuration, and a first frame second image 1114 captured using a second optical configuration. In one arrangement, the first optical configuration is the left camera image from the camera 1301 with a DAF sensor, and the second optical configuration is the right camera image from the camera 1301 using a DAF sensor.

In capturing step 1120, second frame images are captured a short time after step 1110, under execution of the processor 1305, resulting in a second frame first image 1122 captured using the first optical configuration, and a second frame second image 1124 captured using the second optical configuration. Images captured in each frame are captured with the same camera exposure and lens settings on each optical configuration, and the start and end of the frame exposures are synchronised across both optical configurations.

In determining step 1130, a first image temporal warp map is determined using CMI warp map determination using the first frame first image 1112 and the second frame first image 1122. In determining step 1140, a second image temporal warp map is determined using CMI warp map determination using the first frame second image 1114 and the second frame second image 1124. The temporal warp maps contain estimates of the image shifts at each pixel, where the image shift is the sum of an image shift distortion caused by a turbulence phase screen between the camera 1301 and the object being imaged and an image shift caused by motion of the object being imaged.

In determining step 1150, the object velocity is determined according to the method previously described for step 550, where the velocity is determined by compensating for the image shift distortion caused by the turbulence phase screen. The turbulence image shift distortion is determined using the first temporal warp map, the distance between the camera 1301 and the turbulence phase screen 1155, and the second temporal warp map. In one arrangement, the second temporal warp map is offset by an image offset value determined using the phase screen distance 1155 with the method 800. In an alternative arrangement, the image offset may be provided by the user.

A first image shift at a pixel co-ordinate in thefirst temporal warp map corresponds to a second image shift in the offset second temporal warp map at the same pixel co-ordinate, such that the first image shift includes an image shift distortion caused by a region in the turbulence phase screen, and the second image shift includes an image shift distortion caused by the same region in the turbulence phase screen. A relative object velocity may be determined by subtracting the first image shift from the second image shift, such that the image shift distortion caused by the turbulence phase screen cancels out and the remaining image shift is the image shift caused by the relative motion of the object being imaged. The motion is relative between the pixel co ordinate and the offset pixel co-ordinate. The object velocity of the object being imaged is then estimated by ID integration.

Fig. 12 is a flow diagram showing a method 1200 of determining velocity of an object moving in a scene being imaged by the camera 1301, using a pixel position offset. The method 1200 may be implemented as one or more software code modules of the software application program 1333 resident in the storage module 1309 and being controlled in its execution by the processor 1305.

The method 1200 begins at capturing step 1210, where frame images are captured, under execution of the processor 1305, resulting in a first frame first image 1212 captured using a first optical configuration, and a first frame second image 1214 captured using a second optical configuration. In one arrangement, the first optical configuration is the left camera image from the camera 1301 with a DAF sensor, and the second optical configuration is the right camera image from the camera 1301 using a DAF sensor.

In capturing step 1220, second frame images are captured a short time after step 1210, under execution of the processor 1305, resulting in a second frame first image 1222 captured using the first optical configuration, and a second frame second image 1224 captured using the second optical configuration. Images captured in each frame are captured with the same camera exposure and lens settings on each optical configuration, and the start and end of the frame exposures are synchronised across both optical configurations.

In determining step 1230, a first image temporal warp map is determined using CMI warp map determination using the first frame first image 1212 and the second frame first image 1222. In determining step 1240, a second image temporal warp map is determined using CMI warp map determination using the first frame second image 1214 and the second frame second image 1224. The temporal warp maps contain estimates of the image shifts at each pixel, where the image shift is the sum of an image shift distortion caused by a turbulence phase screen between the camera 1301 and the object being imaged and an image shift caused by motion of the object being imaged.

In determining step 1250, the object velocity is determined according to the method previously described for step 550, where the velocity is determined by compensating for the image shift distortion caused by the turbulence phase screen. The turbulence image shift distortion is determined using the first temporal warp map at a first pixel position and the second temporal warp map at a second pixel position, where the second pixel position is related to the first pixel position by a pixel position offset 1255 determined according to the method 900. After application of the pixel position offset, a first shift in the first temporal warp map at afirst pixel position corresponds to a second shift in the second temporal warp map at a second pixel position, where the second pixel position in the second image is subject to the same region of the turbulence phase screen as the first pixel position in the first image.

A relative object velocity is determined by subtracting the first image at the first pixel position from the second image shift at the second pixel position, such that the image shift distortion caused by the turbulence phase screen cancels out and the remaining image shift is the image shift caused by the relative motion of the object being imaged. The motion is relative between the first pixel position and the second pixel position. The object velocity of the object being imaged is then estimated by ID integration.

Industrial Applicability

The arrangements described are applicable to the computer and data processing industries and particularly for image processing.

The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including", and not "consisting only of'. Variations of the word "comprising", such as "comprise"and "comprises" have correspondingly varied meanings.

Claims

1. A method of determining velocity of an object in a scene imaged by an imaging device, the method comprising:

capturing a first frame and a second frame using the imaging device, each frame being associated with a first image captured using a first optical configuration and a second image captured using a second optical configuration;

determining a first shift for a first pixel position between the first image associated with the first frame and the first image associated with the second frame; and

determining velocity of the object moving in the scene by compensating for distortion caused by a turbulence phase screen based on the first shift, a distance from the imaging device to the turbulence phase screen and a second shift affected by the turbulence phase screen, wherein the second shift is determined from the second image associated with the first frame and the second image associated with the second frame, and wherein the second shift is subject to the turbulence phase screen.

2. The method of claim 1, wherein the frames are captured using a dual-pixel auto-focus image sensor.

3. The method of claim 1, wherein the imaging device comprises four cameras with at least two baselines in different orientations.

4. The method of claim 1, further comprising determining a cross-warp for the second frame.

5. The method of claim 1, further comprising determining a cross-warp for the captured frames and averaging the cross-warp over the frames.

6. The method of claim 1, further comprising determining a cross-warp for a plurality of captured frames and averaging the cross-warp over the plurality of captured frames.

7. The method of claim 1, further comprising segmenting foreground regions of the frames containing moving objects.

8. The method of claim 7, further comprising using the determined velocity to stabilise object motion in at least one foreground region; reapplying object motion to the at least one foreground region to create at least one undistorted foreground region; and compositing the undistorted foreground region onto a background region to create an undistorted image.

9. The method of claim 1, further comprising determining an undistorted image for at least one of said frames using the determined velocity.

10. An apparatus for determining velocity of an object in a scene imaged by an imaging device, the apparatus comprising:

11. A system for determining velocity of an object in a scene imaged by an imaging device, the system comprising:

a memory for storing data and a computer program;

a processor coupled to the memory for executing the computer program, the computer program comprising instructions for: capturing a first frame and a second frame using the imaging device, each frame being associated with a first image captured using a first optical configuration and a second image captured using a second optical configuration; determining a first shift for a first pixel position between the first image associated with the first frame and the first image associated with the second frame; and determining velocity of the object moving in the scene by compensating for distortion caused by a turbulence phase screen based on the first shift, a distance from the imaging device to the turbulence phase screen and a second shift affected by the turbulence phase screen, wherein the second shift is determined from the second image associated with the first frame and the second image associated with the second frame, wherein the second shift is subject to the turbulence phase screen.

12. A computer readable medium having a computer program stored on the medium for determining velocity of an object in a scene imaged by an imaging device, the program comprising:

code for determining a first shift for a first pixel position between the first image associated with the first frame and the first image associated with the second frame; and

13. A method of determining velocity of an object in a scene, the method comprising: capturing a first frame and a second frame, each frame being associated with afirst image captured using a first optical configuration and a second image captured using a second optical configuration; determining a first shift for a first pixel position between the first image associated with the first frame and the first image associated with the second frame; determining a second pixel position in the second image of the first frame and the second image of the second frame which is subject to the turbulence phase screen as the first pixel position; determining a second shift at the second pixel position between the second image of the first frame and the second image of the second frame; and determining velocity of the object moving in the scene based on comparison of the first shift and the second shift.

14. The method of claim 13, wherein the frames are captured using a dual-pixel auto-focus image sensor.

15. The method of claim 13, wherein the imaging device comprises four stereo cameras.

16. The method of claim 13, further comprising determining a cross-warp for the second frame.

17. The method of claim 13, further comprising determining a cross-warp for the captured frames and averaging the cross-warp over the frames.

18. The method of claim 13, further comprising determining an undistorted image for at least one of said frames using the determined velocity.

19. The method of claim 13, further comprising segmenting foreground regions of the frames containing moving objects.

20. The method of claim 19, further comprising reapplying object motion to each of the foreground regions using the determined velocity to create at least one undistorted foreground region; and compositing the undistorted foreground region onto a background region to create an undistorted image.

21. An apparatus for determining velocity of an object in a scene, the apparatus comprising:

means for determining a first shift for a first pixel position between the first image associated with the first frame and the first image associated with the second frame;

means for determining a second pixel position in the second image of the first frame and the second image of the second frame which is subject to the turbulence phase screen as the first pixel position;

means for determining a second shift at the second pixel position between the second image of the first frame and the second image of the second frame; and

means for determining velocity of the object moving in the scene based on comparison of the first shift and the second shift.

22. A system for determining velocity of an object in a scene, the system comprising:

a memory for storing data and a computer program;

capturing a first frame and a second frame, each frame being associated with a first image captured using a first optical configuration and a second image captured using a second optical configuration; determining a first shift for a first pixel position between the first image associated with the first frame and the first image associated with the second frame; determining a second pixel position in the second image of the first frame and the second image of the second frame which is subject to the turbulence phase screen as the first pixel position; determining a second shift at the second pixel position between the second image of the first frame and the second image of the second frame; and determining velocity of the object moving in the scene based on comparison of the first shift and the second shift.

23. A computer readable medium having a computer program stored on the medium for determining velocity of an object in a scene imaged, the program comprising:

code for determining a first shift for a first pixel position between the first image associated with the first frame and the first image associated with the second frame;

code for determining a second shift at the second pixel position between the second image of the first frame and the second image of the second frame; and

code for determining velocity of the object moving in the scene based on comparison of the first shift and the second shift.

24. A method of determining an undistorted image of a scene, said method comprising: capturing a first frame and a second frame, each frame being associated with a first image of the scene captured using a first optical configuration and a second image of the scene captured using a second optical configuration; determining velocity of an object moving in the scene based on comparison of the first image and the second image; segmenting foreground regions of the captured frames containing moving objects using the determined velocity to create at least one undistorted foreground region; and compositing the undistorted foreground region onto a background region of the captured frames to create the undistorted image.

CANON KABUSHIKI KAISHA

Patent Attorneys for the Applicant/Nominated Person

SPRUSON & FERGUSON