GB2565544A

GB2565544A - Image processing

Info

Publication number: GB2565544A
Application number: GB1712974.3A
Authority: GB
Inventors: Pawlik Bartek
Original assignee: Nokia Technologies Oy
Current assignee: Nokia Technologies Oy
Priority date: 2017-08-14
Filing date: 2017-08-14
Publication date: 2019-02-20
Also published as: GB201712974D0

Abstract

A method for inpainting a defect pixel in a first image. First and second images representing a scene captured from different locations are acquired 6.1. A defect pixel in the first image is identified 6.2, and a neighbouring block of pixels adjacent the defect pixel are determined 6.3. A replacement pixel for the defect pixel is then identified based on the similarity of the neighbouring block to candidate blocks of the first and second images 6.4. Preferably a plurality of first candidate blocks all having different positions in the first image and a plurality of second candidate blocks all having different positions in the second image are determined. The similarity of the neighbouring block to these blocks is then determined, and the most similar block is identified as the replacement block (figure 7). A search window may be formed such that the defect pixel is at the centre of it.

Description

Field of the Invention

This invention relates to methods and systems for image processing, particularly methods and systems for correcting defect pixels in digital images.

Background

Digital sensors, and hence the images derived from them, may comprise one or more defect pixels. A defect pixel is, in terms of the sensor, one that operates differently than neighbouring pixels. Defect pixels, in terms of the resulting image, are apparent to a user viewing the displayed image. For example, defect pixels maybe displayed in a different colour to surrounding pixels or pixel regions and therefore stand out from the remainder of the image.

Correcting defect pixels in images can involve significant amounts of computation and/or may not be suitable where there are large numbers of defect pixels.

Summary of the Invention

A first aspect of the invention provides a method comprising: providing first and second images representing a scene captured from different locations, the first and second images each comprising a two-dimensional array of pixels; identifying a defect pixel in the first image; determining a neighbouring block of pixels adjacent the defect pixel; and identifying a replacement pixel for the defect pixel based on the similarity of the neighbouring block to candidate blocks of the first and second images.

The method may further comprise determining a plurality of first candidate blocks, each associated with a respective first candidate pixel of the first image, the first candidate pixels having different respective positions in the first image; determining a plurality of second candidate blocks, each associated with a respective second candidate pixel of the second image, the second candidate pixels having different respective positions in the second image corresponding to those of the first candidate pixels; determining the similarity of the neighbouring block to each of the first and second candidate blocks for each position; and identifying the replacement pixel as the candidate pixel of the most-similar candidate block of the first and second candidate blocks.

The similarity may be determined, by determining, for each of the plurality of first and second candidate blocks, a similarity value, and identifying the most-similar candidate block by comparing the similarity values.

The similarity value may be generated using a sum of differences between the neighbouring block and the first and second candidate blocks.

The similarity value maybe generated using a sum of absolute differences (SAD.)

The similarity value maybe generated using a sum of squared differences (SSD.)

The similarity value may be generated using the Li-norm and/or L2-norm algorithm.

The first and second images may be colour images, and wherein the plurality of first and second candidate blocks maybe associated with first and second candidate pixels having the same colour component as the defect pixel.

Each pixel in the pixel array may be represented as either a red (R), Green (G) or Blue (B) colour component.

The first and second candidate blocks may comprise pixels adjacent the first and second candidate pixels respectively.

The first and second candidate blocks may comprise all pixels adjacent the first and second candidate pixels respectively.

The candidate blocks may have the same size and shape as the neighbouring block.

The method may further comprise determining an n x m search window within each of the first and second images, the search window surrounding the identified defect pixel, wherein the candidate blocks may occupy corresponding spatial positions within the search window.

The n x m search window may be determined such that the defect pixel is substantially at the centre of the search window.

The n x m search window may be either a 5 x 5 search window or a 7 x 7 search window.

The neighbouring block may comprise all pixels adjacent to the defect pixel.

The neighbouring block may comprise only a subset of pixels adjacent to the defect pixel.

The neighbouring block may be an L-shaped block.

The L-shaped neighbouring block may comprise three pixels.

The plurality of images may represent substantially simultaneously-captured content.

The method may further comprise receiving the plurality of colour images from separate image sources having a predetermined spatial relationship to one another such that overlapping parts of the images can be identified.

The m x n search window maybe positioned such that it is within the overlapping parts.

The method may further comprise transforming the received colour images by rotation and/or warping.

A second aspect of the invention comprises computer program comprising instructions that when executed by a computer control it to perform the method of any preceding definition.

A third aspect of the invention provides a non-transitory computer-readable medium having stored thereon computer-readable code, which, when executed by at least one processor, causes the at least one processor to perform a method, comprising: providing first and second images representing a scene captured from different locations, the first and second images each comprising a two-dimensional array of pixels; identifying a defect pixel in the first image; determining a neighbouring block of pixels adjacent the defect pixel; and identifying a replacement pixel for the defect pixel based on the similarity of the neighbouring block to candidate blocks of the first and second images.

A fourth aspect of the invention provides an apparatus, the apparatus having at least one processor and at least one memory having computer-readable code stored thereon which when executed controls the at least one processor: to provide first and second images representing a scene captured from different locations, the first and second images each comprising a two-dimensional array of pixels; to identify a defect pixel in the first image; to determine a neighbouring block of pixels adjacent the defect pixel; and to identify a replacement pixel for the defect pixel based on the similarity of the neighbouring block to candidate blocks of the first and second images.

Brief Description of the Drawings

The invention will now be described, by way of non-limiting example, with reference to the drawings, in which:

Figure 1 is a schematic diagram of components of a digital camera according to embodiments of the invention;

Figure 2 is a partial view of an sensor a colour filter array of the Figure 1 digital camera;

Figures 3a - 3c are schematic views of different colour patterns received at photo sensors of the Figure 1 sensor;

Figure 4 is a perspective view of a system comprising a plurality of cameras and an image processing system, according to embodiments;

Figure 5 is a schematic diagram of components of the Figure 4 image processing system, according to embodiments;

Figure 6 is a flow diagram showing processing operations for performing correction of a defect pixel according to embodiments;

Figure 7 is a flow diagram showing processing operations for performing an operation of the Figure 6 method;

Figure 8 is a schematic view of a first image from a first camera, the first image comprising a defect pixel and a neighbouring block of pixels surrounding the defect pixel;

Figure 9 is a schematic view of a second image from a second camera;

Figure 10 is a schematic view of the first image, with candidate blocks indicated;

Figure 11 is a schematic view of the second image, with corresponding candidate blocks indicated;

Figures 12a and 12b are schematic views of the first and second images for indicating the defect correction process using a numerical example;

Figure 13 is a flow diagram showing processing operations for preparing images for the defect correction process;

Figures 14a - 14c are schematic views of first to third images, useful for understanding part of the Figure 13 processing operation;

Figures 15a - 15c are schematic views of the Figure 14 first to third images, useful for understanding another part of the Figure 13 processing operation;

Figures 16a - 16c are schematic views of the Figure 14 first to third images, useful for understanding the determination of search windows according to the Figure 13 processing operation;

Figure 17 is a graphical view showing a provided image comprising singlet defects and a corrected image, resulting from preferred embodiments; and

Figure 18 is a graphical view showing a provided image comprising couplet defects and a corrected image, resulting from preferred embodiments.

Detailed Description of Preferred Embodiments

Embodiments herein relate to methods and systems for correcting defect pixels in a digital image.

Camera sensors are used in camera devices to capture images, which may be still or moving. In some cases, not all pixels in the camera sensor behave the same way; one or more pixels may be stuck to one value and/or one or more pixels may respond faster or slower than others. A defect pixel, in terms of the sensor, is one that operates differently than neighbouring pixels of the sensor. In terms of the resulting image, a defect pixel is one resulting from a defect sensor pixel.

For example, a dead pixel is a type of defect pixel which is permanent. This maybe because no power was received by the corresponding sensor element. Most digital camera sensors have colour filter arrays (CFA), e.g. Bayer filters, in front of them and hence dead pixels do not necessarily show up as black spots, but rather show up with a different colour than the adjacent pixels, e.g. slightly darker. Another type of defect pixel is a hot pixel, which may appear when the camera sensor gets hot. As such, they may appear and disappear over time.

A singlet defect is one where there is no other defect pixel in a defined cluster of pixels, e.g. a 3 x 3 cluster. A couplet defect is one where there are two defect pixels connected, i.e. adjacent, to one another. A defect cluster is one where there are more than two connected defect pixels.

Defect pixels may detected using any known method. For example, one method (a mapped method) comprises a calibration phase in which a series of images are captured using a relatively long exposure time and various analogue gains, both in complete darkness and with uniform illumination. Defects are easily identified below and above a certain threshold. Another method (a dynamic method) uses dynamic thresholds based on the standard deviation and mean value of neighbouring pixels.

Embodiments particularly relate to correcting one or more defect pixels of a colour image. The general approach outlined below however is applicable to all image types, including black and white images.

For example, a colour image maybe received from a digital camera which can be of any suitable type, including a still image camera, a video camera and/or a multicamera system such as Nokia’s OZO camera.

The camera maybe a dedicated camera, fixed or portable, or maybe provided as part of a mobile handset such as a mobile telephone, laptop, or tablet computer. The camera may also form part of more dedicated equipment such as medical imaging equipment or thermal imaging devices.

Cameras typically comprise a solid state sensor for receiving light on a twodimensional surface after the light passes through a lens. The sensor may, for example, be a charge-coupled device (CCD) sensor or a Complementary Metal Oxide Semiconductor (CMOS) sensor, or a hybrid of both. The sensor converts received light into electrical signals that convey information on the light at respective spatial positions of the sensor’s two-dimensional surface, each spatial position corresponding to a picture element, or pixel, of the captured image.

In some embodiments, a colour-separation filter may be provided in front of the sensor. This is because a digitised colour image typically comprises three channels per pixel, each pixel carrying information of a different light wavelength band, e.g. red (R), green (G) and blue (B). Rather than using three separate sensors, preceded by a different colour filter to capture the information for the respective bands, most cameras employ a single sensor with an overlaid colour filter array (CFA). This is more efficient in terms of size and cost. It follows that only one colour is captured at each spatial position. The data produced at the sensor is sometimes referred to as RAW image data.

A commonly-used CFA is the Bayer filter. The Bayer filter employs alternating red and green filters for odd rows and alternating green and blue filters for even rows. There are therefore twice as many green filters as red or blue filters. It is known that human vision is more sensitive to green light. Each pixel location of the sensor is behind a particular filter, and hence the RAW output is an array of pixel values indicating the intensity of one of the filter colours.

The resulting image data, e.g. the RAW data, may comprise defect pixels not present in the object or scene that is imaged. Such defect pixels may be of any type, for example dead pixels, hot pixels and may be singlet, couplet or cluster defects.

A certain number of defect pixels may be allowed in some applications due to production cost constraints. In so-called single camera solutions, the use of averaging, median calculation or edge-directed polynomial approximation to correct defect pixels tends to produce unsatisfactory results if large numbers of defect pixels are present. For example, the results may leave out pixels, or cause a smearing effect in the image. Some methods involve significant computation resources. Some methods require separate logic for different types of defect, e.g. dead or hot pixels, singlets or couplets.

In overview, embodiments relate to identifying a replacement pixel for a defect pixel using information from a plurality of images relating to a common scene, for example generated by respective cameras which maybe directed at an overlapping region of the scene.

The correction algorithms to be described below may be performed on noninterpolated image data, for example the RAW data received via a CFA. The amount of data required to be processed may therefore be smaller. Embodiments herein therefore relate to image processing on the RAW output from an image sensor of a digital camera, which may be of any form, and for any purpose as outlined above.

Referring to Figure 1, a digital camera 1 comprises a lens 3, a CFA 5 and an image sensor 7 disposed behind the CFA relative to the direction of received light (indicated by the arrow 9.) In some embodiments, the lens 3 may not be required or may be provided by an external lens system detachably mounted to the camera 1. In some embodiments, the CFA 5 and sensor 7 may not be aligned with the shown direction of light, in which case a mirror or prism (not shown) may be provided to redirect light towards the filter and sensor, as appropriate.

The camera 1 may further comprise a controller 11, RAM 13, a memory 15, and, optionally, hardware keys 19 and a display 17. The camera 1 may comprise an interface 25, which maybe a data port for connecting the camera via a cable to an external terminal such as a computer, television, printer or a mobile device such as a mobile phone or tablet computer. The interface 25 may be one or more of a USB port, micro-USB port or Firewire port, for example. The interface 25 may be used for transmitting and/or receiving data, e.g. image data to and/or from the external terminal.

The camera 1 may also comprise a radiofrequency receiver and/or transmitter 27 for the wireless reception and/or transmission of data to an external terminal such as a computer, television, printer or a mobile device such as a mobile phone or tablet computer. The camera 1 may also comprise a memory card slot (not shown) for receiving a removable storage device, such as a SD memory card or the like.

The memory 15 may be a non-volatile memory such as read only memory (ROM), a hard disk drive (HDD) or a solid state drive (SSD). The memory 15 stores, amongst other things, an operating system 21 and may store software applications 23. The RAM 13 is used by the controller 11 for the temporary storage of data. The operating system 21 may contain code which, when executed by the controller 11 in conjunction with the RAM 13, controls operation of each of the hardware components of the camera 1.

The controller 11 may take any suitable form. For instance, it may be a microcontroller, plural microcontrollers, a processor, or plural processors.

In some embodiments, the camera 1 may also be associated with external software applications not stored on the camera. These may be applications stored on a remote server device and may run partly or exclusively on the remote server device. These applications maybe termed cloud-hosted applications. The camera 1 maybe in communication with the remote server device in order to utilize the software application stored there.

In some embodiments, the controller 11 may be implemented as a Field Programmable Gate Array (FPGA). An FPGA is an integrated circuit designed to be configured ‘in the field’ by a user after manufacture. FPGAs comprise a matrix of programmable logic blocks and a hierarchy of reconfigurable interconnects that permit the logic blocks to be connected together according to a user’s design. Defining the operation of the FPGA may be by means of a hardware description language (HDL) and a design automation tool which generates a netlist for a subsequent place-and-route stage using the FPGA vendor’s proprietary software.

FPGAs are useful for implementing image processing applications. This is because their structure is able to exploit spatial and temporal parallelism. FPGAs offer particular advantages therefore in real-time image processing, which is otherwise difficult to achieve on more conventional serial processors. FPGAs are often therefore the technology of choice in modern digital cameras.

Figure 2 is a partial view of the CFA 5, shown overlaid on a small, two-dimensional section of the sensor 7 corresponding to an 8x8 array of photo sensors 37. It will be appreciated that both the CFA 5 and sensor 7 will in reality have a much greater area and array size, but similar principles apply. The CFA 5 in the embodiments is a Bayer filter for arranging red, green and blue (RGB) colour filters 31, 33, 35 on the square grid of photo sensors 37. Each individual RGB filter 31, 33, 35 of the mosaic allows light of the corresponding colour wavelength to pass to the underlying photo sensor 37, blocking the other colours.

The arrangement of the RGB filters 31, 33, 35 is as follows. One row comprises alternate green and blue filters 33, 35. The adjacent row comprises alternative red and green filters 31, 33. The pattern then repeats. The filter pattern is therefore 50% green, 25% red and 25% blue, owing to human vision being more sensitive to green light. Effectively, the image is spatially sub-sampled.

Figures 3a - 3c show the resulting patterns received at the photo sensors 37 of the sensor 7. To distinguish between a green colour detected in a row 37 of red photo sensors and a row 39 of blue photo sensors, the convention is to use the label Gb for the former and Gr for the latter.

The photo sensors 37 of the sensor 7 generate an electrical signal responsive to received light; the signal represents the intensity of the light which is then passed to an Analog-to-Digital Converter (ADC) to provide a digital representation of the pixel at the corresponding spatial location. The pixel may be represented in, for example, 8,16, 32 or 64 bits, or even greater, depending on the required bit depth for the particular application. It will therefore be appreciated that image processing is by its nature an intensive process.

The collection of pixels for the entire area of the sensor 7 corresponds to an unprocessed RAW image that may be stored and processed. As each photo sensor 37 captures only one colour per spatial position, demosaicing is required to provide data corresponding to the other two colours. There are a number of known methods for demosaicing, including (at the most simple level) interpolation using the values of nearby pixels.

As already mentioned, one or more of the photo sensors 37, or pixels, may be defective and therefore produce defect pixels when displayed in the resulting image.

The embodiments herein relate to correcting one or more defect pixels in an image using data from a plurality of said images which cover or capture at least part of a common scene or object. The images may be captured substantially simultaneously, although this is not essential.

Figure 4 shows a scenario in which a space, e.g. a room 41, comprises a plurality of cameras 43, 45, 47 which may be of the Figure 1 type mounted at spatially separate locations. The sensors (not shown) of the respective cameras 43, 44, 47 are directed towards a common object 51 in the space such that at least part of the sensors cover an overlapping part of the scene, including the object in this case. In some embodiments, only two cameras maybe provided. In other embodiments, a greater number of cameras may be provided. In some embodiments, one or more of the cameras 43, 44, 47 maybe a multi-camera device, such as Nokia’s OZO device.

An image processing system 53 is provided external to the cameras 43,45, 47. As will be explained below, the image processing system 53 is configured to receive the RAW image data from each of the cameras 43, 45, 47 and to perform correction of one or more defect pixels. The image processing system 53 may additionally perform detection of the one or more defect pixels, which employs known techniques.

The correction process maybe considered ‘collaborative’ in the sense that two or more images are used to correct the one or more defect pixels.

Alternatively, however, the functionality of the image processing system 53 may be provided in one or more of the cameras 43, 45, 47 without the need for a dedicated external system.

The RAW image data from each camera 43, 45, 47 is transmitted to the image processing system 53. If the RAW image data corresponds to video, i.e. as a series of sequentially captured images, the data maybe transmitted in accordance with a predetermined refresh rate, e.g. 25 frames per second. The data transmission maybe by means of cables or wires 55 connected between the interface 25 of each camera 43, 45, 47 and a corresponding interface of the image processing system 53. Alternatively, the data transmission may be by means of a wireless communications protocol.

Referring to Figure 5, the image processing system 53 may comprise a controller 61, RAM 63, a memory 65, and, optionally, hardware keys 67 and a display 69. The image processing system 53 may comprise a network interface 71, which may be a data port for connecting the system to the cameras 43, 45, 47 via a cable. The interface 71 maybe one or more of a USB port, micro-USB port or Firewire port, for example. The interface 71 may be used for transmitting and/or receiving data, e.g. image data to and/or from the external terminal.

The network interface 71 may additionally or alternatively comprise a radiofrequency wireless interface for transmitting and/or receiving the image data using a wireless communications protocol, e.g. WiFi or Bluetooth. An antenna 73 may be provided for this purpose.

The memory 65 may be a non-volatile memory such as read only memory (ROM), a hard disk drive (HDD) or a solid state drive (SSD). The memory 65 stores, amongst other things, an operating system 75 and may store software applications 77. The RAM 63 is used by the controller 61 for the temporary storage of data. The operating system 75 may contain code which, when executed by the controller 61 in conjunction with the RAM 63, controls operation of each of the hardware components of the image processing system 53.

The memory 65 may also store a set of camera alignment data 79 which represents intrinsic and extrinsic properties and characteristics of the cameras 43,45, 47, including their relative spatial positions and other data sufficient to determine the common overlapping area of their respective fields-of-view. The camera alignment data 79 may for example comprise the yaw, roll and pitch angles of the different cameras 43, 45, 47 and their relative orientation angles. The camera alignment data 79 may be derived in a calibration stage.

An example multi-camera calibration model is described in “A software for complete calibration of multicamera systems” Tomas Svoboda, Czech Technical University, Faculty of Electrical Engineering, Center for Machine Perception, http://cmp.felk.cvut.cz/~svoboda/SelfCal/Publ/talk.pdf.

The controller 11 may take any suitable form. For instance, it maybe a microcontroller, plural microcontrollers, a processor, or plural processors.

In some embodiments, the image processing system 53 may also be associated with external software applications not stored on the camera. These may be applications stored on a remote server device and may run partly or exclusively on the remote server device. These applications maybe termed cloud-hosted applications. The camera 1 may be in communication with the remote server device in order to utilize the software application stored there.

In some embodiments, the controller 11 may be implemented as a Field Programmable Gate Array (FPGA). As mentioned above, FPGAs are useful for implementing image processing applications of the sort described herein. This is because their structure is able to exploit spatial and temporal parallelism. FPGAs offer particular advantages therefore in real-time image processing, which is otherwise difficult to achieve on more conventional serial processors. FPGAs are often therefore the technology of choice in modern digital cameras.

In the following, therefore, it should be appreciated that the same functionality is realizable using alternative integrated circuit methods, such as using FPGAs or Application Specific Integrated Circuits (ASIC). In such embodiments, the software functionality described herein maybe implemented by the FPGA or ASIC rather than using program code.

Figure 6 is a flow diagram indicating operations performed by a software application 77 for correcting one or more defect pixels.

A first operation 6.1 comprises receiving a plurality of images, in this case first and second images, from a plurality of cameras. In some embodiments, the images are captured substantially simultaneously, although this is not essential.

A subsequent operation 6.2 comprises identifying in each of the images a group of pixels representing the same or similar content. For example, in the Figure 4 scenario, the same or similar content maybe pixels corresponding to the object 51 and possibly including surrounding regions. In more complex scenarios, the background may also comprise same or similar content between the images, there may be further objects and so on.

The result will be a group of pixels identified in each captured image so that there is a correspondence between the groups (in the different images) in terms of constituent pixels, albeit from different capture positions. For example, the corresponding groups maybe associated with a common colour component.

A more detailed explanation of the identifying operation 6.2 will be described later on.

A subsequent operation 6.3 comprises determining a neighbouring block of pixels, adjacent an identified defect pixel.

A subsequent operation 6.4 comprises identifying a replacement pixel for the defect pixel based on the similarity of the neighbouring block to candidate blocks of the first and second images.

A subsequent operation 6.5 may comprise replacing, or substituting, the defect pixel with the identified replacement pixel.

If there are further defect pixels, a subsequent operation may comprise repeating operations 6.2 to 6.5 for each further defect pixel.

Referring to Figure 7, example operations that may comprise the identification operation 6.4 are shown.

A first operation 7.1 may comprise determining first candidate blocks associated with respective first candidate pixels. The first candidate blocks and their associated first candidate pixels maybe in a first image. Each first candidate block and their associated first candidate pixel maybe associated with a different position within the first image.

A subsequent operation 7.2 may comprise determining corresponding second candidate blocks associated with respective second candidate pixels. The second candidate blocks and their associated second candidate pixels maybe in a different, second image. Each second candidate block and their associated second candidate pixel may be associated with a different position within the second image.

Where more than two cameras are present, third candidate blocks associated with respective third candidate pixels may be determined, and so on.

A subsequent operation 7.3 may comprise determining the similarity of the neighbouring block (from operation 6.3) to each of the corresponding first and second candidate blocks.

A subsequent operation 7.4 may comprise identifying the replacement pixel as a candidate pixel of the most similar corresponding first and second candidate blocks.

It will be appreciated that certain operations maybe re-ordered and their numbering is not necessarily indicative of the order of processing.

The above operations will now be explained in further detail with reference to a graphical example.

Figure 8 is a schematic diagram of part of a first image captured by a first camera, which first image comprises an identified defect pixel ιοί. The image part maybe a search window loo determined from within a larger image. The search window loo may be arranged such that the defect pixel ιοί is substantially at its centre. In the shown example, the search window 100 is a 7 x 7 search window.

For ease of explanation, each pixel is identified by a reference number P_rc, where r refers to the row number and c to the column number. The defect pixel 101 is at position P₃₃.

In terms of CFA colour components, the defect pixel 101 corresponds to a red (R) pixel and other R pixels are shown in bold for convenience, i.e. Pn, Pi₃, Pi₅, P_3b P₃₅, P₅i, P₅₃ and P₅₅. In some embodiments, the defect pixel 101 may correspond to the position of a different colour component.

A neighbouring block of pixels 103 is determined within the search window 100. In this example, the neighbouring block 103 comprises all pixels adjacent to the defect pixel 101, i.e. a 3 x 3 rectangular block that entirely surrounds the defect pixel. In other embodiments, different sizes and/or shapes maybe used for the neighbouring block, provided the pixels are directly adjacent the defect pixel 101. For example, an L-shaped neighbouring block may be determined.

Figure 9 is a schematic diagram of part of a second image captured by a second camera, which second image covers at least some common content captured by the first camera. The image part maybe a second image search window no that corresponds with that of the first image search window 100, in that it corresponds to identified common content.

As with the first image search window 100, the second image search window no is a 7 x 7 search window identified by a reference number P_rc where r refers to the row number and c to the column number of the second image. The centre pixel 113 of the second image search window no is referred to as P₃₃ and is selected to have the same colour component as the defect pixel 101, i.e. R in this case. The other R pixels are shown in bold for convenience, i.e. Pn, Pi₃, Pi₅, P_3b P₃₅, P₅i, P₅₃ and P₅₅.

Figure io is a schematic diagram of the first image search window loo in which a first candidate block 120 is determined. The first candidate block 120 is of the same size and shape as the neighbouring block 103. The first candidate block 120 is associated with a first candidate pixel 123 which is selected to have the same colour component as the defect pixel 101, i.e. R in this case.

Figure 11 is a schematic diagram of the second image search window no in which a second candidate block 130 is determined. The second candidate block 130 corresponds with the first candidate block 120 in that it is of the same size and shape (and therefore also has the same size and shape as the neighbouring block 103) and is associated with a second candidate pixel 133 having the same corresponding position (Ph) as the first candidate pixel 123 and the same colour component, i.e. R in this case.

In this respect, the first and second candidate blocks 120,130 are referred to as ‘corresponding’ or ‘in correspondence’ because they have corresponding positions in the search windows 100, no, associated with the same colour component.

Further sets of first and second candidate blocks may be determined within the search windows 100, no at different corresponding positions. These further corresponding candidate blocks are associated with the other positions of the same colour component as the defect pixel (excluding the position of the defect pixel itself.)

In the shown example, further corresponding candidate blocks are be determined, and associated with (in this case surrounding) the R candidate pixel positions Pi₃, Pi₅, P₃i, P₃₅, P_5b P₅₃ and P₅₅. Figures 10 and 11 show one further set of corresponding first and second candidate blocks 125,135, centred on candidate pixel position P₁₃ of the first and second search windows 100, no.

It will be appreciated that for a greater number of images, e.g. three or more, there will be a respective number of corresponding candidate blocks per candidate pixel position.

For each candidate pixel position, the similarity is determined between the neighbouring block in the first image and the first and second candidate blocks in the first and second images respectively.

The similarity maybe determined by means of calculating a similarity measure di. For example, the similarity measure di maybe expressed as the sum of absolute differences (SAD) or a sum of squared differences (SSD), e.g. Li or L2 norm.

For example, the similarity measure di maybe expressed as:

For example, the similarity may be measured using SAD between the neighbouring block and each candidate block. This may be performed for each of the first and second images.

For example, referring to Figures 12a and 12b, first and second images 100’, 110’ are shown with example pixel intensity values. A defect pixel 101’ is identified using conventional methods at the position P₃₃, and the neighbouring block is identified by reference numeral 103’.

A SAD determination is performed between neighbouring block 101’ and each of the R candidate pixel positions Pn, Pi₃, Pi₅, P_3b P₃₅, P₅i, P₅₃ and P₅₅ for both the first and second images 100’, 110’. A similarity value di for each candidate pixel position is obtained. For example, taking only the candidate pixel positions Pn and Pi₃ we obtain the following similarity measures (note that the determination is performed in a clockwise manner):

Image 1100’ du = I12-4I + |5-6| + |6-i| + I9-4I + I9-12I + |8-6| + I2-5I + I9-3I = 8 + 1 + 5 + 5 + 3 + 2 + 3 + 6 = 33 dig = |12-1| + |5-4| + |6-3| + |9-4| + |9-6| + |8-5| + |2-12| + |9-4| = 11 + 1 + 3 + 5 + 3 + 3 + 10 + 5 = 41

Image 2 ιοί’ dn =|12-8| + |5-6| + |6-3| + |9-8| + |9-9| + |8-8| + |2-2| + |9-6| = 2 + 1 + 3 + 1 + 0 + 0 + 0 + 3 = io di₃ = |12-3| + |5-5| + |6-3| + |9-9| + |9-8| + |8-2| + |2-9| + |9-8| =6+0+3+0+1+6+7+1 = 24 and so on for di₅, d_3b d₃₅, d₅i, d₅₃ and d₅₅for the first and second images 100’ ιοί’.

A replacement pixel for the defect pixel may be determined by identifying the candidate pixel for which the similarity measure di is the smallest, i.e.:

= min(d_;)

This is because di is in this case a distance measure, and hence the most similar candidate pixel will be that having the smallest distance. Using the numerical examples given above for just two candidate pixel positions in each of the first and second images 100’ ιοί’, it follows that dn from the second image ιοί’ is selected as the replacement pixel for pixel position P₃₃in the first image ioo’.

Having identified the replacement pixel, it is substituted for the defect pixel in the first image ιοί’. The process may then be repeated for other defect pixels in the images.

In an optional operation, in order to extend the search area, the candidate blocks can also be modified, e.g. by rotation, flipping, mirroring and the above process repeated. For example, by performing a horizontal flip on the candidate block 120 associated with the first pixel position Pn, we obtain for the first image ioo’:

dll = |12-1| + I5-6I + |6-4| + I9-3I + I9-5I + |8-6| + |2-12| + I9-4I = 11 + 1 + 2 + 6 + 4 + 2 + 10 + 5 = 41 and so on for di₅, d₃i, d₃₅, d₅i, d₅₃ and d₅₅for the first and second images 100’ 101’.

Again, having calculated the extended range of similarity values di, the smallest distance is identified and this pixel selected as the replacement pixel for the defect pixel.

Modified candidate blocks should preserve the Bayer colour channels as before, so that when calculating the similarity value, e.g. using SAD, we are subtracting values from pixels of the same colour. In the above example, for the first component 112 -11. these values should relate to the same colour, and so on.

In cases where the neighborhoods are not square, e.g. L-shaped, we have an even wider possibility for extending the search area because there are more ways to rotate L-shaped neighborhoods.

For example, an L-shaped neighborhood associated with candidate pixel Pnmay comprise pixels P_oo, Pio, P20, P21, and P22. This may be compared with corresponding pixels of the neighbourhood block, i.e. P22, P32, P42, P43 and P₄₄. Alternatively, or additionally, rotation may involve comparison with the pixels P22, P23, P24, P34, and P44.

In the case of couplet defects, a couplet is treated as two singlets and otherwise the procedure is the same as above.

In some embodiments, the neighbouring block of other defects are excluded from the candidate blocks, and therefore do not take part in the correction process.

As mentioned previously, the neighbouring and candidate blocks can be of any shape and size (provided they comprise more than one pixel). For example, as well as the above example where the blocks completely surround the defect/candidate pixel, it has been shown that the proposed algorithm works well in a 5x 5 search window, with an L-shaped neighbouring block around the defect pixel, which may also be rotated and/or flipped in the manner mentioned above.

In some embodiments, the above algorithm maybe extended also to the temporal domain by block matching within more than one frame, increasing the search area and therefore the probability of a better match.

In overview, therefore, the defect pixel correction process involves the use of two or more images to provide a replacement pixel; there is no need to calculate an average or median value which can be computationally complex.

Figure 13 is a flow diagram indicating optional operations for preparing each image for the correction process.

A first operation 13.1 may comprise rotating and/or warping each image. In this respect, the intrinsic and extrinsic properties and characteristics of the cameras 43, 45, 47 may be such that their resulting images may be at different orientations and/or be distorted. For example, the image from one camera may be upside-down compared with the image from the other camera. In this case, rotation may be employed so that the two images are of the same orientation. For example, one or more of the cameras 43, 45, 47 may employ a wide-angle lens such as a fish-eye lens. In this case, warping may be employed to correct for differences in the lens shapes.

A subsequent operation 13.2 may comprise removing non-overlapping areas of the received images. This reduces the amount of subsequent processing required, leaving only the overlapping area common to the images captured by the cameras 43, 45, 47.

A subsequent operation 13.3 may comprise defining a search window in each image. The search windows can be generated using a so-called PatchMatch algorithm, for example see http://gfx.cs.princeton.edu/pubs/Barnes 2009 PAR/patchmatch.pdf, the contents of which are incorporated herein by reference.

The search window may be a two-dimensional search window of any size. For reasons of efficiency, the search window is defined with a predetermined size of n _x m pixels, smaller than the image size, and has an initial position within the image. In the shown examples of Figures 8 to 11, a 7x7 search window is employed.

The initial position of the search window in a first image is such that the defect pixel is at its centre. The initial position of the corresponding search window in the second (and any further) image is such that the centre pixel has the same Bayer colour component as the defect pixel, i.e. one of a R, G or B colour component.

It will be appreciated that certain operations of the Figure 13 method maybe reordered and/or performed in parallel.

The above method will now be graphically illustrated with reference to Figures 14 to

16.

Referring to Figures 14a - 14c, the images respectively captured by the cameras 43, 45, 47 in Figure 4 are shown. In this case, no rotation or warping is needed because the images are of the same orientation and a conventional lens has been used. The broken line 140 indicates a division in the image whereby the shaded region 141 represents a region that does not overlap with the other images. The shaded region 141 can therefore be removed from each image.

The result of the removal or cropping is shown in Figures 15a - 15c, which respectively correspond to Figures 14a - 14c.

Referring to Figures 16a - 16c, a search window 150 is shown when located around the object 51 in each of the images. The search window 150 may be, for example, a rectangle or any suitable shape or size. The initial position of the search window 150 is such that a defect pixel 151 for correction is at the centre. Search windows 153,155 in the second and third images respectively cover the corresponding object 51 and have centre pixels with the same R, G or B colour component as the defect pixel 151.

Having identified corresponding groups of pixels in each image, the process moves onto the defect correction process described above.

For completeness, Figures 17 and 18 show the result of applying the algorithm to images comprising singlet and couplet defects respectively. The right hand images show the original captured images and the left hand images the corrected versions.

In overview, the described method of pixel defect correction is computationally efficient.

It will be appreciated that the above described embodiments are purely illustrative and are not limiting on the scope of the invention. Other variations and modifications will be apparent to persons skilled in the art upon reading the present application.

Moreover, the disclosure of the present application should be understood to include any novel features or any novel combination of features either explicitly or implicitly disclosed herein or any generalization thereof and during the prosecution of the present application or of any application derived therefrom, new claims may be formulated to cover any such features and/or combination of such features.

Claims

1. A method comprising:

providing first and second images representing a scene captured from different locations, the first and second images each comprising a two-dimensional array of pixels;

identifying a defect pixel in the first image;

determining a neighbouring block of pixels adjacent the defect pixel; and identifying a replacement pixel for the defect pixel based on the similarity of the neighbouring block to candidate blocks of the first and second images.

2. The method of claim 1, further comprising determining a plurality of first candidate blocks, each associated with a respective first candidate pixel of the first image, the first candidate pixels having different respective positions in the first image;

determining a plurality of second candidate blocks, each associated with a respective second candidate pixel of the second image, the second candidate pixels having different respective positions in the second image corresponding to those of the first candidate pixels;

determining the similarity of the neighbouring block to each of the first and second candidate blocks for each position; and identifying the replacement pixel as the candidate pixel of the most-similar candidate block of the first and second candidate blocks.

3. The method of claim 2, wherein the similarity is determined by determining, for each of the plurality of first and second candidate blocks, a similarity value, and identifying the most-similar candidate block by comparing the similarity values.

4. The method of claim 3, wherein the similarity value is generated using a sum of differences between the neighbouring block and the first and second candidate blocks.

5. The method of claim 4, wherein the similarity value is generated using a sum of absolute differences (SAD.)

6. The method of claim 4, wherein the similarity value is generated using a sum of squared differences (SSD.)

7. The method of claim 4, wherein the similarity value is generated using the Linorm and/or L2-norm algorithm.

8. The method of any of claims 2 to 7, wherein the first and second images are colour images, and wherein the plurality of first and second candidate blocks are associated with first and second candidate pixels having the same colour component as the defect pixel.

9. The method of claim 8, wherein each pixel in the pixel array is represented as either a red (R), Green (G) or Blue (B) colour component.

10. The method of claim 8 or claim 9, wherein the first and second candidate blocks comprise pixels adjacent the first and second candidate pixels respectively.

11. The method of claim 10, wherein the first and second candidate blocks comprise all pixels adjacent the first and second candidate pixels respectively.

12. The method of any preceding claim, wherein the candidate blocks have the same size and shape as the neighbouring block.

13. The method of claim any preceding claim, further comprising determining an nxm search window within each of the first and second images, the search window surrounding the identified defect pixel, wherein the candidate blocks occupy corresponding spatial positions within the search window.

14. The method of claim 13, wherein the nxm search window is determined such that the defect pixel is substantially at the centre of the search window.

15. The method of claim 13 or claim 14, wherein the nxm search window is either a 5 x 5 search window or a 7 x 7 search window.

16. The method of any preceding claim, wherein the neighbouring block comprises all pixels adjacent to the defect pixel.

17. The method of any of claims 1 to 15, wherein the neighbouring block comprises only a subset of pixels adjacent to the defect pixel.

18. The method of claim 17, wherein the neighbouring block is an L-shaped block.

19. The method of claim 18, wherein the L-shaped neighbouring block comprises three pixels.

20. The method of any preceding claim, wherein the plurality of images represent substantially simultaneously-captured content.

21. The method of any preceding claim, further comprising receiving the plurality of colour images from separate image sources having a predetermined spatial relationship to one another such that overlapping parts of the images can be identified.

22. The method of claim 21, when dependent on any of claims 13 to 15, wherein the m x n search window is positioned such that it is within the overlapping parts.

23. The method of claim 21 or claim 22, further comprising transforming the received colour images by rotation and/or warping.

24. A computer program comprising instructions that when executed by a computer control it to perform the method of any preceding claim.

25. A non-transitory computer-readable medium having stored thereon computerreadable code, which, when executed by at least one processor, causes the at least one processor to perform a method, comprising:

identifying a defect pixel in the first image;

26. An apparatus, the apparatus having at least one processor and at least one memory having computer-readable code stored thereon which when executed controls the at least one processor:

to provide first and second images representing a scene captured from different locations, the first and second images each comprising a two-dimensional array of pixels;

to identify a defect pixel in the first image;

to determine a neighbouring block of pixels adjacent the defect pixel; and to identify a replacement pixel for the defect pixel based on the similarity of the neighbouring block to candidate blocks of the first and second images.