Demosaicing Algorithms for Digital Cameras

 

 

Most digital cameras sample an image with red, green and blue sensors arranged in a photoreceptor array, as shown in the figure below.

This figure was provided by Fuji Corporation.  The purpose of the microlens array is to focus the light onto each pixel. The color filter array changes the color responsivity of each pixel.  Finally, the sensor array captures the light and generates the electrical signal.

 

 This figure illustrates a spatial arrangement of a color filter array (CFA). This array is a commonly used arrangement referred to as the Bayer CFA.

Each pixel in the sensor image is the output of a red, green OR blue sensor.  To display the image, we must create an image that has a red, green AND blue pixel at each location.  We do this by interpolating the missing sensor values. Methods for interpolating the missing sensor values are referred to as “demosaicing”.

In the last ten years, demosaicing has become one of the most popular image processing exercises in digital signal processing.  The literature describing demosaicing algorithms is overwhelming.  Fortunately, demosaicing algorithms can be grouped into a few categories.  At the low end there are simple “fill-up the data” algorithms; in the middle are linear interpolation algorithms; at the high end there are algorithms that choose an interpolation rule based on image contents (adaptive). 

There are two more distinctions to make between demosaicing algorithms. First, some algorithms interpolate using information within a color type.  For example, unknown green values are filled in only by known green pixels, and similarly for red and blue. The bilinear algorithm uses this approach. Other algorithms try to take advantage of the high correlation between color types.  In these algorithms, the green pixels are interpolated using all of their spatial neighbors.  The Laplacian algorithm interpolates using data from multiple color types classes.

Second, amongst the various linear algorithms it is possible to chose a range of spatial filters.  The Laplacian filter is one.  But, depending on the available computational power it can be appropriate to use filters with different size support (3x3, 5x5, 7x7).  As the size of the filter increases, one might use different linear weights.  These weights are chosen for two purposes:  First, the filters do some amount of averaging.  This reduces image noise. Also, by choosing certain patterns of filter weights, the demosaicing algorithm can adjust the sharpening after blurring. (Laplacian is an example of a simple linear method that includes all color channels and some sharpening.) 

ImagEval designed this test image to highlight the visual artifacts introduced by demosaicing algorithms. Sampling or aliasing artifacts are visible as color.