Skip to main content

Intelligent Signal Systems

Rewiring Filterbank

A series of new results outlining equivalences between certain “rewirings” of filterbank system block diagrams and the corresponding actions of convolution, modulation and downsampling operators. A general framework of reverse-order and convolution subband structures in filterbank transforms, which we show to be well suited to the analysis of filterbank coefficients arising from subsampled or multiplexed signals. These results thus provide a means to understand time-localized aliasing and modulation properties of such signals and their subband representations — notions that are notably absent from the global viewpoint afforded by Fourier analysis and signal recovery from sampled sequences based on their filterbank characterizations.

Display Color Filter Array

In digital image display devices, data are typically presented via a spatial subsampling procedure implemented as a color filter array, a physical construction whereby each light emitting element controls the intensity level of only a single color. We examine the problem of color filter array design with respect to spatial resolution and human vision and quantify the fundamental limitations of existing designs by explicitly considering the spectral wavelength representation induced by the choice of array pattern and propose a framework for designing and analyzing alternative patterns that minimize aliasing. An empirical evaluation on color images confirms our theoretical results and indicates the potential of these patterns to significantly increase spatial resolution while improving color image fidelity.

Optimal Color Filter Array

In digital imaging applications, data are typically obtained via a spatial subsampling procedure implemented as a color filter array — a physical construction whereby only a single color value is measured at each pixel location. We consider here the problem of color filter array design and its implications for spatial reconstruction quality. We pose this problem as one of simultaneously maximizing the spectral radii of luminance and chrominance channels subject to perfect reconstruction, after proving sub-optimality of a wide class of existing array patterns, provide a constructive method for its solution that yields robust, new panchromatic designs implementable as subtractive colors. Empirical evaluations on multiple test sets support our theoretical results and indicate the potential of these patterns to increase spatial resolution for fixed sensor size and to contribute to improved reconstruction fidelity and significantly reduced hardware complexity.

Optimal Exposure Bracketing

Exposure bracketing is a common technique used to acquire high dynamic range image data requiring short exposure times to capture bright regions of the image without saturation. Long exposure times capture darker image regions effectively. This work investigates the statistics of the photon arrival process to derive optimal exposure control for maximizing signal recoverability in high dynamic range imaging.

Image Quality Assessment

We developed a foundation for assessing visual quality with “corrupted reference” (CR-QA) — a new quality assessment (QA) paradigm for reasoning about human vision and image restoration problems jointly. In most image restoration problems, the notion of an ideal reference exists only in theory and not in practice. CR-QA assesses the visual quality of a restored image signal relative to this ideal reference image with the help of observed image (i.e. corrupted reference). In contrast with today’s QAs, which either require direct access to the ideal reference image or disregards the ideal reference entirely.

Noise Parameter Estimation

Noise is present in all images captured by image sensors because of photon emission and photoelectric effects that are the foundations of the ways in which quantum mechanics enable image sensors, in fact, random noise is a “necessary evil” of image sensors that will continue to require our attention. This work provides a comprehensive characterization of random noise in ways that enhance post-image-capture signal processing steps. We derive the Poisson approximation to model the measurement noise that is the result of photon arrival and photon recapture. A novel methodology to learn the parameters that describe the noise is developed. We present preliminary evidence that accurate noise modeling would improve image denoising, especially in the low photon count/high noise regimes.

3D Display Crosstalk

We analyze the reproduction of light fields on multiview 3D displays. A three-way interaction between the input light field signal, the joint spatioangular sampling grids of multiview 3D displays and the interview light leakage in modern multiview 3D displays is characterized in the joint spatioangular frequency domain. Reconstruction of light fields by all physical 3D displays is prone to light leakage, which reduces sharpness of the images shown in the 3D displays. Stereoscopic image recovery is recast as a problem of joint spatioangular signal demodulation, where the combination of the 3D display point spread function and human visual system provides the narrow-band low-pass filter, which removes spectral replicas in the reconstructed light field on the multiview display. The proposed light field reconstruction method performs light field antialiasing as well as angular sharpening to compensate for the nonideal response of the 3D display.

Single Shot HDR

A combination of photographic filter placed over the lens and the color filter array on image sensor induces differences in red, green and blue channel sensitivities. Spectrally selective single-shot HDR (S4HDR) imaging treats this as an exposure bracketing. Optimally exposed regions of low dynamic range red/green/blue color components are merged in a principled manner to yield a single HDR color image. Though not expected to yield results superior to the traditional time multiplexing counterparts, the single-shot HDR solution we propose is a robust alternative that can be realized with conventional camera hardware.

Denoising Plus Demosaicking

New approaches to demosaicing of spatially sampled image data observed through a color filter array, in which the correlation of color components are exploited in order to reconstruct a subsampled image. Two frameworks for applying existing image denoising algorithms to color filter array data are developed: one for wavelet domain processing, and the other for pixel domain processing. Recent advances in spatio-spectral sampling and panchromatic pixels have contributed to increased spatial resolution and enhanced noise performance. It is necessary to consider the universality of demosaicking design principles instead of CFA-specific optimization for signal recovery.

Object Motion from Blur

There are three types of blur in images: defocus, camera shake and object motion. While defocus and camera shake have received considerable attention, object motion is by far the most difficult feature to detect because of its nonstationarity. Object motion blur is potentially useful for scene analysis because it provides temporal cues from a single image. We developed a technique to infer object motion by detecting spatially varying motion blur from a single image. We are able to remove the blur without introducing ringing or iterative steps.

Multispectral Sensor Design

Color filter arrays (CFA) are popular in single sensor imaging, including consumer, professional and scientific imaging. Color image sensors now deliver high quality images because of technical breakthroughs that led to high sensor resolution and high quality CFA interpolation algorithms. We investigate the plausibility of extending CFA sampling to enable multispectral imaging capabilities. We take a principled modeling approach based on a three dimensional Fourier transform that reveals a surprising degree of “spatial-spectral” structure. We propose a new spectral filter array (SFA) design aimed at maximizing the “recoverability” of the multispectral image signals and provide concrete trade-offs between spatial and spectral resolution stemming from SFA sampling strategies.

Color Constancy

The color of a scene recorded by a trichromatic sensor varies with the spectral distribution of the illuminant. We seek a color representation that is unaffected by illumination changes. Achieving such color constancy is an ill-posed problem because both the spectral distribution of the illuminant and the scene reflectance are unknown. Most methods have approached this problem by leveraging the statistics of individual pixel measurements, independent from their spatial contexts. We show that the strong spatial correlations that exist between measurements at neighboring image points encode useful information about the illuminant and should not be ignored. We develop a method to encode these correlations in a statistical model and exploit them for color constancy. The method is computationally efficient, allows for the incorporation of prior information about natural illuminants and performs well when evaluated on a large database of natural images.

Universal Demosaicking

Recent advances in spatio-spectral sampling and panchromatic pixels have contributed to increased spatial resolution and enhanced noise performance. It is necessary to consider the universality of demosaicking design principles instead of CFA-specific optimization for signal recovery. We introduce a new universal demosaicking method that draws from the lessons learned in Bayer demosaicking designs but can be applied to arbitrary array patterns. We recast the data-dependence of Bayer demosaicking as a parsimonious reconstruction of the underlying image signal that is inherently sparse in some representation. Using properties of filterbanks, we generalize this principle to yield a nonlinear recovery method that is consistent with the state-of-the-art Bayer demosaicking methods.

Binning Artifact Removal

Pixel binning refers to the concept of combining the electrical charges of neighboring pixels together to form a superpixel. The combined charges would overcome the read noise at the sacrifice of spatial resolution is the main benefit. Binning in color image sensors results in superpixel Bayer pattern data and demosaicking yields the lower resolution, less noisy image. Binning introduces severe artifacts. Our analysis proves that these artifacts differ from the ones stemming from loss of resolution or demosaicking and cannot be eliminated by increasing the sensor resolution. We propose a post-capture binning data processing solution that succeeds in suppressing noise and preserves image details. We verify experimentally that the proposed method outperforms the existing alternatives by a substantial margin.


Intelligent Signal Systems Laboratory, Dr. Keigo Hirakawa, Director

Kettering Laboratories
300 College Park
Dayton, Ohio 45469 - 0232