Instruction: Write an abstract for the following acadamic paper.

Input:
Paper: we have recorded the flight of a fly during take off and landing using digital high speed photography . it is shown that the dynamics of flexible wings are different for these procedures . during this observation fly flew freely in a big box and it was not tethered .
Abstract: in this fluid dynamics video , we demonstrated take off and landing of a fly . the deformation of wings is in focus in this video .

Paper: in small , low - power cmos image sensors and related optics have revolutionized consumer photography@xcite . these technologies have improved dramatically the spatial resolution , dynamic range , and low light sensitivity of digital photography . in addition to improving conventional photography , these technologies open up many possibilities for novel image systems architectures . the new optics and cmos sensors capabilities have already motivated novel camera architectures that extend the original bayer rgb design . for example , in recent years a new generation of architectures have been produced to increase spatial resolution@xcite , control depth of field through light field camera designs@xcite , extend dynamic range and sensitivity by the use of novel arrangements of color filters@xcite and mixed pixel architectures @xcite . to develop these opportunities requires that we innovate on the third fundamental component of image systems , the image processing pipeline . the pipeline is the set of algorithms , including demosaicking , noise reduction , color management , and display nonlinear transforms ( gamma curves ) , that convert the sensor data into a rendered image . even modest changes to the camera architecture , such as more color pixels into the mosaic @xcite or including near infrared detectors @xcite can require substantial rethinking of the image processing pipeline . new image processing pipelines , specialized for the new types of cameras , are slow to develop . consequently , the design of new imaging sensors is far outpacing the development of algorithms that can take advantage of these new designs@xcite , and the vast majority of image processing algorithms are still designed for sensors that use the classic single plane bayer rgb spatial sampling mosaic .    in this paper , we describe a new framework that enables image systems engineers to rapidly design image processing pipelines that are optimized for novel camera architectures . the general idea of the framework was first proposed by lansel et al . in 2011@xcite . here , we introduce the framework in the form of a set of software tools that use simulation and learning methods to design and optimize image processing pipelines for these new camera systems . this paper is organized into three sections that define our contributions . first , we explain the image processing architecture : the input data are grouped by their local features into one of a set of local classes , where locality refers to both position on the sensor array ( space ) , pixel type ( color ) and response level . the optimal affine transform in each class is learned using camera simulation technology . we refer to this framework as @xmath0 to emphasize its key principles : local , linear and learned . second , we assess the performance of @xmath0 method by comparing the rendered quality with the ones from high - end modern digital cameras . we specifically show that such a collection of affine transforms accurately approximates the complex , nonlinear pipelines implemented in modern consumer photography systems . third , we illustrate how @xmath0 method can learn image processing pipelines for new camera architectures . there are a number of related efforts that incorporate system joint optimization and data - driven learning methods in designing camera image processing pipeline . we discuss the relationship between @xmath0 and these contributions more fully in discussion section after introducing @xmath0 framework . the @xmath0 method comprises two main steps : rendering and learning . the rendering step adaptively selects from a stored table of affine transformations to convert the raw camera sensor data into an image in the target color space ( e.g. srgb ) . the training step learns and stores the transformations used in rendering .    conventional image processing pipelines often include nonlinear elements , including thresholding operations and gamma transforms@xcite . the @xmath0 rendering algorithm uses a collection of affine transforms that are applied to data in the different classes . the accuracy of a collection of affine transforms for approximating the image processing pipeline , including nonlinearities such as the gamma , can be controlled by the number of classes ; this is the conventional local linear approximation to continuous functions . the challenge of managing discrete transitions , such as the transition from the linear response region to saturation , represented by thresholds can be managed by selecting proper category boundaries . as a practical matter , there is rarely a strong need to specify a precise and sharp category boundary in natural image processing .    in the following , we explain these two steps of @xmath0 method in detail . we explain the method using the example of a camera sensor with rgbw ( red , green , blue and white ) color filter array . the method can be applied to other designs , which we describe in the results section . the rendering pipeline begins with the sensor data in the spatial neighborhood of a pixel , @xmath1 . we illustrate the method for a @xmath2 neighborhood , so that the neighborhood comprises 25 pixel responses ( figure [ fig : l3render ] ) . each pixel is classified into one of many classes , c , based on its local features : the identity of the pixel , the mean response level of the local patch , the spatial variance of the neighborhood , and pixel saturation status , etc . the total number of classes can be estimated as the product of the number of categories for every feature . for example , suppose that there are 4 types of pixels , and we categorize the mean neighborhood intensity into 10 response levels . this produces 40 different classes . if we further classify each neighborhood into textured ( high variance ) or uniform , then there will be 80 classes ( @xmath3 ) . additionally , pixel saturation is also a condition that requires careful management . in some designs , one type of pixels ( e.g. , the @xmath4 pixel in an rgbw ) can saturate while the rgb pixels will be well within their operating range . it is important to create separate classes that account for pixel saturation . allowing for this increases the number of effective classes , typically by a factor of three for the rgbw case . the number of classes is a hyperparameter of the @xmath0 framework and can vary in different applications .     rendering method . * in @xmath0 rendering , each pixel is classified by its local features into one of a few hundred classes ( e.g. a red pixel , with high intensity , surrounded by a uniform field ) . each class has a learned affine transform ( stored in a table ) that weights the sensor pixel values in order to calculate the rendered outputs . hence the rendered output of each pixel is calculated as an affine transform of the pixel and its neighbors.,scaledwidth=48.0% ]    for each class @xmath5 and output channel @xmath6 , we retrieve an affine transform to map the 25 values into one rendered value . the affine transforms for each class and rendered output channel , @xmath7 , are pre - computed and stored . the rendered output , @xmath8 is the inner product of the stored affine transform and the neighborhood data ( augmented with a 1 to account for the affine term ) . @xmath9    there are several practical considerations in @xmath0 implementation : the content and application dependent class definitions , the target color representations ( monochrome , highly saturated ) and even the fitting model in each class ( affine or polynomial ) . these choices impact the algorithm efficiency and precision , and we describe experiments evaluating different choices in results . these choices are part of the design process when implementing the @xmath0 method .    finally , the computations for each pixel are independent , meaning that the architecture can be highly parallelized with a graphical processing unit ( gpu ) or other hardware to shorten rendering time@xcite . and @xmath0 is designed to perform real - time rendering for high quality images on mobile devices . it is challenging to create algorithms for cameras that are being designed , rather than cameras that already exist , because of limitations in obtaining sensor data@xcite . we solve this problem by using image systems simulation tools to model the proposed camera architecture and to create the training data@xcite . the image systems engineering toolbox ( iset ) begins with a spectral representation of the scene and includes simulations of the optics and sensor . the simulator has been validated against data from several real devices @xcite . once we have simulations of the critical data , we have a variety of options for how we select the transforms that map the sensor data into the rendered images . we describe our approach to simulation and transformation derivation in the next two sections . realistic camera training requires scene spectral radiance data sets that are representative of the likely application . we have obtained scene spectral radiance and accumulated examples from a number of public scene radiance data sets . in addition , we have also used spectral computer graphics methods to simulate a variety of scene spectral radiance images . these simulations can produce scene spectral radiance examples for training that establish special spatial resolution and color requirements that extend what we would be likely to find by merely sampling a range of natural images . a further advantage of the simulation method is that the desired output image and sensor data are precisely aligned , at the pixel level . for each input scene spectral radiance we calculate the calibrated color representation ( e.g. cie xyz ) and the sensor response at each pixel ( figure [ fig : l3train ] ) . such correspondence is very difficult or even impossible to obtain from empirical measurements .    finally , because the training pairs are produced by simulation , we can produce many examples of scenes and measurement conditions . through simulation we can control the properties of the ambient lighting , including its level and spectral power distribution . we can perform simulations with a wide range of optical parameters . training can be performed for special content ( e.g. , faces , text , or outdoors scenes ) . the pairs of images produced by the simulation methods can provide a virtually limitless collection of input data to the training system . the purpose of the training is to calculate a set of transforms that optimally map the sensor responses to the target output image and minimize the empirical risk . stated formally , our task is to find for each class , @xmath10 , a transformation , @xmath11 such that @xmath12    here , @xmath13 is a row vector containing the @xmath14-th patch data from the sensor ; @xmath15 are the corresponding target rendered image values . @xmath16 is the loss function ( error ) between the target image and the transformed sensor data . in consumer imaging applications , the visual difference measure cie @xmath17@xcite can be a good choice for the loss function . for other applications , regularized rmse is widely used . in nearly all photography applications , the transformation from sensor to rendered data is globally non - linear . @xmath0 method approximates the global nonlinear transformation with a collection of affine transforms for appropriately defined classes @xmath10 .    at first , we solve the transforms for each class independently . this problem can be expressed in the form of ordinary least - squares . to avoid noise magnification , we use ridge regression and regularize the kernel coefficients . that is    @xmath18    the data from each patch are placed in the rows of @xmath19 ; the regularization parameter is @xmath20 , and @xmath21 is the output in the target color space as an @xmath22 matrix . we have experimented using several target color spaces , including the xyz , cielab and srgb representations , and we can find satisfactory solutions in all cases . the closed - form solution for this problem is given as    @xmath23    the computation of @xmath11 can be further optimized by using singular vector decomposition ( svd ) of @xmath19 . that is , if we decompose @xmath24 , we have    @xmath25    the regularization parameter ( @xmath20 ) is chosen to minimize the generalized cross - validation ( gcv ) error @xcite .    once the transforms for each class are defined , it is possible to review them for properties that reflect our prior knowledge , such as continuity over the sensor input space , symmetry and uniformity ( see discussion ) . the software implementation includes methods to check for these conditions and to bring the transforms into alignment with this knowledge . in this section , we characterize the performance of the @xmath0 method and illustrate how it can be used to generate image processing pipelines for novel camera architectures . first , we analyze whether the @xmath0 rendering method based on many affine transforms is sufficient to approximate the performance of commercial image processing pipelines ( nikon and dxo ) . second , we use @xmath0 to learn a collection of transforms for non - standard color sensor designs ( rgbw and rgb - nir ) . the @xmath0 pipeline is designed to be computationally efficient and to learn algorithms for novel arrays . before applying the method to new designs , it is important to analyze whether the simple pipeline is capable of supporting high quality rendering expected from camera systems that have been optimized . to evaluate any performance limits , we compared how well the @xmath0 rendering algorithm can approximate the image processing pipeline embedded in a high quality commercial camera .      in one experiment , we used an image dataset of 22 well - aligned raw and jpeg natural images from a nikon d200 camera . we used 11 randomly selected images for training the local linear transforms and the other half for testing ( cross - validation ) . the @xmath0 pipeline parameters were set to use 50 luminance levels for the four pixel types ( red , blue and two types of green ) , for a total of 200 classes . we analyzed the data with @xmath2 local patches ( affine transforms of 26 parameters ) . the effect of patch size is discussed later . figure [ fig : l3nikon ] ( upper left ) shows a typical example of an image produced by the nikon processing pipeline and the corresponding image produced by the @xmath0 method ( lower left ) . by visual inspection the images are very similar ; the largest visual differences are the blue sky and the bush in the lower left . we use a perceptual space - color visual difference metric s - cielab @xcite to quantify the visual difference . perceptual metrics require specifying the display and viewing distance , and for the s - cielab calculation we assumed the images are rendered and viewed on a calibrated lcd monitor with @xmath26 dpi at viewing distance of one meter . for the @xmath27 nikon images , the horizontal field of view is @xmath28 deg . the @xmath17 error image ( lower right ) and histogram ( upper right ) are typical of the test data : the mean s - cielab @xmath17 is 1.74 for the 11 test images ( psnr 40.36 ) , which is a small visual difference . these experiments show that the collection of @xmath0 transforms approximates the full commercial rendering produced by this nikon d200 camera for this collection of outdoor images . local linear transforms . * at the left , we compare camera rgb ( upper left ) with @xmath0 rendered image ( lower left ) . the image at the lower right measures the perceptual difference between the two images using s - cielab @xmath17 values for each pixel . the metric is calculated assuming that the images are displayed on a known , calibrated monitor ( see inset text , upper right ) . the histogram of errors is shown on the upper right . the mean error is 1.84 , the peak error is near 8 , and the standard deviation of the @xmath17 values is 0.9 . the @xmath0 transforms were learned from one set of 11 images . this image is from an independent data set of 11 images . the errors reported for this image are typical for all the images in the independent test set . , scaledwidth=48.0% ] we also applied the @xmath0 method to learn the local linear transforms that approximates a commercial image processing pipeline ( dxo ) . in this experiment , 26 raw and rgb image pairs are analyzed . the rgb images are generated with dxo optics pro using parameters tuned by a professional photographer , dave cardinal . this dataset includes multiple types of cameras and the content spans various natural scenes , human portraits , and scenic vistas .     local linear transforms . * we compare the image rendered by a dxo pipeline with parameters tuned by a professional photographer ( left ) with one rendered with the @xmath0 method ( middle ) . the perceptual error is measured in s - cielab @xmath17 image ( right ) . the error calculations are based on the same monitor as in figure [ fig : l3nikon].,scaledwidth=48.0% ]    each of the individual images can be well - approximated by the @xmath0 method . the images at the left and middle of figure [ fig : l3dxo ] show a typical example of the dxo image and the @xmath0 image . the image at the right is the s - cielab visual difference error for each pixel . the mean s - cielab @xmath17 value for this image is 1.458 and the accuracy of the approximation is similar to what we achieved for the nikon processing pipeline . the expert s settings vary significantly as the scene and camera types change ; for example , in some scenes the expert chooses more sharpening and for others a softer focus is preferred . hence , no single set of @xmath0 transforms applies to all of the images . the broad issue of selecting @xmath0 transforms for specific acquisition conditions or rendering aesthetics is further analyzed in discussion . the dxo and nikon d200 experiments show that the @xmath0 kernel regression approach is sufficient to approximate the transforms embedded in commercial rendering products . next , we examine the properties of the learned transforms that approximate the nikon d200 processing pipeline . when learning the @xmath0 transforms , a few parameters must be selected : the number and distribution of response levels , and the size of the local patch . the transforms change substantially with the response level ( figure [ fig : l3transformbayer ] ) . at low levels , the weights are relatively equal across the entire patch , and there is less selectivity for color channels . at high response levels the weights are concentrated on the appropriate pixel type . for example , when the center pixel is green and the output channel is also green , at high response levels the transform weights are concentrated on the central green pixel . at low light levels the transform weights at non - central green pixels are relatively high . transforms depend on response level . * the three monochrome images show the relative weights of the transforms that convert data at a g pixel centered patch into the green output channel . the patch size is @xmath2 and the three images show the weights that are learned for a class defined by low ( left ) , mid ( second left ) and high ( third left ) response levels . the spatial distribution of the weights becomes more concentrated at the central pixel as the mean response level increases . the cfa pattern of the full @xmath2 patch is shown at the right.,scaledwidth=48.0% ]      the learned transform weights change more rapidly at the lower response levels compared to the higher levels . for this reason , it is efficient to use a logarithmic spacing of the mean response levels that define the classes ; that is , we use more finely spaced classes at the low response levels than the high response levels .    through simulation , we can evaluate the difference between linear and logarithmic spacing of the mean response levels . we analyzed the nikon d200 data using different numbers of mean response levels ( figure [ fig : l3classesspacing ] ) . the levels were spaced either linearly or logarithmically . to achieve the same image quality ( e.g. 2 @xmath29 ) , logarithmic spacing is equivalent to linear spacing with about 50% more number of classes . as the total number of classes becomes large , say 50 for this example , the performance of the two spacing methods is very similar . we expect that the specific parameter values , such as number of response levels , will differ slightly for different optics and sensor combinations . but the principle of using logarithmic spacing is likely to hold across conditions .    ) of the rendered image declines as the number of response level classes increases . the two curves show the effect for linear ( blue dashed ) and logarithmic ( red solid ) spacing of the mean response levels in the patch . this inset images are the renderings for 4 , 7 and 38 mean response level classes ( logarithmic spacing ) . the rendering of the flower ( zoomed view ) changes substantially as the number of levels increases , and the mean color reproduction error declines significantly as the number of mean response levels increases to about 15 . there is only a very small advantage for the logarithmic spacing when using a small number of classes , and no advantage beyond about 12 levels.,scaledwidth=48.0% ]      there is a significant computational cost to increasing patch size . changing the patch size from a @xmath2 to @xmath30 ( @xmath31 ) approximately doubles ( triples ) the number of coefficients and computational cost . moreover , if the training dataset is limited , the risk of overfitting can increase with the patch size .    for the nikon and dxo approximations , we found little improvement in the approximation as we changed the patch size beyond @xmath2 . figure [ fig : l3patchsize ] shows the mean @xmath29 values of s - cielab on the 11 test images . the plotted data points show the mean error for individual test images , and the solid red line shows the average of all 11 images . the mean @xmath29 values decrease very slightly as the patch size increases from @xmath2 to @xmath30 and there is no further decrease as the patch size increases to @xmath31 . it might be more effective to create classes based on other features ( e.g. nonlinear or global features ) rather than increasing the patch size . errors for the 11 test images . the inset monochrome images are examples of the learned affine transforms . the red rectangle in each image denotes the center @xmath2 region . the weights learned by the transform outside of the @xmath2 patch are small , and increasing the patch size has little effect on the color accuracy.,scaledwidth=48.0% ]      the ability to automate the process of learning the rendering pipeline is an important objective of the @xmath0 method . in this section , we use @xmath0 to generate the image processing pipelines for two challenging cfa designs . we first apply @xmath0 method to generate an image processing pipeline for camera with rgbw sensor . the cfa repeating pattern of rgbw sensor contains both rgb and clear ( white ) pixel . adding a white pixel extends the operating range of the camera and makes the camera usable in low light imaging . the key challenge in designing a pipeline for this sensor is the large mismatch in the sensitivity between the w and rgb pixels@xcite    we then consider a cfa that combines rgb channels with a near infrared ( nir ) channel . there is a great deal of interest in adding an nir channel to support applications of depth sensing . the nir channel , which is invisible to the human eye , can be used to measure a projected nir pattern for depth estimation . the nir pixels do not contain significant information for image reproduction , so that this design reduces the pixel count significantly . we analyze concerns how to best render an image for the rgb - nir design , and how this rendering depends on factors such as pixel size and optics .      in this experiment , we simulated an rgbw camera with exactly one r , g , b and w in each cfa repeating pattern . the spectral transmittance of the color filters and other key sensor and lens parameters of the camera simulation are shown in figure [ fig : l3rgbw ] . the relative sensitivity of the w to the rgb pixels and the spatial arrangement of the four pixel types differs between vendors @xcite , and this simulation represents one of a range of possible choices . patch centered on a green pixel . , scaledwidth=48.0% ]    we apply simulation methods ( figure [ fig : l3train ] ) to prepare the training data to learn the local linear transforms . specifically , we first calculate the sensor response of multispectral scenes using the iset camera simulator . then , we compute the ideal ( cie xyz ) value at each pixel location . the local patches are classified by mean response levels and the center pixel type . affine transformations are learned for each class , using ridge regression with the regularization parameters set using a cross - validation error minimization . figure [ fig : l3transformrgbw ] shows examples of the learned transforms for this rgbw camera in four different classes , from low response levels to near saturation . the low mean response class transforms ( low ) heavily weight data from the w pixel , presumably because the signal - to - noise ratio ( snr ) of the w pixel is substantially higher at generally low response levels . as the response level increases ( mid ) , the w pixel snr advantage is less important than the color information provided by the rgb pixels , and the weights redistribute to using more of the data from these pixels than the white pixel . as the w pixel saturates ( high ) the @xmath0 transforms further discounts the w responses . as the g pixel begins to saturate as well ( saturate ) , the weights on both the w and g pixels decline , with most of the weight being assigned to the r and b pixels . by designing tables that include many response levels , we assure a smooth transition from the w - dominated to the rgb - dominated domain . notice that for the mid and high response levels , the red output depends significantly on the the g center pixel response . the algorithm learns that there is a strong spatial and color correlation between the g center pixel value and output red channel@xcite . this confirms the previous observation that the value at the g center pixel is useful in predicting the red output value , and the linear transform quantitatively estimates the proper amount that g pixel should contribute to the red channel output . however , as the center g pixel starts to saturate ( saturate ) , the transforms assign them lower weights . we analyzed two important aspects of the @xmath0 performance : color accuracy and image resolution . we performed this analysis by training on a standard data set and then testing performance on targets designed to analyze color and resolution . to assess color accuracy , we compute the cielab @xmath29 values between the @xmath0 rendered image and ideal values of a standard macbeth color checker . the @xmath0 pipeline for the rgbw sensor achieves a mean @xmath29 of 1.7 , which is very accurate . we then replaced the w pixel with a green pixel to form a traditional bayer cfa pattern . the mean @xmath29 difference is similar ( @xmath29= 1.65 ) . hence , the @xmath0 method learns how to incorporate the w pixels to achieve an accurate color reproduction .    to assess resolution , we calculated the spatial frequency response ( sfr ) using the iso 12233 slanted bar method@xcite . this method measures the image intensity in the region near the edge of a slanted bar . the intensity measurements are converted from a spatial representation into a modulation transfer function ( mtf ) . the metric is defined by the spatial frequency at which the mtf drops to half of its peak value ( mtf50 ) ; higher mtf50 values imply higher spatial resolution . the value of the mtf50 depends on a number of system features , including the optics and the pixel size . for the system described in figure 8 ( f/#=4 , pixel size 1.4 um ) , the mtf50 is 154.80 cycles / mm , which is close to the upper bound imposed by the optics 186.70 cycles / mm . we assessed the spatial resolution for different lens ( diffraction - limited ) and sensor combinations ( figure [ fig : l3mtf50]a ) . the radius of the blur circle grows proportionally with the f/ # , blurring the optical irradiance at the sensor . when the pixel size is small and the f/ # is large , the spatial resolution , assessed by mtf50 , is limited by the optics . when the pixel size is large and the f/ # is small , the spatial resolution is limited by the pixel size . the exact value of the mtf50 also depends on the training data . in figure 10a , the @xmath0 transforms are trained with a collection of human faces which do not have sharp edges . to evaluate the upper limit of spatial resolution , we trained with a scene containing only a spatial resolution chart and again used the mtf50 metric to evaluate spatial resolution ( figure [ fig : l3mtf50]b ) . in this case , the @xmath0 transforms achieve almost double the spatial resolution in the optimal region ( small pixel size , small f/ # ) . in other regions , the benefits of using spatial resolution target is much smaller . in addition to the optics and pixel size , scene illumination level and exposure conditions also matter ( figure [ fig : l3mtf50]c ) . we evaluated the mtf50 for a fixed pixel size using over a range of response levels and f/#s . for small f/ # when the resolution can be very high , the @xmath0 transforms differ between the response levels . at low response levels , the transforms reduce noise by placing significant weights on most of the pixels in the patch . thus the mtf50 is relatively low . when the response level is high , the mtf50 is much higher . for large f/ # the loss is dominated by the lens and thus the difference in the mtf50 between low and high response levels is minimal . next , we use the @xmath0 method to design an image processing pipeline for an rgb near infrared ( nir ) sensor . the main application for including an nir channel is to acquire extra information that is used in combination with an ir projector to estimate depth @xcite . there are several ways to implement nir sensors . one approach removes one of the two green filters and the ir cutoff filter that is normally placed on the sensor surface . modern color filters pass significant amounts of ir , so this approach allows nir photons to enter the same pixels that are used by the visible light@xcite . the image processing pipeline must estimate and remove these correlated ir signals , which introduces noise and reduces sensor dynamic range . an alternative approach , recently implemented by panasonic , selectively blocks ir photons in the rgb channels placing metal within the pixel@xcite . in this approach , each cfa block contains a pixel of each type , and the rgb channels are protected from absorbing nir photons by an infrared cut filter layer which includes a stack of silicon oxide and titanium oxide films . the panasonic rgb - nir differs significantly from rgbw because the nir channel captures very little information in the visible range . however , there may be useful image reproduction information in the nir channel , and in any case the pipeline must run effectively even if the imaging component only uses the rgb channels .    in this example , we simulate the panasonic design and the key parameters are shown in figure [ fig : l3rgbnir]a . learning the @xmath0 pipeline for this sensor requires a hyperspectral scene radiance dataset that extends into the nir . we used a dataset of 12 scenes that were measured from 415 to 950 nm . the data set includes calibration targets , fruits , buildings and natural scenes . we used the @xmath0 method to learn a set of local linear transforms for the rgb - nir sensor . figure [ fig : l3rgbnir ] shows the normalized weights of nir pixels for g centered patches at different response levels . for low response levels , the @xmath0 transforms assign significant weight on the nir pixels ; the weights on these pixels become very small at high response levels . we evaluated the color accuracy for the rgb - nir sensor as we did for the rgbw case . the cross - validated median cielab @xmath29 value is 2.87 . if we black out the ir pixel and solve for the transforms again , the cross - validated median @xmath29 increases slightly to 3.08 . this shows that the nir data can slightly help estimate color . if we replace the ir sensor with a green pixel , forming a conventional bayer pattern , the median @xmath29 value is 2.69 . hence , the rgb - nir acquires some information in the invisible range at a cost of slightly worse rgb image . the simulations and @xmath0 algorithm quantify how to take advantage of this information in natural indoor images . we also evaluated the spatial resolution of the rgb - nir design using the mtf50 measure . for the panasonic camera , the mtf50 value is 137 cycles / mm ( optics f/#=4 , 2.75 um pixel size with proper exposure ) . replacing the nir pixel with a g pixel , increases the mtf50 to 151 cycles / mm . this shows that the rgb - nir design has slightly lower spatial resolution than the matched bayer design . we quantified how accurately the @xmath0 architecture renders images . we used a perceptual error metric to compare the @xmath0 rendering with two different commercial rendering methods ( cf . nikon and dxo ) . the @xmath0 architecture , based on kernel regression approximation , produces images that are about 2 @xmath29 ( spatial cielab ) from the original . we also showed that the @xmath0 architecture can automatically generate rendering pipelines that are optimized for novel sensor designs . we implemented and evaluated image rendering pipelines for a sensor designed to extend the dynamic range by including a clear pixel ( rgb - w ) , and also for a sensor that includes a set of pixels capable of measuring near infrared patterns projected into the scene to estimate depth ( rgb - nir ) .    in this section , we review the relationship between @xmath0 and other new ideas in image processing . then , we discuss some design choices of the @xmath0 method that arise in practice . first , we discuss the need to create multiple tables of transforms that should be applied in different acquisition conditions ( color balancing ; exposure duration ; imaging conditions ) . second , we describe how we account for knowledge about the transformations that should be incorporated to improve the learned transforms . third , we consider the choice of a target space for the rendering . fourth , we discuss how @xmath0 might be extended to address additional modules in the rendering pipeline . there are several themes in the literature that share common elements with the @xmath0 method . at the most general level , milanfar et al . proposed using the multidimensional kernel regression and non - parametric learning methods in image processing and reconstruction @xcite . that work generalizes several image processing methods , including bilateral filtering , and denoising algorithms , under kernel regression schema . the general principles of kernel regression - classifying local data and interpolating measurements - can be applied to a range of imaging problems , such as learning super - resolution kernels @xcite . the proposal that is closest to our work comes from khabashi et al . @xcite similar to our work , they describe a nonparametric regression tree models , together with gaussian conditional random fields , to demosaic raw sensor response . they classify the data near each pixel into one of a large number of classes ; the class is based on the color filter type of the pixel and a measurement of the local edge direction in the @xmath32 neighborhood of the pixel . for each class they use example camera data to find a quadratic transform that maps pixel data to the rendered value . in addition to these similarities in the approach , there are several significant differences . first , khabashi et al . perform their training by processing mosaicked sensor data from existing cameras to estimate the ground truth , full resolution . in contrast , @xmath0 makes extensive use of image systems simulation technology to create sensor data for training . the use of simulations enables @xmath0 to support analyses for cameras that do not yet exist . second , khabashi et al . classify the sensor data based on pixel type and spatial orientation of the data in the of @xmath2 patches within the sensor mosaic . @xmath0 classifies the sensor data based on response level and local contrast . relying on response level is important because the different levels have very different noise characteristics , and the optimal transform differs significantly between low and high response levels ( figure [ fig : l3transformrgbw ] ) . finally , khabashi et al . focus on the demosaicking stage of the process , while @xmath0 training replaces additional pipeline components , including denoising and color transforms that map the sensor data directly into the target color space . another related set of ideas concerns the development of image processing pipelines that are based on joint optimization across optics , sensor , and display . an example is from stork and robinson @xcite who offered a theoretical foundation for jointly optimizing the design and analysis of the optics , detector , and digital image processing for imaging systems . they optimized the image processing pipeline for different lenses , assuming a monochrome sensor . the @xmath0 method incorporates lens properties into the simulation , so that the table of transforms accounts for the specific lens properties . different tables are generated as the lens properties ( e.g. , aperture , f/ # ) are varied . hence , the @xmath0 method is also a co - design approach in the sense that the learned rendering parameters depend on the whole system , including the optics and sensor . heide et al . also conceive of the image processing pipeline as a single , integrated computation@xcite . they suggest a framework ( flexisp ) in which they model the relationship between the sensor data and a latent image that represents the fully sampled sensor data prior to optical defocus . they propose to estimate the latent image from the sensor data by solving an optimization problem ; the optimization accounts for both the data and a set of natural image priors . hence , a key difference is that heide et al . calculate a solution separately for each sensor acquisition , while @xmath0 pre - computes a fixed table of transforms and applies this table to all images . another difference is that heide et al . , like khabashi et al . , begin their calculations with the sensor data . in contrast , @xmath0 simulates a camera system beginning with scene radiance , accounting for properties of the optics , pixel , and sensor . by working from scene spectral radiance data , @xmath0 can be used to create pipelines at the earliest stages of the design process , when no hardware implementation yet exists . the simulations also make it possible to optimize @xmath0 parameters for different types of scenes , some of which may be difficult to create in a laboratory environment . convolutional sparse coding ( csc ) methods share some features of the @xmath0 method . csc representations begin with a full image representation and decompose the image into a linear sum of component images @xcite . each component is the convolution of a single , usually small , kernel with a sparse feature map ( most entries are zero ) . the csc learns local features from the input training images , and the core calculations are linear . however , the csc learning methods and target applications of the differ significantly from @xmath0 . first , csc learns kernels and feature maps that decompose an image into separate components . @xmath0 performs the reverse computation ; it starts with partial sensor data and creates a complete image . second , the learning methods are different . the csc kernels are learned through advanced bi - convex optimization methods that require substantial computational power . the affine ( or simple polynomial ) transforms learned by @xmath0 use prior knowledge of the camera and training about the camera design but very simple optimization methods . in summary , @xmath0 is an architecture for designing new image processing pipelines and efficient rendering ; csc is a technique for feature extraction and applications to learning image features for machine vision applications and computational photography , such as inpainting . training @xmath0 for the range of settings ( optics , sensor properties ) of a single camera , leads to different transform tables ( figure 10 ) . for mobile devices , the camera settings do not vary extensively . when only a few number of possible settings are available , the best solution might be to pre - compute and store a table of transforms for each setting . in addition to hardware settings , there is the question of whether the @xmath0 training would produce different tables as we change the scene characteristics . we have explored important case : how the tables depend on the spectral power distribution of the illumination germain . in this case we found that the tables learned for different illuminant are similar enough so that we can render the image with a single table and then apply a @xmath33 color transform to render a color - balanced image . there are many other different scene characteristics that remain to be explored . the optimal table of transforms may depend on factors such as image motion , image content , optics parameters such as depth of field and focus . it is possible that multiple tables will be required or that a single set of tables followed by simple transforms will suffice for most conditions . @xmath0 training depends on the specific training data . this can be used to our advantage , say if we know we want to optimize the rendering for a particular condition and target ( e.g. , human faces , outdoors ) . the reliance on specific samples produces transforms that may differ in some small way from our expectations . for example , in many cases we expect the transforms to be left - right symmetric . further , we expect that transforms at nearby response levels will be similar to one another . we developed functions that can be applied to the learned table of transforms to enforce these expectations ( prior knowledge ) . * * symmetry * + when the underlying color filter array of each patch is symmetric in some way ( up - down , left - right , transpose , circular ) , we also expect the learned transform to be symmetric . imposing symmetry helps avoid over - fitting to the training data . we transform general transforms into symmetric transforms by creating symmetric versions of the learned transform and then using the average . * * smoothing and interpolation * + the @xmath0 coefficients change relatively smoothly as the response level increases . we smooth the transforms by fitting a spline to each of the coefficients and then replacing the coefficients with the value of the smooth spline . we use the same method to interpolate for transforms in classes that have a small amount or insufficient training data . * * uniformity * + a uniform scene should be rendered as a uniform image . this requirement , unlike the previous two , requires operating on the transforms from different pixel types . specifically , the sum of the transform weights of each pixel type must be equated between the classes of different center pixels . we emphasized @xmath0 consumer photography applications : the sensor data are transformed to a rendered image . even for consumer photography applications , there are multiple choices for the target rendering space . we have trained @xmath0 instances to transform into various colorimetric spaces ( e.g. , cie - xyz ) , and we have also trained to transform into nonlinear representations ( e.g. , srgb , cielab ) . because the global @xmath0 transformation is nonlinear ( though locally linear ) , the input data can be effectively transformed to most representations that are smoothly related to colorimetry representations . choosing the target space is equivalent to choosing the error function . for example , rendering to cielab space minimizes the point - by - point color error . the current @xmath0 formulation does not account for the position of the center pixel within the sensor . thus , the algorithm is effectively shift - invariant . there are two important aspects of the rendering pipeline that are space - varying . first , lens shading produces an uneven illumination level from the center to the periphery of the sensor . second , geometric distortion of the image ( e.g. , barrell distortion ) varies the relationship between the position of the pixel within the sensor and its appropriate position in the output image . the pixel type and response level are used to identify a class , and we do not include the position of the pixel within the sensor . hence , the @xmath0 method is fundamentally space - invariant . correcting for the shift - varying components ( lens and geometric distortion ) are shift - varying . the parameters of these features are determined by the main taking lens and are independent of the image processing pipeline . hence , like illuminant correction , for the moment we think it is best to perform these steps separately rather than extending the number of classes and setting @xmath0 the task of accounting for the position - dependent factors .      the data and methods necessary to reproduce the figures are available from the stanford digital repository . we introduce a methodology to automate the design of image processing pipelines . the image - processing pipeline is approximated as a locally linear operation in which sensor data are grouped into various classes , and the data from a class are rendered by a linear transform into the rendered image . we illustrate that the local transforms can produce high quality rendered images . then , we use image systems simulation to create the table of affine transforms for novel camera designs , including sensors with clear or near infrared pixels . we evaluate the performance of these tables using color metrics ( s - cielab ) , spatial resolution metrics ( mtf50 ) , and simulations of captured images . hence , this paper combines image systems simulation technology and modern computational methods into a methodology that creates image processing pipelines . bishop , tom e , and paolo favaro . , _ the light field camera : extended depth of field , aliasing , and superresolution . _ , pattern analysis and machine intelligence , ieee transactions on 34.5 ( 2012 ) : 972 - 986 . baranov , pavel , and olesya drak . _ a new color filter array with high light sensitivity and high resolution properties . _ , young researchers in electrical and electronic engineering conference ( eiconrusnw ) , 2015 ieee nw russia 2 feb . 2015 : 17 - 23 . wang , jue , chao zhang , and pengwei hao . _ new color filter arrays of high light sensitivity and high demosaicking performance . _ , image processing ( icip ) , 2011 18th ieee international conference on 11 sep . 2011 : 3153 - 3156 .
Abstract:

Answer: many creative ideas are being proposed for image sensor designs , and these may be useful in applications ranging from consumer photography to computer vision . to understand and evaluate each new design , we must create a corresponding image processing pipeline that transforms the sensor data into a form that is appropriate for the application . the need to design and optimize these pipelines is time - consuming and costly . we explain a method that combines machine learning and image systems simulation that automates the pipeline design . the approach is based on a new way of thinking of the image processing pipeline as a large collection of local linear filters . we illustrate how the method has been used to design pipelines for novel sensor architectures in consumer photography applications . local linear learned , camera image processing pipeline , machine learning