Quantization Error in Practice

Prepared 2015-09-19, revised 2015-09-23 by Bill Claff

When a continuous signal is sampled into discrete values quantization
error occurs.

In digital photography this happens when the analog to digital conversion is
performed.

It can be shown that the maximum quantization error is
1/sqrt(12) Least Significant Bits (LSB) for any Analog to Digital Converter
(ADC).

However, quantization error is not a constant.

Therefore, when modeling we can't add quantitation error in any simple fashion.

Similarly, when measuring signals in the presence of small amounts of noise, it
difficult to remove the effect of quantization error.

Therefore, it is useful to understand how quantization error varies with noise in the analog signal.

I constructed a brute force model in Microsoft Excel.

My analog input was a normal distribution of varying standard deviation.

My discrete output was a histogram from which an observed standard deviation
was computed.

I used 6,000,000 samples around a mean of 512.

I computed values for the Perfect ADC and for Worst Case. Worst Case being an
ADC offset of 0 rather than 0.5

Here is the ratio of observed to input versus input:

Note that it is only for small values of Input that quantization error has any
practical effect on the observed value.

To accentuate the differences let's change the y-axis to
log2(observed / input):

For Input below about 0.6 LSB the Worst Case starts to
diverge significantly from Perfect behavior.

For Input above approximately 0.6 LSB the discrepancy is less than 1/6 EV.

For the Perfect curve it interesting to note that the curve
crosses the x-axis at an Input of 1/sqrt(12).

The peak is at an Input of (3/2)/sqrt(12) LSB which has an Observed value of
1/2 so the ratio peaks at sqrt(12)/3.

It's hard to pick out but there is an inflection at the Input of approximately
0.6 LSB.

We can compute the observed quantization error as sqrt(Observed^2
- Input^2).

We know this value cannot exceed 1/sqrt(12) (for the Perfect ADC) so for
readability the y-axis in the following chart is normalized to 1/sqrt(12):

So it is clear that quantization error for the Perfect ADC rises quickly from
zero at an Input of 1/sqrt(12) LSB to the maximum value of 1/sqrt(12) at about
0.6 LSB.

Worst Case quantization error starts at approximately sqrt(3) and drops to
nearly meet the Perfect at about 0.6 LSB.

Above 0.6 LSB we can use the 1/sqrt(12) value for quantization error although
this is of no practical consequence above 3.0 LSB.

It would be useful to more accurately map Input and Observed
values at small values.

The following chart shows Observed Variance versus Input Variance:

The lower dotted line represents Observed Variance= Input Variance also
Observed = Input.

The upper dotted line represents Observed Variance = Input Variance + 1/12.

The boundaries shown are at Input values of 0 LSB, 1/sqrt(12) LSB,
(3/2)/sqrt(12) LSB, and 2/sqrt(12) LSB.

Note that on the graph these are variances so the transition Input values are
squared.

Also note the 2/sqrt(12) = 1/sqrt(3) = 0.577 which appears to be the
approximate 0.6 value we encountered above.

The values were originally chosen to perform a piecewise fit to the curves.

This works out well but has limited value since there is an infinite family of
curves between these two.

However, given the fractional part of the Observed mean we ought to be able to
solve using this model to obtain Input from Observed.

I have yet to put this into practice.