Nikon Raw Values are Quantized
Prepared 2004-11-23 (150/128462) by Bill Claff

Data Table:

 

 

Count

 

 

Quantized ("exact")

 

Quantized (rounded)

E

U

B

G1

G2

R

 

Q

B

G1

G2

R

 

B

G1

G2

R

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

1.179

1.207

1.199

1.236

 

3476

3393

3417

3313

 

 

 

 

 

 

 

 

0.003

-0.024

0.064

-0.013

 

11.76

11.73

11.74

11.69

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

0

0

306

38

42

748

 

0

0.003

-0.024

0.064

-0.013

 

0

0

0

0

1

1

389

65

92

713

 

1

1.181

1.184

1.263

1.223

 

1

1

1

1

2

2

713

152

154

1023

 

2

2.36

2.391

2.462

2.46

 

2

2

2

2

3

3

0

0

0

0

 

3

3.538

3.598

3.66

3.696

 

4

4

4

4

4

4

1049

302

300

1380

 

4

4.717

4.805

4.859

4.933

 

5

5

5

5

5

5

1404

529

472

1718

 

5

5.895

6.013

6.058

6.169

 

6

6

6

6

6

6

1719

782

798

2102

 

6

7.074

7.22

7.257

7.406

 

7

7

7

7

7

7

1677

1033

1021

2426

 

7

8.252

8.427

8.455

8.642

 

8

8

8

9

8

8

1669

1171

1227

0

 

8

9.431

9.635

9.654

9.879

 

9

10

10

10

9

9

1702

0

0

3125

 

9

10.609

10.842

10.853

11.115

 

11

11

11

11

10

10

0

1270

1312

3955

 

10

11.788

12.049

12.052

12.352

 

12

12

12

12

11

11

1828

1310

1227

4634

 

11

12.966

13.256

13.25

13.588

 

13

13

13

14

12

12

2028

1145

1156

5082

 

12

14.145

14.464

14.449

14.825

 

14

14

14

15

13

13

2704

1083

1071

0

 

13

15.323

15.671

15.648

16.061

 

15

16

16

16

14

14

3392

964

891

5298

 

14

16.502

16.878

16.846

17.297

 

17

17

17

17

15

15

4100

0

0

5016

 

15

17.68

18.085

18.045

18.534

 

18

18

18

19

16

16

0

849

837

4563

 

16

18.859

19.293

19.244

19.77

 

19

19

19

20

17

17

4477

774

761

3737

 

17

20.037

20.5

20.443

21.007

 

20

21

20

21



Explanation:
These is the first 18 rows of a 683 row table resulting from an analysis of a raw image from my D70.
E and U are the Encoded and Unencoded raw values from the CCD.
If you don't understand the distinction, or why there are 683 values, don't worry.
The 4 values under Count are a histogram for the raw CCD for each position in the Bayer CFA (B is blue, G1 and G2 are the 2 greens, and R is red).

Note the zero values. These are always present when noise reduction is off and always have the same pattern for a given camera.

The Quantized ("exact") columns show a hypothetical CCD value for a given Q (quanta).
The two values at the head of the columns are the slope and intercept of a least squares fit on the non-zero data in Count.
The Quantized (rounded) columns are the Quantized ("exact") values rounded to the nearest integer.

Note that the integer values missing in each column correspond to the zero values in Count.

The two values at the head of the columns are the number of quantized values that fit in the 12-bit range 0-4095 and the number of bits this value represents.

Conclusions/questions/thoughts:

  1. I think the results are not surprising and reflect a quantization effect that results from a system gain that is slightly above 1.
  2. I'm a little surprised the values are different for R, G1, G2, and B. One implication is that there are 4 A/D converters for efficiency.
  3. Are the gains a factory adjustment or just the result of component variation?
  4. If I were preparing data for demosaicing I'd map from U to Q first to make the raw data evenly distributed.