NefUtil
v1.7
Prepared 2009-01-02 by Bill Claff
Last updated 2011-11-08
NefUtil is a Microsoft Windows console executable. It is used at a Windows “DOS prompt”.
This program is provided “as is” with no expressed or implied warranties.
You may distribute the program provided that it is not modified and that no fee is charged.
The program takes parameters consisting of options and filenames or directory names.
Input files are never written to, they are opened in read only mode.
Options start with a plus sign (‘+’) or a minus sign (‘-‘).
A plus sign turns an option on while a minus sign an option off.
Options consist of one letter that is often immediately
followed by one or more optional modifiers.
Options and modifiers are not case sensitive.
There must be spaces between options but not between options
and modifiers.
You can place quotes around any command line parameter if you really need to.
If an environment variable called ‘NefUtil’ exists then it is
processed before any command line parameters.
This is convenient for setting personalized default options.
The edges of the sensor are covered in a material called
“Optical Black”; they receive no light.
When data is available from this inactive area it can yield interesting
information about the Read Noise of the sensor.
The area inside of the Optical Black is called the Active
area.
The active area is covered by, among other things, a Color Filter Array (CFA).
The CFA restricts each photosite to only one color.
I call each of these colors a “channel” and label them R, Gr, Gb, and B where R
is red, Gr is green in the red column, Gb is green in the blue column, and B is
blue.
Regardless of the CFA layout of the chip all results are reported in R, Gr, Gb,
B order.
Inside the Active area is the Effective or Visible area; this is the size of the image you see after the raw data is processed.
Statistics are gathered on the Optical Black area, when present, and the Visible area, but not on Active photosites that are not Visible.
In addition to the entire Visible area statistics are gathered on a central circle and on a grid.
The central circle has a radius of 36 photosites and comprises about 4096 photosites per channel.
A grid comprising 64x64 photosite squares is centered over the Visible area. The number of squares depends on the size of the Visible area.
The basic statistics gathered for a particular area are count, minimum, maximum, average (signal), and standard deviation (noise).
Generally statistics are gathered for the signal but sometimes for other quantities such as the Signal to Noise Ratio (SNR).
The ‘S’ option generates statistical summaries.
Summaries consist of minimum, maximum, average, and standard deviation values.
Generally summaries show the four Color Filter Array (CFA) channels separately in the order R, Gr, Gb, and B where R is red, Gr is green in the red column, Gb is green in the blue column, and B is blue.
Summary values are tab separated so they can be easily imported into applications such as Microsoft Excel.
To facilitate extracting summaries from multiple files each starts with a label, camera model, bits-per-sample, and ISO.
The ‘H’ option generates histograms.
When turned on one or more histograms will be produced.
To facilitate extracting summaries from multiple files each starts with a
label, camera model, bits-per-sample, and ISO.
The histogram labeled ‘H1’ is of the optical black area of the
sensor.
This data is not available for all camera models.
The histogram is of interest in measuring read noise.
If the statistics option is on then STATH1: is also produced.
The histogram labeled ‘H2’ is of the image area.
If the statistics option is on then STATH2: is also produced.
Note that STATH2: is for the entire image as well as a central circle
encompassing about 1024 photosites per channel.
The output option specifies the directory where output files
will be placed.
The default location is the directory from which the program is executed.
+O will cause the output directory to be the same as the input file directory.
You can also place a specific existing directory after +O as a modifier.
When turned on the full resolution JPG embedded in a NEF is
extracted.
The output option controls the location of the file.
It will have the same name as the NEF and the extension ‘jpg’.
Warning, if you shoot RAW+JPEG then
the +J option will overwrite the JPG from the camera if it is in the output
directory.
Ordinarily you want the larger primary embedded image.
But if you want the (usually) smaller image associated with the MakerNote
thumbnail then use the ‘T’ modifier, eg. +JT.
The ‘P’ option extracts user‑friendly photo information.
One or more modifiers are required.
The ‘S’ modifier produces a summary, for example:
NIKON D300 70-180mm f/4.5-5.6D ED AF
Zoom Micro-Nikkor@129mm f/20 d=1'6" f/20 1/250s ISO 200
[The distance information is approximate and infinity focus will currently
appear as a large distance.]
The ‘E’ modifier produces an exposure summary, for example:
Manual mode Spot metering
The ‘F’ modifier produces a focusing summary, for example:
MANUAL Mode: Single Area AF: Off AF
Point Used: none
The ‘L’ modifier produces a lighting (flash) summary, for
example:
On,
Return not detected
[This modifier needs the most additional work.]
The ‘B’ option tests for bad photosites (“hot pixels”).
This option requires a specially prepared image called a flat
field image.
A flat field image is a well exposed, out of focus image of a uniform target.
An image of a white computer screen taken in manual focus at close range with
the focus set to infinity and overexposed by about 2 stops generally works
well.
However, the image must not be clipped!
For each bad photosite that is detected there will be a line
of the form:
Bad photosite: 995 1951 2600 24.5 R 575
This indicates a bad photosite with a raw value of 995 at column 1951
and row 2600.
The photosite has a Z-score of 24.5, is red, and the average of the surrounding
red photosites is 575.
(A Z-score is the number of standard deviations from the average.)
A STATGRID: statistical summary is also produced.
The underlying values are the Signal to Noise Ratios (SNRs) of the analysis
grid.
For the purposes of analysis the image is divided into squares with 4,096
photosites per channel.
The SNR values should be as high as possible and if any drops below 16.0 the
analysis is not performed.
The ‘B’ option takes up to three optional modifiers.
Remember, statistical summaries are included only if you use the ‘+S’ option.
Although NefUtil is used primarily on Nef, Nrw, Dng, and Jpg files it can decode some other files that are sometimes embedded in those primary targets.
Other filetypes that NefUtil decodes are: Icc, Ndf, and Ntc.
Although wildcards are not allowed in filenames it is straightforward to use the DOS for command to process multiple files.
For example, this batch file:
for %%f in (*.nef) do nefutil %1 %2 %3 %4 %5 %6 %7 %8 %9
"%%~dpnf.nef">"%%~dpnf. txt"
will process all of the NEF files in the current directory and redirect the output to TXT files with the same names as the NEF files.
If you add the /R option to the for command then you can process entire directory trees.
The ‘C’ option is used to invoke one or more complex custom operations, often as part of a collaboration.
By ‘Collaboration’ I mean data that is collected by your that you are encourage to report to me for aggregation that can be shared with others.
The ‘+CPTC’ option will produce a histogram. The lines produced will be of the form:
PTC: NIKON D200 12 200 0
76.04 5.79
PTC: NIKON D200 12
200 0 79.60 6.03
PTC: NIKON D200 12
200 0 83.39
6.31
PTC: NIKON D200 12
200 0 95.35
7.14
PTC: NIKON D200 12
200 0 95.77
7.16
The camera model is Nikon 200. The data is 12 bits-per-photosite. The ISO is 200. The datapoint is for channel 0 (red).
For the first line the signal is 76.04 and the noise is 5.79
The output can be imported into a program, such as Microsoft
Excel.
A scatter plot makes a good display.
Data from all of the NEF and DNG in the current or specified
directory will be processed (but no subdirectories).
All files should be for the same ISO.
Files for analysis need to be specially prepared.
[More on that to be added.]
The signal range covered will depend on the number of files and the data in those files.
Although the complete Photon Transfer Curve (PTC) is interesting the most important features are at the high and low end of the curve.
At the high end there will be a sharp drop-off in noise where the sensor becomes non-linear.
At the low end the point at which the curve crosses a particular SNR determines Photographic Dynamic Range (PDR).
If the PTC data crossed the appropriate are your will get a line such as:
STATPDR: NIKON D200 12 200 6.96 7.31 7.30
7.16 7.18
The last five values are the PDR for each of the four channels
and the average of all channels.
Lo indicates the curve was below the PDR range and Hi indicates the curve was
above.
If you have PDR values for a camera model not listed at my site then please
e-mail your value(s) to me.
The ‘+CPDR’ option is quite similar to the Photon Transfer Curve option.
Either one DNG or NEF file, or all DNG and NEF files in the
current or specified directory will be processed.
Generally the files will be for different ISOs.
Files for analysis need to be specially prepared.
Run DR_Grid.
Resize the window so it’s somewhat larger than you can view in the viewfinder at closest focus and in focus.
At
this first screen adjust your monitor brightness so you are properly exposed at
1/30s with the lens wide open and at your native ISO (the lowest numbered ISO
for your camera).
1/30s is ideal but if you cannot achieve 1/30s do not worry.
Switch
the exposure from 1/30s to 1/250s and press ‘G’ on your keyboard to proceed to
the grid.
If you did not achieve 1/30s in the preceding step then choose a shutter speed
that is 3 stops faster than the shutter speed you can achieve.
So if proper exposure is 1/15s then switch the exposure to 1/125s.
Take a slightly out of focus image of the grid. The image should not contain any of the surrounding white.
Take
an image at each full ISO such as 100, 200, 400, 800, 1600, 3200, etc.
Including any Lo or Hi ISO that your camera may have such as Lo 1.0 or Hi 1.0.
The STATPDR: lines will be captured into a file named “C_PDR.txt”.
A typical file might look like:
STATPDR: NIKON D200 12 100 7.66 8.08 8.09 7.65 7.87
STATPDR: NIKON D200 12 200 6.99 7.20 7.25 7.00 7.11
STATPDR: NIKON D200 12 400 6.03 6.32 6.34 6.03 6.18
STATPDR: NIKON D200 12 800 5.03 5.28 5.35 5.02 5.17
STATPDR: NIKON D200 12 1600 4.02 4.32 4.35 4.04 4.18
STATPDR: NIKON D200 12 3200 2.85 3.28 3.30 3.05 3.12
The last five values are the PDR for each of the four channels
and the average of all channels.
Lo indicates the curve was below the PDR range and Hi indicates the curve was
above.
Please e-mail your value(s) to me especially if you have PDR values for a camera
model not listed at my site.
The ‘+CGain’ option will produce gain calculation.
All DNG and NEF files in the current or specified directory
will be processed.
Generally there will be sets of files for different ISOs.
Files for analysis need to be specially prepared.
Put
any solid (white) image on the screen, such as an empty notepad.
(The solid magenta image from my DR_Grid
program is even better.)
All
readings and images are taken wide open in manual focus with the lens set to
infinity and touching the computer screen.
(If your screen curves a lot then be sure to use a lens with a recessed front
element.)
Take
a set of 14 images at each full ISO such as 100, 200, 400, 800, 1600, 3200,
etc. using the steps as outlined.
Including any Lo or Hi ISO that your camera may have such as Lo 1.0 or Hi 1.0.
As you increase ISO you can either decreasing shutter duration or stopping
down.
The C_GAIN: lines will be captured into a file named “C_Gain.txt”.
A typical file might look like:
C_GAIN: NIKON D300 14 100 (Lo1.0) 1.403 1.766 1.731 1.470 1.592
C_GAIN: NIKON D300 14 200 1.060 1.320 1.381 1.177 1.234
C_GAIN: NIKON D300 14 400 0.527 0.693 0.693 0.586 0.625
C_GAIN: NIKON D300 14 800 0.268 0.330 0.329 0.298 0.306
C_GAIN: NIKON D300 14 1600 0.136 0.164 0.170 0.147 0.154
C_GAIN: NIKON D300 14 3200 0.069 0.083 0.082 0.073 0.077
C_GAIN: NIKON D300 14 6400
(Hi1.0) 0.034 0.041 0.039 0.034 0.037
The columns show the camera, bits, ISO, and gain values.
The gain values are in electrons per ADU and are for the R, Gr, Gb, and B
channels.
The rightmost column is simply an average of the R, Gr, Gb, and B values.
Please e-mail your value(s) to me especially if you have gain values for a camera model not listed at my site.
The ‘+CLensData’ option will produce a file named
“C_LensData.txt”.
To participate in the collaboration simple e-mail the file to me.
All JPG and NEF files in the directory tree below the execution directory will be processed.
The data includes aperture, focus ring position (distance), zoom ring position (focal length), and other values.
Uses include mapping minimum aperture versus focal length for variable aperture zoom lenses.