rawproc Version 1.4

New Tools

• cmake build system. There was one in version 1.3, but it really didn't work. 1.4's works; I used it to build the release packages. I'm going to delete the autotools files some time after I publish this release, so you might want to learn the cmake system soon. It's well-described in the README.
• spot tool: We did a 2 1/2 week road trip last summer; on the first day I discovered large spots on my sensor. Not having the resources to clean them, I wrote a spot tool sitting in motel rooms and such. The UI might be a bit tedious; I found once I got used to the key-clicks I could collect a whole bunch of spot corrections in short order.
• lens distortion and vignetting correction: These are in addition to the lensfun-based lenscorrection tool. Two new tools, lensdistortion and lensvignetting, provide both the lensfun and Adobe algorithms for manual entry of correction parameters. The tools won't show up by default, they need to enabled in the properties - see the spot doc for more information.

New/Added Behaviors

• colorspace tool: Now lets you select a profile from the current working directory, so you can make session-specific profiles and put them with the images. Accordingly, manually specifying a profile file name now has the following search order: 1) CWD, 2) the directory specified in the cms.profilepath property.
• histogram: Now will show the bucket for the pixel under the cursor. Toggle this on/off with 'b'.
• gmic: Now will embed a script in the toolchain. It's BASE64-encoded to de-conflict G'MIC syntax from rawproc's toolchain syntax.
• copy pixel: now will copy a patch as a matrix, or average the patch to a single triple.
• crop: Now has a 'camera' mode, which will take a raw image and crop the black borders per the metadata. Just noticed it's not documented; using input.raw.libraw.rawdata=crop will still and always work at raw input.

Release Artifacts

• AppImage: Built on Ubuntu 22.04 LTS, using the Appimage library exclude list. YMMV.
• exe: Window installer
• zip: Windows binaries and support files zipped up for you to place manually
• tar.gz: the source tree. Of note, I think you can also use the gihub-supplied source packages just fine...

rawproc 1.3

New

• libraw, littlecms, lensfun, and librtprocess are compiled against their latest master branch commits as of 3/1/2022. These are linked in the Appimage and Windows installer. I misstated this in the 1.2 announcement, as I'm keeping exiv2 at 0.27.3 until I work up the ambition to figure out how they moved to more recent C++...
• Edit -> Edit Metadata: There is now a dialog box to edit the metadata, which I needed to put in the tags necessary to edit TIFFs as raws. Please note that rawproc does not import all available metadata in an image, based on my own myopic view of how metadata should be managed, so your favorite tag for munging is likely not available.
• G'MIC tool: This is the signature add in 1.3. G'MIC was really the inspiration for rawproc and its toolchain organization, so it is long overdue that I include a rawproc tool that exposes all the richness of G'MIC processing to the rawproc tool chain. This tool uses libgmic, and has to be enabled in the rawproc build (./configure...) with --enable-gmic. You also need the libgmic.so.3 library somewhere in your system's library search path.
• Script tool: The more general corollary to the G'MIC tool. This tool will allow the same functionality to other script-based image processing programs. The image transport are temporary TIFFs, and properties need to be configured to include a particular scripting application. This has been tested with G'MIC and ImageMagjick, but the scheme should carry to other programs. It occurs to me as I write this that the .conf file in 1.3 doesn't have example properties; that'll probably be the first fix in 1.3.1...

Fixes

• Exposure tool crashed when disabled; wrong processing call. Thanks, @garagecoder
• Removed various Windows-specific #defines, rawproc now requires a recent GCC C++ compiler, > 5
• Changes to accommodate libraw's movement of shot_select in their data structures. To compile cleanly in most situations, rawproc now only recognizes image 0 in a multi-image raw file... :(

Notes

• See the notes for 1.2, nothing new...

rawproc 1.2

New:

• All libraries (libraw, littlecms, lensfun, exiv2, and librtprocess) are compiled against their latest master branch commits as of about 12/1/2021. These are linked in the Appimage and Windows installer.
• Tone tool now has a 1DLUT option. I implemented it to be able to apply the various camera log curves, but you can use it for any curve you want to define against all three channels. See the help file for usage
• Rotate tool now uses a float text control for <45degree rotations. I'm gradually replacing sliders with the floatctrl, it requires the mousewheel for best use but it provides better control over the value.
• the img command line program now has a verbose (-v) switch. Handy for doing rawproc-instigated batch processing.

Fixes:

• Changed the tool chain pane title from "Commands:" to "Tool Chain". Duh...
• demosaic enable/disable checkbox now works
• USE_LIBRTPROCESS conditional around the use of ApplyHLRecover, cleans up compiling without librtprocess (NOT recommended, by the way... )

Notes:

• Auto display profile switch does not work under Wayland
• If compiling, use wxWidgets 3.1.4 to avoid pesky assert dialogs that I'll eventually work out.
• There's still a segfault isolated to the Parameters pane associated with repeated opening of images in the same rawproc instance. It opens pretty quickly, so I just kill the old instance and start anew for each image. Try as I might, I cannot ferret out the root cause of this one.
• As always, use the rawproc-1.2.tar.gz sources to compile from scratch, NOT the github generated tar.gzs

rawproc 1.1

New Things:

• Use of exiv2 for JPEG/TIFF/PNG reading and writing. For raw files, I still use the libraw-provided data.
• Input histogram in curve pane.
• doublelogistic tone curve, credit @McCap at discuss.pixls.us for the algorithm.

Fixes

A bunch; grep for 'fix:' in the git log --oneline 1.0.2..HEAD for a list.

Note

It still segfaults after drag-dropping a couple of images to it for opening. Near as I can find the culprit, it's in GTK3 somewhere; in the meantime, I suggest opening each image in a fresh rawproc instance. I use a desktop shortcut, and I drag to that from either the file explorer (windows or linux), or Geeqie in linux.

rawproc 1.0.2

Mostly a convenience release, changes to facilitate workflow:

• add: longstring property template. For large property values, allows editing in a separate multiline window. Linux users may have to stretch the window size. For those retaining their old rawproc.conf, add the following lines to the end, in the Templates section:
  batch.toolchain=longstring
input.raw.default=longstring
input.raw.default.*=longstring
• fix: update openpath when app.start.path changes
• fix: display camera black values properly
• add: wildcard templates. This supports the input.raw.default.[camera] and display.cms.displayprofile.[displaynumber]. For those retaining their old rawproc.conf, add the following line to the end:
  display.cms.displayprofile.*=iccfile
input.raw.default.*=longstring (you may have already added this one for the longstring...)
• add: lenscorrection operators enabled/disabled depending on the database entry for the lens
• add: hlrecover. Highlight recovery was disabled in previous versions, I recently figured out that the librtprocess hilite_recon() function requires data to be in the range 0.0-65535.0. No parameters for this initial implementation; add this immediately after scaling the raw data to the 0.0-1.0 range.

rawproc 1.0.1

Based on feedback from pixls.us users and my son Matt, here's a version 1.0.1 that addresses most of it. I expect to do a couple more of these minor releases in the coming months, as more eyes on target sink more ships, so to speak. Thanks to all who provided such feedback.

Changes from 1.0:

• add: The AppImage will now prompt you for creating ~/.rawproc and populating it with rawproc.conf and other data, only if ~/.rawproc doesn't exist.
• add: 'app.window.rememberlast'; if set to 1, rawproc remembers its window state upon exit and restores it at open. Default: 1
• fix: Batch defaults now support proof processing. I used NEF as the raw file extension, you need to change it to suit...
• fix: icc_profile.h is now in Makefile.am, which fixes the dist.tar.gz for compiling
• fix: wse internal profiles for file saving. Thanks, Matt...
• fix: appimage library exclusion is disabled for now, added dialogs to populate ~/.rawproc with data and configuration files
• fix: wininstaller behavior, included lensfun database which will install in the same directory as rawproc.conf. rawproc.conf is excluded from uninstallation, but I'd make a copy of it somewhere else...
• fix: Windows installation destination directory. Now, the application programs can be installed and found in other directories. Note: the configuration and data files still go in c:\Users[you]\AppData\Roaming\rawproc.
• fix: lensfun database path behavior. There are now only two places where rawproc will look for lensfun databases 1) in the path specified by tool.lenscorrection.databasepath, or 2) absent the property, the directory where rawproc.conf resides.
• fix: In README, specifically state rawproc is a raw processor.
• fix: tooltip display default is now 0. You can toggle display of tooltips with the 't' key. I didn't change any of the timeouts...
• fix: Menu Help -> View Help displays main page and expanded table of contents
• fix: output.jpeg.parameters default (quality=95)
• fix: xtran to xtrans. xtran_* demosaic options no longer work, only xtrans_*. I think I'm the only affected user, so I'l just deal with it...
• fix: rawproc version in window title

Notes

1. The AppImage now includes ALL of the dynamic dependencies; I turned off using the AppImage excludelist. If this is a problem, let me know and I'll figure out how to neck it down.
2. A few folk commented on not being able to change a tool's parameters. If you click on a tool in the chain, all of its parameters and ways to deal with them are rendered in the Parameters pane at bottom-left.
3. In Windows with the installer, uninstalling rawproc will not remove rawproc.conf. In my experience, over time I've crafted a rawproc.conf that supports my particular workflows and predilections, so I decided to help preserve it in the installer behavior. For excursions, one can specify other rawproc.confs either by placing them in the same directory as the executable, or with the -c switch on the command line.
4. 1/7/2020: After a lot of angst and wailing, I think I've got a usable (well, for most, anyway) AppImage. Note the date in the filename; I'll be adding this from here on out as I get feedback and update the file based on what doesn't still work. It's still 1.0.1 code. Also, if the AppImage you've already downloaded works, leave it alone! Cripes.....

rawproc 1.0

Finally decided I had a working feature set that was reliable enough to tag as version 1.

Installation

Installation files are found at the bottom of this post. The .tar.gz file is the source code to use with a .configure && make && make install installation; DO NOT use the github-provided sources unless you want to deal with the autotools. The .Appimage is for x64_86 Linux systems; making symbolic links from it to links named rawproc and img will make the AppImage act like those respective programs. The .exe and .zip files are a 64-bit Windows installer and .zip file containing the relevant files for user placement. I've stopped making 32-bit Windows files because the hassle of doing so is not worth the time.

`Major Changes

• Add tool: adds either a number to each channel value or an image file to the image. No particular use for the first, but the second can be used to make a watermark. This is a late add for my son, who's deploying to the South Pole in a couple of days, wanted a watermark capability, and will be rather limited in his ability to download stuff.

• Auto-switched display profiles: Since I incorporated color management in rawproc, it's done its own thing regarding applying display profiles. The property display.cms.displayprofile was the source of the filename, no reliance on XATOM or colord or any of the other OS mechanisms. Keeping that premise, I added automatic display detection and profile switching. If display.cms.displayprofile.n is available, and n is the number of the display where rawproc resides, that profile will be used for the display transform. Further, the switch between profiles is automatic as rawproc is moved between displays in a multi-display setup.

• Built-in ICC profiles: In order to make rawproc more packageable, I incorporated a small set of ICC profiles that can be invoked by name, in the same places where ICC filenames can be specified. Most are linear working profiles, but there's also a V2 sRGB perceptual gamma profile for a display if one hasn't yet produced a calibrated profile for their display.

• Incorporation of LittleCMS fast_float plugin: This sped up the display transform to about one tenth of the previous time. Kind of a PITA, because it's a plugin due to licensing differences, and not yet in a stable release, but well-worth it. This is the thing that has compelled me to double-down on static linking LittleCMS, LibRaw, librtprocess, and lensfun for the packaged deployments, and to further discourage building from source. Between fast_float and keeping up with new cameras, it's the way I as the sole developer of rawproc can effectively support other users without a lot of angst.

• Filter for the EXIF dialog: exiftool is my go-to for metadata inspection, and the Edit->EXIF tool in rawproc has proven useful in using it. With 1.0, I added the ability to filter the exiftool output on a text string, which substantially improves the EXIF dialog's usefulness.

• Spectral sensitivity data: There are a couple of tool in img, the command line part of rawproc, that support extraction of spectrum data from images:
  -- cropspectrum in img: Rather than manually cropping the image to the spectrum extents, this tool finds the spectrum center and crops all but pixel rows above and below it.
  -- data-save file type: ".dat" as a file type tells rawproc to, instead of saving to an image format, save the pixel values as a comma-separated file.
  This capability is better-implemented in tiff2specdata in the ssftool repo.

• Normalization mode in blackwhitepoint: Normalization as defined for G'MIC -normalize

• Snapshots of Picpanel: The "n" key in the picture panel captures a snapshot of the panel in a separate window. This can be done multiple times, supporting comparisons.

• More-extensive copy-paste capability: Tools in the chain can be copy/cut - pasted to other places in the toolchain, as well as to other rawproc instances. Curve control points can be copied from both the curve and tone tools. A 256-bucket histogram can be copied to the clipboard from the histogram pane. RGB values at a pointed-to xy in the picture panel can be copied to the clipboard.

• USM sharpen: Applied the old-fashioned way, with a blurred copy of the image. Also added display of the calculated 3x3 matrix used for convolution sharpening.

• Batch processing: This was a relatively simple addition, with substantial benefit. I wrote a simple shell window program, wxcmd, and a batch dialog that uses it to invoke img command strings. Invoked with Edit->Batch. This capability has transformed my workflow; with it, after copying NEFs to my directory structure, I open rawproc, bring up the batch dialog, select the directory and kick off the default processing specified in the batch.toolchain property. wxcmd starts its thing on all the files, depositing the result JPEGs in the directory above. Afterward, in reviewing the proofs, if I want to do different processing to one I'll drag it from Geeqie to the open rawproc window; it open-sources the NEF file and re-applies the proof processing as the starting point for modification. Then, if I decide that processing would work better for other images, I delete their proofs, re-open the batch dialog, and this time the toolchain is the current one for the open rawproc image. Kick it off, and all the absent images are replaced by ones with better processing... Hard to describe, I might do a video just on this.

• Cutoff threshold for NLMeans denoise: This algorithm is still the bog-slow one I implemented initially. So much so that I refrained from using it. Decided to implement a luminance cutoff threshold that only applies the denoise to pixels below the value, and it helps a bit. Getting a 12-core CPU helped much more, however... :D

• Per-channel HSL color saturation: Added this capability after messing with that cat PlayRaw. I'm not a fan of HSL saturation, but if one wants to play a bit with their renditions...

• Lensfun database update: The menu selection Edit->Data Update will eventually include camconst.json, but for now it only updates the lensfun database at tool.lenscorrection.databasepath. The status of the database is included in the About box, menu Help->About.

• Per-camera input.raw.default properties: Specifying a make_model on the end of the property name, e.g., input.raw.default.Nikon_Z_6 (yes, there's a space between "Z" and "6" in the metadata) lets one specify a different default toolchain for each camera. Reverts to input.raw.default if the camera isn't matched.

• A lot of processing cleanup to support more cameras. discuss.pixls.us PlayRaws have been instrumental to this endeavor...

Residual Problems

Occasionally crashes when opening new images on top of a previous edit. I've tried and tried to trap this with gdb and -fsanitize-address, but it works just fine in both. Speculation is some sort of GTK3 problem. Mitigation is to open each new edit as a fresh rawproc instance.

rawproc 0.9

rawproc 0.9

This is the probably penultimate release of rawproc; if I do 1.0, it'll be a relatively minor addition to the current capability. Version 0.9 does just about all the things I need, and supports the rendition-oriented workflow I've envisioned since starting its development.

New Capabilities

• dcraw Independence: rawproc now has all the tools needed to process a raw image without using the dcraw processing in libraw. I now use this in my workflow by setting these properties:

  • input.raw.libraw.rawdata=crop: This tells libraw to deliver the unmodified raw data, cropped to remove the camera-specific margins.
  • input.raw.default=group:colorspace:camera,assign;subtract:camera;whitebalance:camera;demosaic:half blackwhitepoint:rgb,data: This is the tool chain inserted in the Commands tree if a raw file is opened. All of the processing up to and including demosaic is done as a group, more on that in a bit. The result is a data-scaled linear RGB image, ready for tone manipulation or other processing.

• Groups: rawproc from the start has been a memory-intensive program, by design - each tool added to the chain has it's own copy of the internal floating- point image. This design decision bared its downside with my new camera, a Nikon Z6. The increase in image size from my D7000, 16MP -> 24MP, was enough to bog down my Surface 3 tablet to almost un-useability. That, and the configuration of the new image tools to wean off of dcraw, led me to implement a new tool to combine tools in sequences that would operate on a single copy of the image. A group is just a list of semicolon-separated tool commands, with the same syntax as is used in the img command line program. Right-clicking on a group in the Commands pane will expose a couple of new menu selections, one to turn the whole group into regular individual tools, one to extract the last tool in the group and make it an individual tool placed after the group. The individual tools show up as children to the group tool, and the group Parameters pane allows one to edit the individual commands and update the group. This is an initial, expedient implementation; it introduced memory vexations that may cause crashes in using the same rawproc instance to edit sequential files. This vexation caused me, with much sadness, to disable rawproc as a drop target for opening files dragged from the file explorers. rawproc-next will address this with a better organization of the group tool internals.

• Tone Tool: All non-linear tone transforms have been collected into one tool, tone. This includes gamma, although gamma as a separate tool is still recognized but converted to tone:gamma. Of particular interest is the filmic operator. This implementation is the original equation attributed to HPDuiker, see the help file for more information. Also implemented are two variations of Reinhard, loggamma (an interesting marriage of log and gamma, from the ARIB STD-B67 specification), and log2, which is not correct and really not useful. Oh, and plain-old power gamma.

• Subtract Tool: Needed for dcraw replacement, one can now subtract 1) arbitrary values, 2) the camera metadata black value, or 3) a dark-frame file.

• librtprocess Demosaic Algorithms: What a gift from RawTherapee; just as I was about to tackle the old libraw GPL2/3 demosaic routines, the RawTherapee team started the librtprocess endeavor, to share RT algorithms with the community. As a benefactor of this largesse, rawproc now has a full demosaic suite, including x_trans algorithms for Fuji cameras. Thank you, RT team...

• wavelet Denoise: Added as an operator to the denoise tool. This is a hack of the dcraw code. It performs about the same, but behaves differently based on image size. I'd rather use NLMeans, but it's still too slow, needs optimization...

Major Changes

• Ratio-based crop: I changed the pixel-coordinate top-left-bottom-right crop specification to a percentage of width and height. This allows an original crop specification to survive insertion of a resize tool above it.

• blackwhitepoint: A recalc button was added to the slider operator for one-time auto-recalculation. Added operators are:

  • data: This option sets the black and white points to the minimum and maximum of the RGB data. A 'minwhite' modifier is included to change the whitepoint selection to the minimum of the RGB maximums, useful for blasting through blown pixels after white balance.
  • camera: This option sets the black and white points to the metadata black and maximum values. To this point, I'm not finding this option to be that useful...

• Thumbnail save properties: These enable and specify behaviors for saving thumbnail versions when an image is saved.

• 90-, 180-, and 270-degree rotations in the rotate tool, as well as rotation normalization properties.

• Patch Specification for exposure compensation: One can now select a patch in the image, along with a EV value for determining exposure compensation.

• Camera primaries from dcraw.c and camconst.json: The colorspace tool now contains an option to assign color primaries based on camera make/model lookup in dcraw.c and RawTherapee's camconst.json. rawproc will search for these files in the standard locations; putting them with the rawproc.conf file is the most straightforward alternative. The Windows installers and the AppImage come with recent versions of both files.

• Histogram: The histogram code got a major reorganization, and changes to its functionality. It now displays the floating-point center of each bucket instead of the bucket number. The 'd' key toggles between "display" and "data" extents, allowing one to see the buckets past the black and white points. The 'e' key toggles a EV scale (display only, for now). See the help file for specifics and relevant properties.

Bugs/Issues:

The only one I know of at release is the segfault behavior when sometimes opening a subsequent image after working on a previous one. It just doesn't happen when running rawproc with the debugger, leading me to believe it has a timing component. From what I have been able to gather from running a -fsanitize=address version, it probably has something to do with how I implemented the group tool. This will be the first order of business in rawproc-next.

rawproc 0.8.2

Even the smallest changes can have a big impact on the rest of the program. It turns out demosaic is rather dependent on the original orientation of the image, duh. That, in turn, exposed a few brittles about how I was handling image orientation. The resulting changes for 0.8.2 are really a bit of a hack, and for those who intend to use the demosaic tool, you need to not normalize image rotation at file open. demosaic will now do it for you after it's done its thing, so set input.orient=0. There are now three such properties, input.orient, display.orient, and output.orient, all 0|1, disable|enable. The other side effect I encountered is that crop and rotate get confused with an image whose orientation is not normalized, so you really need to normalize as soon as possible, either with input.orient=1 at file open or demosaic (hard-coded that one).

Other changes include:

• Modification (again) of the rawdata clip loop; I finally just found what libraw does and coded it equivalently.
• Histogram now stops plotting each channel when the data runs out, rather than 0s all the way to the display limit.
• Help file updates were missing from 0.8; now they're there.
• img didn't really do a good job with input parameters, so i fixed it so: 1) the property file's input.raw.parameters is loaded first; 2) the input.raw.libraw.* properties are loaded next, and finally 3) the command line input parameters are loaded last. Each of these loads will replace previously loaded parameters, so the command line trumps.

rawproc 0.8.1

It turns out I didn't do a comprehensive job of sorting out the implications of image orientation. Now that demosaic is a tool available for insertion in the chain, it is dependent on the original orientation of the raw image to use the mosaic pattern specified in the metadata. Well, that runs counter to my previous thinking on orientation, where I gave the option to do it a output with output.orient=1. So I did some refactoring and cogitating, and came up with:

• input.orient: If 1, rawproc orients the image per the Orientation tag and sets the tag to 1, removing the nastiness of "the image lays so, but really needs to be viewed thus". The default is 1, and some downstream tools that mess with image dimensions like crop and rotate depend on a normalized orientation.
• tool.demosaic.orient: If 1, tells the demosaic tool to normalize orientation after it's done. Default is 1.
• output.orient: Same behavior as before, but now the default is 1, turned on. If input.orient or tool.demosaic.orient already did it, at output the normalization call just returns without doing any work.

So, if you're just going to use the dcraw processing, leave all at defaults and it'll work. However, if you want to use the demosaic tool and you expect to have 'portrait' or other non-nominally oriented images, set input.orient=0 and let demosaic do it with the default tool.demosaic.orient=1. Oh, crop, resize, and rotate now have checks for exif Orientation=1, and will not allow themselves to join the toolchain where the previous tool has any other value.

I considered things like re-orienting the metadata bayer pattern when rotates were applied, but decided in favor of the approach above that exposes the consideration to the operator. There's learning in that...