GSoC 2024: Raw Photograph Decoding in Rust

[elbertronnie]

My name is Elbert Ronnie and I am excited to start contributing to Graphite. Throughout the 3 months of GSoC, I will be working on creating a RAW Photograph decoder in Rust which will help to convert raw photographs from different cameras to image bitmaps.

Synopsis

Most cameras capture photos in RAW format before being processed into JPEG and PNG. The most famous open source library for loading RAW files is LibRaw, but it is written in C++. All RAW decoding libraries in the Rust ecosystem are GPL-licensed and therefore cannot be used in Graphite which uses the Apache 2 license. This project aims to create a new Rust library to provide an alternative to LibRaw so that Graphite can directly import RAW files.

Benefits

Users of Graphite will be able to directly import RAW files into the editor without going through a conversion process from any external tools. This will also provide a permissively licensed RAW parser into the rust ecosystem, which could benefit many other image processing applications in the rust ecosystem.

Deliverables

• Code that can convert RAW files to an image bitmap for the formats used by Sony, Canon and Nikon.
• An end-to-end testing system that can check if the output from LibRaw and Raw-rs are the same or not.

GSoC 2024 Final Report

• 13 merged PRs
• 11.6k added lines, 6.1k removed lines

Create new library Raw-rs including a basic TIFF decoder #1757

In this PR, all the general code required to parse and extract metadata from TIFF files was committed into the repository by creating a new library with name raw-rs. Like-wise the code for reading the raw data of uncompressed ARW files was also included. A Test runner for raw files was also created that checked if the decoded raw data from this library matches exactly with the output from libraw.

Raw-rs: make decoder for ARW1 and ARW2 formats #1775

After creating the decoder for uncompressed ARW format, this PR continues the work from the previous PR wherein it added support for decoding ARW 1 and ARW 2 formats. This PR also included some notable changes in API of the TIFF decoder. A new derive macro was created that could be used to specify all the tags to be extracted at once.

Raw-rs: Add preprocessing and demosaicing steps #1796

This PR adds the subtract black step, scale colors step (scale white balance + scale to 16bit) and the demosaicing step in the raw image processing pipeline. The Linear demosaicing algorithm was used to implement the demosaicing step. For the implementation of scale colors step, the white balance needs to be derived from a matrix that converts camera's color space to sRGB. This matrix is constant for a particular camera model. This PR also adds the matrices of 40 Sony camera models in the form of toml files. A new procedural macro was also created that loads the data from the toml files and includes it as a part of the binary.

Raw-rs: add post-processing steps #1923

This PR adds convert to RGB step and the gamma correction step in the raw image processing pipeline. With this PR merged, the entire core of the raw image processing pipeline was complete. This library could be used to convert actual raw image into image bitmaps although only for a few models. A sample image that was used in the tests is given below:

Raw-rs: use camera white balance when available #1941

This PR adds code to read the white balance data of the camera from the metadata of the TIFF file. Some files have this information contained within while others don't. This PR changes the white balance selection strategy to use the white balance information from the metadata if it available and fallback to calculating it from the color space conversion matrix if unavailable. The same image after using white balance from the metadata is given below:

Raw-rs: Flip and rotate image based on camera orientation #1954

This PR adds a last optional step in the image processing pipeline which is transform step. This step reads the orientation from the metadata and appropriately rotates and flips the image to orient the image accordingly.

Raw-rs: Refactor to run multiple steps in a single loop #1972

This PR improves the performance of the processing steps by reducing the number of times it has to loop throughout the image. The external API to use the library was changed in such a way that it tries to combine multiple steps in a a single loop wherever possible while still providing the same level of flexibility to the user. The final API is shown below:

let subtract_black = self.subtract_black_fn();
let scale_white_balance = self.scale_white_balance_fn();
let scale_to_16bit = self.scale_to_16bit_fn();
let raw_image = self.apply((subtract_black, scale_white_balance, scale_to_16bit));

let convert_to_rgb = raw_image.convert_to_rgb_fn();
let mut record_histogram = raw_image.record_histogram_fn();
let image = raw_image.demosaic_and_apply((convert_to_rgb, &mut record_histogram));

let gamma_correction = image.gamma_correction_fn(&record_histogram.histogram);
if image.transform == Transform::Horizontal {
	image.apply(gamma_correction)
} else {
	image.transform_and_apply(gamma_correction)
}

[elbertronnie]

Week 1 Report

General Information

All RAW formats are essentially an TIFF file. TIFF is a format to specify metadata along with the raw pixel data in the form of pairs of tag and value. But the exact tags used varies with manufacturer. Even the raw data is stored in a format that depends on camera model and manufacturer. A good analogy would be compare TIFF with JSON where each manufacturer has different JSON schema. EXIF is an extension to TIFF keeping the same format but adding additional tags to support more use cases. The entire list of tags can be found at EXIF Tags

The raw data does not represent the image we are used to. Instead of having 3 channels, it has only a single channel which is able to provide information about all 3 colors. The information is arranged in a format known as the Bayer Color Filter Array which is shown in the figure below. The process of converting this Bayer CFA to an RGB image is known as Debayering or Demosaicing.

To convert a RAW image to an image bitmap, the following steps need to be performed:

• Decode TIFF and extract metadata
• Identify the manufacturer and camera model
• Find the raw data in TIFF file and pass it through the appropriate decoder
• Apply Preprocessing steps (subtract black, scale colors, etc)
• Apply Debayering algorithm to convert to an image of 3 channels
• Apply Postprocessing steps (mix green, highlights, etc).

Community Bonding Period

• During the community bonding, I finished up an incomplete PR about to adding Area and Centroid Nodes and then started to map out the steps in the pipeline which are of immediate importance to Graphite. I had analyzed all the steps till Demosaicing.
• Eventually I started to code the very stage of the pipeline - the TIFF decoder. This was originally supposed to be delegated to a library, but I soon found them to be lacking in many proprietary EXIF tags and soon came to the decision to implement it myself. The approach taken is slightly different from existing rust libraries. Each Tag has a particular type which instructs on how to decode it. Therefore after decoding the values of each tag, it is automatically available in the form of a rust-native type.
• The decoder for uncompressed ARW format is complete.

Tasks completed this week

• Made a testing framework which will download the raw images if not present already and run it through raw-rs and libraw-rs. It will check if the output images from both of them are the same or not. For each image, two checks are run. The first one runs after the decoding the raw data wherein it will compare both images in bayer CFA form. The second one runs with the final RGB image. All the code from community bonding period and this task was merged in Create new library Raw-rs including a basic TIFF decoder #1757
• The decoder for ARW 1 format is complete.

Tasks for next week

• Make the decoder for ARW 2 format. This will require additional code for the TIFF decoder to process the curve table.
• Enhance the TIFF decoder to load multiple tags in a single request within a single struct.
• Write code for the Subtract Black Preprocessing step.

[elbertronnie]

Week 2 Report

General Information

Sony's RAW images follows an EXIF-based format known as ARW (Alpha RAW). This format has gone through multiple versions with each version storing it a different manner. At this point in time, Raw-rs has added support for 3 versions of ARW:

ARW 1

This format allows for 12 bits per sample and stores the Bayer CFA in an interleaving format so that all the pixels of same color are together. It also uses a form of differential encoding where only pixel differences of consecutive pixels of same color are stored to reduce space.

ARW 2.3.1

This format also allows for 12 bits per sample and stores the Bayer CFA in an interleaving format. But the major difference is that it uses lossy compression to store the values. The file also provides data to generate a tone curve. This tone curve is used to apply a mapping that is similar to (but not exactly) an exponential/logarithmic transformation (exponential for decoding and logarithmic for encoding). Such transformation reduces the range of values to be stored in the file thereby allowing it to be stored by fewer bits.

ARW 2.3.5

This format allows for 12 bits or 14 bits depending the camera model. It stores the values of Bayer CFA as it is, in a byte-aligned manner with no interleaving or compression.

Tasks completed this week

• Fixed an error related to the incorrect size of RAW image in ARW 1 format. Apparently some cameras models require hard-coding the sizes since sizes given in the metadata are incorrect.
• The decoder for ARW 2 format is complete along with the processing of the curve table. Currently, windsock.arw also passes the tests.
• Converted the Tag type to a Tag trait. Each tag that was previously a constant will now be a type. But creating a derive macro is still left.

Tasks for next week

• Complete the derive macro for Tag trait.
• Write code to identify the camera model and the appropriate decoder from the Makernotes tag.
• Write code for the Subtract Black Preprocessing step.

[elbertronnie]

Week 3 Report

Tasks completed this week

• Completed the derive macro for Tag trait.
• Completed the code to identify the camera model and the appropriate decoder.

Tasks for next week

• Write code for the Subtract Black Preprocessing step.
• Write code for the Scale Colors Preprocessing step.

[elbertronnie]

Week 4 Report

General Information

Moving forward in the pipeline, the next steps after decoding the raw data within the TIFF file are as follows:

Raw to Image

This is a simple step where the raw data is converted to the standard form of 3-channel RGB image. But only one channel contains the information for each pixel. This steps also handles cropping of the image. The dimensions of cropping is usually provided in the metadata.

Subtract Black

Every camera's sensor will always have some kind of zero error. Essentially even when no light passes through the sensor, the sensor could still output a value that is greater than 0. This error is usually removed by subtracting a value from the pixels taking it back to 0. This value is known as the Black Level. This information is usually present in the metadata within the raw file. But this is usually a single value for all pixels. If more correctness is required, then an dark frame image is taken by placing the lens cap to allow no light. This dark frame is used for subtracting values for each individual pixel.

Scale Colors

The color intensities recorded by the sensor is different from what is felt in the eyes. You can consider it as if camera sensors record colors in a different color space than the standard RGB. This steps scales the colors in a way so to match the standard RGB color space. The transformation matrix is different for every camera model.

The values in the raw file can use 12 or 14 bits per sample, but the default RGB image uses a 8 bits to represent a single color. This step also scales the values to the expected bps.

Tasks completed this week

• Added Subtract Black step
• Added Scale Colors step

Tasks for next week

• Write code for the raw to image step
• Write code for simple linear demosiacing

[elbertronnie]

Week 5 Report

Tasks completed this week

• Added Raw To Image step

Tasks for next week

• Write code for simple linear demosiacing

[elbertronnie]

Week 6 Report

Journal

After taking a break of 2 weeks, I resumed my work of creating raw-rs. In the first half of the week I was able to complete the algorithm for linear demosaicing. It simply takes the average of the neighboring pixels which have the color which is supposed to be computed. I ran the program through sample images to check the output, but the images were much darker than expected. After spending some time with the codebase, I realized that I had not done a color space conversion that happens in post-processing. This was also the time that I realized that even libraw does not output correct images by default. By passing a parameter that states to use camera's white balance instead of the one derived from camera_to_xyz, the output images were close to the expected one. I created code to store image output of raw-rs and libraw-rs in an output folder so that it will be easier to compare them.

I moved on to part of code that loads the camera_to_xyz matrix. After a few initial attempts of trying out different ways, I and Keavon decided that we will use 1 toml file per camera model which will be required to be loaded merged at compile time. So I spent the rest of the week working on a macro to load the camera data from all the toml files. During the attempts, I also realized that the downloader crate (which was used in tests) was LGPL licensed. This was a mistake from my part for not checking this correctly. I will be implementing the same with barebones reqwest and std::io::copy.

Tasks completed this week

• Added Linear Demosaicing step
• Wrote a macro to load the camera data from all toml files

Tasks for next week

• Generate all the toml files from camera metadata
• Write code for the final color space conversion in post-processing

[elbertronnie]

Week 7 Report

Journal

In the previous week, I created a macro to load the camera data from all the toml files. But these toml files are not yet created. This week, I continued on by creating a script to extract the camera data from DNG files and store it in the form of toml files. This will also make it easier to add new models later on. Now the only step left is to get the DNG files in the first place. To achieve this, I downloaded sample RAW images of various camera models so as to convert them into DNG files using Adobe DNG Converter. Currently only RAW images of half of all the models is currently downloaded. I will do the rest of them in the next week.

I was not able to give 30 hours this week due to starting of a new semester in my college. I plan to compensate for it by putting more hours in the next week.

Tasks completed this week

• Wrote a script to extract the camera data from DNG files

Tasks for next week

• Generate all the toml files from camera metadata
• Write code for the final color space conversion in post-processing

[elbertronnie]

Week 8 Report

Journal

Previously, only half of all the models were downloaded. This week I have downloaded all the remaining ones. Some camera models don't even have a RAW sample and therefore I have skipped them. Then I ran all the DNG files through the script to extract the color matrix. Some models have "MODEL-NAME" in their model tag. Since such information is of no use, I have have skipped them. In total, color matrix for 40 camera models of Sony have been extracted. The images used in the test suite is a part of these 40 models so no extra effort was required there. Finally I resolved some small errors in #1796 made it ready for review.

Since the start of my college I was not able to give 30 hours per week in a consistent manner, which is why I have decided to extend the deadline for GSoC so only 20 hours per week will be required from me from here on.

Tasks completed this week

• Generate all the toml files from camera metadata

Tasks for next week

• Write code for the final color space conversion in post-processing

[elbertronnie]

Week 9 Report

Journal

I made some changes to the existing PR #1796 based on review wherein the camera matrix will stored in decimal format instead of integers. Henceforth, the long standing PR was finally merged. I continued to work in the next phase: Post-processing by implementing the Convert to RGB step. But the result was not I was expecting. The final image is still darker than required. I am yet to find exact reason for this which is what I will be doing in the next week.

Tasks completed this week

• Write code for the final color space conversion in post-processing

Tasks for next week

• Find the reason for difference in expected image
• Implement code for taking white balance from camera metadata instead of the toml files.

[elbertronnie]

Week 10 Report

Journal

In the past week, I was able to find out the reason for incorrect images. I had applied the gamma correction step after the scaling from 16 bits to 8 bits whereas the opposite order should have been followed. Hence I added a new step in the pipeline named as gamma correction. It is slightly sophisticated than a simple exponentiation. It involves calculating the histogram of the image which is used for generating the gamma curve table. This is used to apply the transformation and finally convert from 16 bits to 8 bits with a simple bit shift.

With all the above steps done, The entire pipeline is complete and raw-rs is ready to be used for generating final images for a very small portion of Sony Cameras. Here is a output from raw-rs for the the file blossoms.arw which is used as a test case in the CI:

Now that the pipeline is complete, my next steps would be to implement some cases in previous steps that I missed like using white balance data from the metadata of file or fixing the orientation of the image. Eventually support for almost all Sony camera will be added and each step in the pipeline will become its own Node in Graphite's Node Graph.

Tasks completed this week

• Implement the gamma correction step

Tasks for next week

• Implement code for taking white balance from camera metadata instead of the toml files.
• Add a step to rotate and flip image based on the orientation data.

[elbertronnie]

Week 11 Report

Journal

In the past week I implemented code to extract the white balance parameters from the camera metadata and use them instead of deriving it from the camera matrix within this PR #1941. The PR has been already merged. The same blossoms.arw image now looks much better with more accurate colors:

After this, I started implementing the image transforms that needs to be applied based on the orientation of the camera like flip and rotation in #1954. Only half of this task is currently complete. The code for extracting the camera orientation from metadata is complete and applying the transformation is left which will be done in the next week.

Tasks completed this week

• Implement code for taking white balance from camera metadata instead of the toml files.

Tasks for next week

• Add a step to rotate and flip image based on the orientation data.
• Make raw-rs tests run in parallel.

[elbertronnie]

Week 12 Report

Journal

In the past week I completed the step to transform the image and its corresponding PR #1954 had also got merged. Henceforth, I had a discussion with Keavon and TrueDoctor regarding the final architecture of the steps in raw-rs to maximize its performance while still keeping it modular enough for different kinds of use cases. The structure of its equivalent Graphite Node was also discussed. Currently all the steps require a full loop through the image to do the operation. The new architecture will minimize the number of loops by grouping operations in closures similar to how rust iterators work under the hood.

From here on the focus will be shifted to maximize performance and add support for as many cameras as possible. The time taken to run a single test image was very high and would definitely not scale well when many more are added. Hence I changed the code to run the tests in a parallel manner in #1968. This along with the increased performance benefits by the new architecture, should make it easy to scale.

Tasks completed this week

• Add a step to rotate and flip image based on the orientation data.
• Make raw-rs tests run in parallel.

Tasks for next week

• Optimize all the steps to run in as few loops as possible
• Add test images for more cameras

[elbertronnie]

Week 13 Report

Journal

After taking a leave for 3 weeks due to exams in my collage, I have continued to work where I had left off. This week I changed how each step in the pipeline functions. Previously each step was a function that took a RawImage and returned RawImage after applying the transformation. This meant that each step was making a single loop through the image. To prevent this, each step has been modified to return a transformation function instead. The functions can be chained together to run in a single loop similar to how Rust iterators work. Each transformation function could be skipped or replaced with a custom function as the user pleases. This allows the user of this library to customize this pipeline very easily while still retaining performance as multiple transformation functions could be chained in a single apply call.

The final API looks like:

let subtract_black = raw_image.subtract_black_fn();
let scale_white_balance = raw_image.scale_white_balance_fn();
let scale_to_16bit = raw_image.scale_to_16bit_fn();
let raw_image = raw_image.apply((subtract_black, scale_white_balance, scale_to_16bit));

let convert_to_rgb = raw_image.convert_to_rgb_fn();
let mut record_histogram = raw_image.record_histogram_fn();
let image = raw_image.demosaic_and_apply((convert_to_rgb, &mut record_histogram));

let gamma_correction = image.gamma_correction_fn(&record_histogram.histogram);
if image.transform == Transform::Horizontal {
	image.apply(gamma_correction)
} else {
	image.transform_and_apply(gamma_correction)
}

The scale colors step has also been split into two different steps: scale_white_balance and scale_to_16bit. This was done since scale_to_16bit is a compulsory step before demosaicing without much customization but scale_white_balance could be heavily customized by the user.

Henceforth the PR #1972 was made ready to review.

Tasks completed this week

• Optimize all the steps to run in as few loops as possible

Tasks for next week

• Create Graphite Node
• Add test images for more cameras