Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow ImageSharp to determine the best pixel format on load. #235

Merged
merged 21 commits into from
Mar 22, 2022

Conversation

JimBobSquarePants
Copy link
Member

@JimBobSquarePants JimBobSquarePants commented Mar 8, 2022

Prerequisites

  • I have written a descriptive pull-request title
  • I have verified that there are no overlapping pull-requests open
  • I have verified that I am following matches the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
  • I have provided test coverage for my change (where applicable)

Description

This one's for @antonfirsov

This is a breaking change that massively optimizes memory usage for most common scenarios by allowing individual IImageWebProcessor instances to return a value indicating whether an alpha component is required in order to correctly process the image based upon the commands passed to that processor.

If an IImageWebProcessor.RequiresAlphaAwarePixelFormat(CommandCollection, CommandParser, CultureInfo) instance call returns true then we load an Image<Rgba32> as previously. If however, it returns false then we simply load Image. This allows individual image formats to deliver the best pixel format based upon the encoded input stream.

Control of the returned value can be fine-grained. For example, ResizeWebProcessor will only return true if it receives commands indicating that a BoxPad or Pad resize operation should be performed since they can have the background color set.

I've also thrown in some cleanup here to fix up the unit tests as the way they were constructed was causing issues against he non-threadsafe AWS test shim.

@codecov
Copy link

codecov bot commented Mar 8, 2022

Codecov Report

Merging #235 (3a639ad) into main (ce43688) will decrease coverage by 0%.
The diff coverage is 87%.

@@         Coverage Diff         @@
##           main   #235   +/-   ##
===================================
- Coverage    85%    85%   -1%     
===================================
  Files        71     71           
  Lines      1945   1961   +16     
  Branches    286    290    +4     
===================================
+ Hits       1672   1680    +8     
- Misses      195    203    +8     
  Partials     78     78           
Flag Coverage Δ
unittests 85% <87%> (-1%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../ImageSharp.Web/Middleware/ImageSharpMiddleware.cs 79% <76%> (-7%) ⬇️
src/ImageSharp.Web/FormattedImage.cs 80% <80%> (ø)
.../ImageSharp.Web/Caching/PhysicalFileSystemCache.cs 98% <100%> (+2%) ⬆️
...harp.Web/Processors/BackgroundColorWebProcessor.cs 100% <100%> (ø)
...rc/ImageSharp.Web/Processors/FormatWebProcessor.cs 100% <100%> (ø)
...c/ImageSharp.Web/Processors/QualityWebProcessor.cs 97% <100%> (+<1%) ⬆️
...rc/ImageSharp.Web/Processors/ResizeWebProcessor.cs 94% <100%> (+<1%) ⬆️
...ageSharp.Web/Processors/WebProcessingExtensions.cs 100% <100%> (ø)
...ching/LruCache/TLruLongTicksPolicy{TKey,TValue}.cs 26% <0%> (-18%) ⬇️
...ching/LruCache/ConcurrentTLruCache{TKey,TValue}.cs 43% <0%> (-14%) ⬇️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce43688...3a639ad. Read the comment docs.

@JimBobSquarePants
Copy link
Member Author

I'll add some tests to bring coverage up tomorrow

@tocsoft
Copy link
Member

tocsoft commented Mar 10, 2022

Is the alpha channel enough? what about grayscale source images? there is currently no way that I can see for processor to tell us they plan on doing things like drawing a 2nd image frame over the top etc.

@JimBobSquarePants
Copy link
Member Author

Yeah should be enough. The idea is that individual processors will instruct the middleware whether an alpha channel is required so a custom drawing processor would always return true unless the developer wanted more fine grained control.

@JimBobSquarePants
Copy link
Member Author

JimBobSquarePants commented Mar 17, 2022

@SixLabors/core OK. I have this working and am happy with the result. Please ignore the code coverage issues around LRU cache. I'm gonna open another PR to deal with that.

Copy link
Member

@tocsoft tocsoft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a huge fan of the name RequiresAlphaAwarePixelFormat() as it more than just alpha it also impacts other things too, user uploading a PNG in L8 and a processor wanting water mark the image with red text they would also need to enable this flag otherwise it'll end up grayscale but none of that output is anything to do with Alpha as it'd work just as well on Rgb24 as well as Rgba32.

@JimBobSquarePants
Copy link
Member Author

JimBobSquarePants commented Mar 21, 2022

Agreed there.... What would you call it then? I've tried two names so far and one was only marginally better than the other 😵😅

"RequiresDeepColorPixelFormat" perhaps?

https://en.wikipedia.org/wiki/Color_depth#Deep_color_(30-bit)

Or should we return a bit depth enum

Default
Bit24
Bit32

and load the appropriate pixel format then?

user uploading a PNG in L8 and a processor wanting water mark the image with red text they would also need to enable this flag otherwise it'll end up grayscale

It's actually impossible with the current architecture to change the color format of your input image unless you adjusted the metadata in one of the event handlers. e.g. OnProcessedAsync so if an input was gray and someone wanted to add a colored watermark then they would have to do that.

If I were writing a watermarking processor I would also always use 32bit because that avoids me having to check the parsed color for an alpha component.

@tocsoft
Copy link
Member

tocsoft commented Mar 21, 2022

I don't think we actually need to worry about 24 bit depths I was using that to demonstrate the difference really.

Some options off the top of my head:

  • RequiresWideColorPixelFormat
  • RequiresFullColorPixelFormat
  • RequiresExpandedColorPixelFormat

or inverted logic and call it

  • SupportsNaturalPixelFormat
  • SupportsAnyPixelFormat
  • SupportsMeoryOptimisedImages

@JimBobSquarePants
Copy link
Member Author

Inverting it is a fantastic idea! I’ll implement the change tomorrow.

@JimBobSquarePants
Copy link
Member Author

JimBobSquarePants commented Mar 22, 2022

After a bit of research I ended up not inverting the logic and simply renaming the method. According to https://en.wikipedia.org/wiki/Color_depth#True_color_(24-bit) RequiresTrueColorPixelFormat is accurate for 32bit RGBA.

SupportsNaturalPixelFormat would have been my favorite choice for the inverted approach but I felt there might be some sense of ambiguity from developers implementing the interface as to what that meant in technical terms. Also inverting means basing decision on an implementation detail rather than on the specific feature requirements.

@JimBobSquarePants JimBobSquarePants merged commit 21a6bc1 into main Mar 22, 2022
@JimBobSquarePants JimBobSquarePants deleted the js/optimize-pixel-memory branch March 22, 2022 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants