Formalize Apple's CgBI chunk

[nigeltao]

Apple has an unofficial PNG extension, used by their tools (a modified pngcrush: xcrun -sdk iphoneos pngcrush -iphone) to create iOS app assets.

If you're updating the PNG spec to 3.0, consider making it official. It might not be the extension I would have designed, but if there's millions of iOS devices out there, there's zillions of PNG/CgBI images already out there in the real world.

I don't have official documentation, but here's what others have deduced:

• a CgBI chunk before IHDR (a serialization of a CGBitmapInfo struct).
• an optional iDOT chunk before IDAT.
• BGRA not RGBA.
• premul alpha not non-premul alpha.
• Raw zlib (i.e. deflate), no zlib header and no Adler-32 checksum.

https://iphonedev.wiki/index.php/CgBI_file_format

https://github.com/DHowett/pincrush

https://www.hackerfactor.com/blog/index.php?/archives/895-Connecting-the-iDOTs.html

http://jongware.com/pngdefry.html
This has an example PNG/CgBI image.

https://github.com/jakubknejzlik/cgbi-to-png
This has an example PNG/CgBI image. Not sure if their code does the premul -> non-premul conversion properly.

https://axelbrz.com/?mod=iphone-png-images-normalizer
Not sure if their code does the premul -> non-premul conversion properly.

[DavidBuchanan314]

Note that Apple's variant is patented: https://patents.google.com/patent/US20080177769

Also IIUC any official documentation, if it exists, is under NDA.

[svgeesus]

CgBI is a critical, private, safe-to-copy chunk. If standardized, it would become CGBI.

from 14.1 Additional chunk types:

New public chunks will be registered only if they are of use to others and do not violate the design philosophy of PNG. Chunk registration is not automatic, although it is the intent of the Registration Authority that it be straightforward when a new chunk of potentially wide application is needed. The creation of new critical chunk types is discouraged unless absolutely necessary.

From 14.3.1 Ordering of critical chunks:

Critical chunks may have arbitrary ordering requirements, because PNG editors are required to terminate if they encounter unknown critical chunks.

So as soon as this chunk was standardized, it would be a backwards-incompatible change because all older decoders would terminate on encountering it. (They would now, which is fine because it is a critical private chunk).

From Guidelines for new chunk types

Do not define new chunks that redefine the meaning of existing chunks or change the interpretation of an existing standardized chunk, e.g., do not add a new chunk to say that RGB and alpha values actually mean CMYK.

CgBI literally does that, redefining IDAT to not contain unpremultiplied RGBA but actually, premultiplied BGRA (note that it is not the alpha vale which changes, in premultilied alpha, but the color component values)

Note that Apple's variant is patented: https://patents.google.com/patent/US20080177769

From Introduction

Freedom from legal restrictions: no algorithms should be used that are not freely available.

It does not seem that this PNG-like asset has been designed with a view to possible later standardization.

[ProgramMax]

I contacted some friends at Apple to see if anyone from Apple would like to join the PNG WG.

But for the moment, I think this issue about the CgBI chunk can be closed. @svgeesus is right that if we wanted to add support for premultiplied and/or BGRA, they should be new chunks anyway.

I'll close for now. If we want to revisit BGRA / premultiplied alpha, lets open a new issue.

[nigeltao]

Note that Apple's variant is patented: https://patents.google.com/patent/US20080177769

https://patents.google.com/patent/US20080177769 links to https://patentcenter.uspto.gov/#!/applications/11650712 which says that that patent expired on 06/11/2018.

In any case, I am following up with a professional patent lawyer.

CgBI is a critical, private, safe-to-copy chunk. If standardized, it would become CGBI.

I'm not asking to create a new CGBI chunk (with a capital G) that nobody already uses and nobody would use (e.g. I wouldn't expect Apple to update their tools just because the PNG 3rd edition said so). I'm asking for the 2022 PNG specification to match PNG as used in 2022.

Yes, it might require an asterisk to the "chunk type's second byte denotes private/public status" rule: "an exception was made for the CgBI chunk for historical reasons blah blah blah". This would be similar to formalizing APNG in the spec (#26). Surely the spec will mention acTL chunks, not aCTL chunks, because the spec and reality should match but we can't change reality. We can't retroactively change all of the APNG files and software already out there.

I develop software that processes PNG images (amongst other things). In practical terms, this means things that start with an 8 byte "\x89PNG\x0d\x0a\x1a\x0a" magic string and/or have a .png file name extension. These PNG/CgBI images match that magic string and can be flung around as foo.png. If the PNG specification does not tell me how to handle these, it just means that the PNG specification is less relevant to my software and I will favor unofficial documentation instead.

A decade or two from now, if a digital archaeologist came across a PNG/CgBI image (there's possibly orders of magnitude more PNG/CgBI images than APNG images in existence, because of iOS), their job would be easier if the official PNG specification described how to decode it.

So as soon as this chunk was standardized, it would be a backwards-incompatible change because all older decoders would terminate on encountering it. (They would now, which is fine because it is a critical private chunk).

I don't see the change as backwards-incompatible: your quote from the spec says "encounter unknown critical chunks", not "encounter unknown critical public chunks". Older decoders used to terminate on encountering it. They can still do after PNG/CgBI gets written up in the spec. CgBI support can be optional in PNG 3rd edition (with asterisks where necessary). Same behavior (no compatibility change) either way.

But for the moment, I think this issue about the CgBI chunk can be closed. @svgeesus is right that if we wanted to add support for premultiplied and/or BGRA, they should be new chunks anyway.

I am reminded of APNG's long road to being officially blessed. As you know, the PNG spec authors were reluctant to embrace it for many years. If I remember correctly, APNG folks were officially told (by Glenn??) that PNG isn't animated, so APNG should use a different magic string. But given a critical mass of software support (Firefox first and other browsers later in APNG's case, iOS in CgBI's case), popular implementation trumps specification.

[randy408]

If the PNG specification does not tell me how to handle these, it just means that the PNG specification is less relevant to my software and I will favor unofficial documentation instead.

Is there precedent for standardizing a reverse-engineered (and incomplete) specification? If someone had to tell us how to handle Apple's variant of PNG it would be the company who invented it.

[ProgramMax]

I'm asking for the 2022 PNG specification to match PNG as used in 2022. ... This would be similar to formalizing APNG in the spec (#26).

That's a good argument. You convinced me, actually.

I'll have to reflect for a bit.

Would this be a good candidate for an extension rather than the core spec?
Is the public/private bit no longer important?

[ProgramMax]

I'm re-opening since 1.) the patent may have expired, and 2.) the public/private bit may not actually be a deciding factor.

I still hope we can get input from a Apple.

[nigeltao]

The related iDOT chunk (and the same https://www.hackerfactor.com/blog/index.php?/archives/895-Connecting-the-iDOTs.html link) gets mentioned on #54 (comment).

[nicolas17]

I have been trying to reverse-engineer this chunk properly, since nobody ever figured out what the chunk content means, most third-party tools just look at the chunk's mere presence and assume "raw zlib, premultiplied alpha, BGRA", which isn't correct (some don't even fix the premultiplied alpha which is worse). While experimenting, I found even Apple software doesn't handle it consistently. I easily made a png file that is shown differently on Preview vs Safari on Mac, and another that is shown differently on Mac vs iOS.

The one thing I found for sure is that bit 0x1 in the first byte of the chunk data (ie. bit 0x10000000 if you consider the chunk as containing a big-endian uint32) is what determines whether the file is using raw zlib. Every third-party tool I found to deal with Apple's PNGs assumes the IDAT will have headerless raw zlib if the CgBI chunk is present, without checking if that bit is set.

I also spent days trying to understand bit 0x2 in the first byte; it maps to an internal flag called APPLE_FILTER_SUB_ONLY but I can't figure out what it does, either by decompiling the code or experimenting with setting it.

...do you really want to put this mess into a standard spec?

[randy408]

most third-party tools just look at the chunk's mere presence and assume "raw zlib, premultiplied alpha, BGRA", which isn't correct

This and the fact that not even Apple implements it consistently are more reasons I won't be supporting it in spng.

I also spent days trying to understand bit 0x2 in the first byte; it maps to an internal flag called APPLE_FILTER_SUB_ONLY but I can't figure out what it does, either by decompiling the code or experimenting with setting it.

It could mean some or all scanlines are encoded with filter type 1 (Sub), some tools might tell you the filter type for each scanline (spng exposes this through spng_get_row_info()).

[ProgramMax]

Thank you for your research, @nicolas17 . This is both good to hear and also unfortunate.
I agree with @randy408 .

On one hand, the PNG Working Group has agreed to accept private chunks into the standard in rare occasions where it makes sense, such as existing APNGs. This opens the doors for accepting the private CgBI chunk.

But on the other hand:

1. I'm trying to get Apple involved but haven't heard back yet.
2. If Apple itself is inconsistent with the chunk, it is difficult to argue just how standard-ready it is.
3. While the US patent may have expired, I'm not sure how this works outside of the US. I would need to rope in W3C's lawyers before we could even consider taking steps here. (And again, without Apple's involvement this step requires extra care.)

@nicolas17 , do you happen to know if most (almost all?) CgBI-supporting decoders are consistent and only a handful are off? For example, maybe only a recent version of MacOS changed behavior and it is consistent-enough to not discredit.

If we want to support premultiplied alpha and/or BGRA, it might be best for us to introduce those as a new chunks in a clean-room approach, not considering what Apple has done. Unfortunately, that rather defeats the purpose of accepting widely-adopted, existing chunks.

[Crissov]

Do CgBI and iDOT offer any benefit over existing capabilities of PNG? It does not sound like it, so I do not see a compelling reason to add it to the standard, even without considering the lack of documentation and interoperability.

[ProgramMax]

There certainly are some situational benefits:

• BGRA can save swizzeling
• Premultiplied alpha can save computation
• Deflate stream can file size

That said, it might be best for us to consider these things outside of Apple's chunks.

I still have not heard back from Apple. And the original intent (IIUC) was to recognize a defacto standard. But if there are multiple interpretations within Apple then I'm not sure the defacto standard idea applies.

It would be great to have Apple's involvement to form a standard, single interpretation. But as-is, we might want to file separate issues to consider each of those benefits individually and not use Apple's chunks.

[nigeltao]

@nicolas17

The one thing I found for sure is that bit 0x1 in the first byte of the chunk data (ie. bit 0x10000000 if you consider the chunk as containing a big-endian uint32) is what determines whether the file is using raw zlib.

I assume you mean 0x1000000 (with 6 zeroes after the one) instead of 0x10000000 (with 7 zeroes after the one).

In any case, I believe CgBI stands for CGBitmapInfo. https://developer.apple.com/documentation/coregraphics/cgbitmapinfo is Apple's official documentation. https://developer.apple.com/documentation/coregraphics/cgimagealphainfo sounds related, and there's constants within that like https://developer.apple.com/documentation/coregraphics/cgimagealphainfo/kcgimagealphapremultipliedlast that says "The alpha component is stored in the least significant bits of each pixel and the color components have already been multiplied by this alpha value. For example, premultiplied RGBA. [emphasis added]".

That kcgimagealphapremultipliedlast web page doesn't actually give a numerical value, but https://docs.rs/core-graphics/0.14.0/i686-apple-darwin/core_graphics/base/constant.kCGImageAlphaPremultipliedLast.html gives a value of 1.

In terms of the CgBI PNG format, maybe that uint32 is little-endian instead of big-endian? [Edit: sorry big-endian, "1 = premul alpha" can't also mean "1 = raw zlib/deflate"]

[nigeltao]

@ProgramMax

I still have not heard back from Apple. And the original intent (IIUC) was to recognize a defacto standard. But if there are multiple interpretations within Apple then I'm not sure the defacto standard idea applies.

Yes, the original intent is to recognize a de facto standard used today by billions of devices. Minting new chunks in a clean-room approach doesn't address that.

"Multiple interpretations" might merely be down to some Apple software using ther internal library version N (and using fallback behavior for unknown feature bits / chunks, or simply a bug whose fix wasn't backported) and other Apple software using version N+1.

It would be great to have Apple's involvement to form a standard, single interpretation.

Yes, that would be the ideal scenario.

[nigeltao]

@ProgramMax

Deflate stream can file size [sic :-)]

A raw deflate stream (instead of zlib) can also be faster to encode and decode. The Adler-32 checksum computation is cheap but it isn't free and can show up in CPU profiles.

[nigeltao]

@Crissov

Do CgBI and iDOT offer any benefit over existing capabilities of PNG?

I believe that iDOT also allows for parallelized decoding. The image is split into multiple stripes (same width, smaller height) and encoded independently (or, equivalently, the zlib stream is reset at stripe boundaries). Multi-threaded decoding can be significantly faster (in terms of wall-clock time).

Of course, iDOT is not the only way to allow parallel decodes. See also #54, but it would still involve a new chunk type.

[nicolas17]

It could mean some or all scanlines are encoded with filter type 1 (Sub)

I made a png using paeth filter on all scanlines, I added a CgBI chunk with the 0x2000_0000 bit set, I hooked _cg_png_read_row and confirmed png_ptr->apple_flags == 2 at that point (that's APPLE_FILTER_SUB_ONLY), and the image still looks the same; it certainly doesn't have the distortion I would expect from applying the wrong filter during decoding. So I'm kind of stumped.

I didn't really make any more research progress during January though, and I have been procrastinating a proper writeup...

[nicolas17]

I assume you mean 0x1000000 (with 6 zeroes after the one) instead of 0x10000000 (with 7 zeroes after the one).

Decompiling Apple's pngcrush shows code like if ((cgbi >> 0x1c & 1) != 0). That's 0x1000_0000 with 7 zeroes, and actually my mistake is saying "bit 0x1 in the first byte" (it should be 0x10), good catch :)

"Multiple interpretations" might merely be down to some Apple software using ther internal library version N (and using fallback behavior for unknown feature bits / chunks, or simply a bug whose fix wasn't backported) and other Apple software using version N+1.

An example: when Apple's pngcrush adds CgBI to a file, it sets the last byte to 0x02 or 0x06 depending on whether there is an alpha channel. When reading a file with CgBI (eg. converting it back to a normal png), it seems to ignore the last byte altogether. The ImageIO framework does look at the last byte, and software like Safari renders images differently depending on the value. So that's Apple's ImageIO and pngcrush interpreting different values of the last byte in different ways (one ignores it).

[ronaldtse]

My +1 here as we just encountered this issue with pngchunk which declares the iDOT chunk as invalid:
metanorma/pngcheck-ruby#17

We may create an extension to pngchunk to deal with iDOT as described at the hackerfactor link.

[ProgramMax]

The 4th letter being capitalized means it is unsafe to change the file if the chunk isn't understood. But if a user is using pngcheck-ruby only to read the file (and not alter it), it should be able to skip ancillary chunks it does not understand. That might be an issue to fix in pngcheck-ruby.

[ronaldtse]

Thanks @ProgramMax for the tip. Since pngcheck is an associated part of the reference libpng library, we will have to either fork pngcheck to allow a "read-only" validation, or implement it at the binding level that encountering iDOT is acceptable for read-only purposes.