Formatting a png image file takes too long

[klouvas]

What are you trying to achieve?

Could you help me understand why a png image of 6MB takes 4x time longer compared to an 20MB image?
Also could you help me with the questions below:

• can we skip compression if already compressed at higher level?
• can we detect compression through the library?

Have you searched for similar questions?

Yes i have already checked:

• sRGB images and hasAlpha #526
• Resulting png image is larger than source jpg image when resizing down. #2213
  Also most of the issues from https://github.com/lovell/sharp/issues?q=png+compress

Are you able to provide a minimal, standalone code sample that demonstrates this question?

// Environment:
//   MacBook Pro (13-inch, 2017, Two Thunderbolt 3 ports) 
//   2,5 GHz Dual-Core Intel Core i7
//   MacOS Catalina 10.15.7 (19H15)
// node version v12.16.1
// sharpjs version 0.27.1
const sharp = require('sharp');
const { pipeline, PassThrough } = require('stream');
const { createReadStream, createWriteStream } = require('fs');
sharp.cache(false);
sharp.concurrency(1);

function resizer(readStream, { format, quality }) {
  const metaDataStream = sharp();
  const outputStream = new PassThrough();

  readStream.pipe(metaDataStream);

  metaDataStream.metadata().then(info => {
    console.log({
      ...info,
      xmp: Buffer.from(info.xmp, 'binary').toString()
    });
    let transformer = metaDataStream.clone();
    transformer = transformer.toFormat(format, { quality });

    pipeline([transformer, outputStream], (err) => {
      err && outputStream.emit('error', err);
    });
  }).catch(err => outputStream.emit('error', err));

  return outputStream;
}

function resizeImage(inputImagePath, outputImagePath) {
  console.time(`resize-${inputImagePath}`);
  const readStream = createReadStream(inputImagePath)
  const outputStream = createWriteStream(outputImagePath);

  readStream.on('error', console.error);
  outputStream.on('error', console.error);

  const resizeStream = resizer(readStream, { "format": "png" });
  outputStream.on('close', () => console.timeEnd(`resize-${inputImagePath}`));

  resizeStream.pipe(outputStream);
}

resizeImage('6mb.png', 'test-6mb.png'); //  resize-6mb.png: 12394.095ms
resizeImage('20mb.png', 'test-20mb.png'); //   resize-20mb.png: 3478.747ms

Are you able to provide a sample image that helps explain the question?

I cannot provide the 6mb picture due to copyrights and i cannot attach the 20mb since it's too big. Those are some informations based on the code snippet above:
6mb metadata info:

 {
    format: 'png',
    size: 6709910,
    width: 3840,
    height: 3840,
    space: 'srgb',
    channels: 3,
    depth: 'uchar',
    density: 72,
    isProgressive: false,
    paletteBitDepth: 8,
    hasProfile: false,
    hasAlpha: false,
    xmp: '<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?> <x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 6.0-c002 79.164488, 2020/07/10-22:06:53        "> <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> <rdf:Description rdf:about="" xmlns:xmp="http://ns.adobe.com/xap/1.0/" xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/" xmlns:stRef="http://ns.adobe.com/xap/1.0/sType/ResourceRef#" xmp:CreatorTool="Adobe Photoshop 22.0 (Windows)" xmpMM:InstanceID="xmp.iid:997FF380648611EB97D5B947DBE4422A" xmpMM:DocumentID="xmp.did:997FF381648611EB97D5B947DBE4422A"> <xmpMM:DerivedFrom stRef:instanceID="xmp.iid:997FF37E648611EB97D5B947DBE4422A" stRef:documentID="xmp.did:997FF37F648611EB97D5B947DBE4422A"/> </rdf:Description> </rdf:RDF> </x:xmpmeta> <?xpacket end="r"?>'
  }

20mb.png metadata info:

   {
    format: 'png',
    size: 21141605,
    width: 5891,
    height: 2271,
    space: 'srgb',
    channels: 4,
    depth: 'uchar',
    density: 72,
    isProgressive: false,
    hasProfile: false,
    hasAlpha: true,
    xmp: '<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 5.4.0">\n   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">\n      <rdf:Description rdf:about=""\n            xmlns:exif="http://ns.adobe.com/exif/1.0/"\n            xmlns:tiff="http://ns.adobe.com/tiff/1.0/"\n            xmlns:xmp="http://ns.adobe.com/xap/1.0/"\n            xmlns:exifEX="http://cipa.jp/exif/1.0/"\n            xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/">\n         <exif:ShutterSpeedValue>10815/929</exif:ShutterSpeedValue>\n         <exif:ColorSpace>1</exif:ColorSpace>\n         <exif:FocalLength>103/25</exif:FocalLength>\n         <exif:PixelXDimension>6048</exif:PixelXDimension>\n         <exif:SensingMethod>2</exif:SensingMethod>\n         <exif:SubsecTimeDigitized>581</exif:SubsecTimeDigitized>\n         <exif:FocalLenIn35mmFilm>33</exif:FocalLenIn35mmFilm>\n         <exif:ExposureTime>1/3195</exif:ExposureTime>\n         <exif:WhiteBalance>0</exif:WhiteBalance>\n         <exif:ExifVersion>0221</exif:ExifVersion>\n         <exif:BrightnessValue>8727/794</exif:BrightnessValue>\n         <exif:ComponentsConfiguration>\n            <rdf:Seq>\n               <rdf:li>1</rdf:li>\n               <rdf:li>2</rdf:li>\n               <rdf:li>3</rdf:li>\n               <rdf:li>0</rdf:li>\n            </rdf:Seq>\n         </exif:ComponentsConfiguration>\n         <exif:SubsecTimeOriginal>581</exif:SubsecTimeOriginal>\n         <exif:Flash rdf:parseType="Resource">\n            <exif:Function>False</exif:Function>\n            <exif:Fired>False</exif:Fired>\n            <exif:Return>0</exif:Return>\n            <exif:Mode>0</exif:Mode>\n            <exif:RedEyeMode>False</exif:RedEyeMode>\n         </exif:Flash>\n         <exif:ApertureValue>4845/1918</exif:ApertureValue>\n         <exif:ExposureBiasValue>0</exif:ExposureBiasValue>\n         <exif:FNumber>12/5</exif:FNumber>\n         <exif:PixelYDimension>2332</exif:PixelYDimension>\n         <exif:CustomRendered>6</exif:CustomRendered>\n         <exif:MeteringMode>5</exif:MeteringMode>\n         <exif:ISOSpeedRatings>\n            <rdf:Seq/>\n         </exif:ISOSpeedRatings>\n         <exif:SceneType>1</exif:SceneType>\n         <exif:FlashPixVersion>0100</exif:FlashPixVersion>\n         <exif:SceneCaptureType>0</exif:SceneCaptureType>\n         <tiff:Orientation>1</tiff:Orientation>\n         <tiff:Make>Apple</tiff:Make>\n         <tiff:ResolutionUnit>2</tiff:ResolutionUnit>\n         <tiff:XResolution>72</tiff:XResolution>\n         <tiff:YResolution>72</tiff:YResolution>\n         <tiff:Model>iPhone 5</tiff:Model>\n         <xmp:CreatorTool>8.2</xmp:CreatorTool>\n         <xmp:CreateDate>2015-03-31T14:24:45</xmp:CreateDate>\n         <xmp:ModifyDate>2015-03-31T14:24:45</xmp:ModifyDate>\n         <exifEX:PhotographicSensitivity>50</exifEX:PhotographicSensitivity>\n         <exifEX:LensModel>iPhone 5 back camera 4.12mm f/2.4</exifEX:LensModel>\n         <exifEX:LensSpecification>\n            <rdf:Seq>\n               <rdf:li>103/25</rdf:li>\n               <rdf:li>103/25</rdf:li>\n               <rdf:li>12/5</rdf:li>\n               <rdf:li>12/5</rdf:li>\n            </rdf:Seq>\n         </exifEX:LensSpecification>\n         <exifEX:LensMake>Apple</exifEX:LensMake>\n         <photoshop:DateCreated>2015-03-31T14:24:45</photoshop:DateCreated>\n      </rdf:Description>\n   </rdf:RDF>\n</x:xmpmeta>\n'
  }

[lovell]

Hello, without the specific images it's hard to tell exactly what's different. The "6mb.png" file is palette-based, so that might take longer to decode, plus it is 3 channel, so might miss out on some auto-vectorised optimisations that only benefit 4 channel images.

This code is setting quality, which only applies for PNG output if you are using a globally-installed libvips compiled with support for libimagequant. Is this the case?

The first thing to do for any PNG performance problem is reduce the compressionLevel from its default value of 9 - perhaps try 6:

- transformer = transformer.toFormat(format, { quality });
+ transformer = transformer.toFormat(format, { compressionLevel: 6 });

https://sharp.pixelplumbing.com/api-output#png

If you're able to create a standalone repo with complete code and sample images, we can try to run it via callgrind to discover where CPU time is being spent.

Also is there a way to retrieve the compression level of the input image?

Please see #2300 (comment) and subscribe to randy408/libspng#114

[klouvas]

This code is setting quality, which only applies for PNG output if you are using a globally-installed libvips compiled with support for libimagequant. Is this the case?

We are using sharpjs prebuilt with libvips binaries (as far as i know libimagequant is not included). I've seen that quality is ignored where there libimagequant is not included. I've used compressionLevel: 6 and it's faster but again it takes a lot time.

I will try to find a similar image with copyrights that will allow me to upload it in a public repo and let you know.
I subscribed to the libspng issue and read the comment. I will try to run it with callgrind and discover CPU usage.

Thank you very much for your help.

[lovell]

We're planning to switch the prebuilt binaries to the SIMD-accelerated zlib-ng when there's a stable release, which should bring a big improvement to PNG de/compression times - see lovell/sharp-libvips#25

[lovell]

The next release will include zlib-ng in the prebuilt binaries - please subscribe to #2604 for updates.

[BobbyBabes]

Have you searched for similar questions?

Yes i have already checked:

* #526

* #526

* #2213
  Also most of the issues from https://github.com/lovell/sharp/issues?q=png+compress

@klouvas You mention issue 526 twice. Do you still remember the third issue that you checked?

[klouvas]

Have you searched for similar questions?

Yes i have already checked:

* #526

* #526

* #2213
  Also most of the issues from https://github.com/lovell/sharp/issues?q=png+compress

@klouvas You mention issue 526 twice. Do you still remember the third issue that you checked?

Since i don't remember the 3rd issue i removed the 1 reference from 526. Thank you for pointing it out.