Question: .clone() performance issues function of sharp input type. How to fix it ?

[Aarbel]

EDIT: after further investigations bellow, tests revealed it's a .clone() performance issue

What is the best input file format to have the best sharp performance ?

In the code bellow, sharp in 20 times slower in case 1 compared to case 2, why ?
What is the ideal input format ? Base64 ? Blob ?

CASE 1 - SLOW

let fileBuffer
await fs.readFile('./image.png')
        .then((data) => {
            fileBuffer = data;
        })

const rawBigImage = sharp(fileBuffer)
        .sequentialRead()
        .raw()
        .toColorspace('srgb');

... + other sharp treatments which are slow

CASE 2 - FAST

const rawBigImage = sharp('./image.png')
        .sequentialRead()
        .raw()
        .toColorspace('srgb');

...+ other sharp treatments which are fast because of this different input

Thanks a lot for your help !

[lovell]

Did you see https://sharp.pixelplumbing.com/performance ?

For the same contents, Buffer-based input is generally slightly faster than filesystem-based input.

If you're still having problems, please can you provide complete, standalone code sample and image(s) that will allow someone else to reproduce.

[Aarbel]

@lovell you can test it here with this repo in 1 minute:
https://github.com/Aarbel/sharp-output-buffer-performance-issue

I have performances issues when the input is a Buffer, it's much faster with a file path string.

The main problem is, in my environment i am forced to use a Buffer input for sharp, but i'm facing this performance issue.

What could be the best way to fix it ?

Thanks a lot for your help !

[lovell]

Thank you. How are you isolating the performance of sharp vs the performance of other parts of this code? For example the use of await fs.readFile() with a large file might be adversely affected by nodejs/node#25741

[Aarbel]

@lovell await fs.readFile() is called only one time in my code and outputs a Buffer.

I guess, once the Buffer is created, it's in the RAM no ? So fs.readFile() is never called again and shouldn't impact sharp treatments.

My file image size is only 3Mb.

I'll send a simpler example in the next minutes.

[Aarbel]

@lovell here we go, you can find a much simpler example here, with the problem isolated:
https://github.com/Aarbel/sharp-simpler-output-buffer-performance-issue

After hours of investigation, it seems that the .clone() method has performance issues when:

• sharp input is an image Buffer (compared to string path input)
• many files are generated in the same process.

---

WHAT THE TESTS IN MY CODE SHOW

First, my tests compare 2 different inputs: a small png (32Kb) and "big" png (3Mo).

The speed difference is not significant between string path and Buffer input when

• sharp input is a small image
• few files are treated in the same process (5 in my tests)

The speed difference is SIGNIFICANT between string path and Buffer input when

• the input image is "big" (3Mo)
• MANY files are treated in the same process (100 in my tests)

---

Do you see a way to fix this performance difference using Buffers as sharp input ?

Thanks a lot for your help !

[lovell]

It looks like the "fast" tests are benefiting from libvips' cache. You can disable it by adding a sharp.cache(false); at the start of the tests to demonstrate a fairer comparison.

For one-input-to-many-outputs scenarios such as this, filesystem-based input is usually a better approach as it allows libvips to manage what is in memory rather than having to have everything in memory.

[Aarbel]

Thanks a lot @lovell, so no way in this case to increase performances with Buffer input ?
Damn. I'm just generating 512x512 images from a 3Mo image.

[lovell]

As I said in answer to your previous question of a similar nature at #2074 (comment) , the cost here is in the repeated decoding of the PNG input. Were you able to try my suggestion of holding a copy of the input image as raw pixel data in memory?

const { data, info } = await sharp(input)
  .raw()
  .toBuffer({ resolveWithObject: true });

const tile = await sharp(data, { raw: info })
  .extract({ ... })
  .png()
  .toBuffer();

[Aarbel]

Wonderfull @lovell, thought it was working in my previous issue because i used string path input instead of fully using your tip.

I was missing the { raw: info } option.

Right now performances are higher than path string input treatments. Absolutely perfect.