Abnormally high resource usage and slow builds

[skourismanolis]

I'm noticing high CPU and RAM usage when building using version >0.5.3.

I've tested most versions from 0.5.1-0.5.8 (latest) and 0.5.2/0.5.3 seem to be the fastest, with version 0.5.1 being a close second.

Each time after I get the last "Image at xxx optimized in yyyms", I get a "Completed in xx." and then the CPU spikes. No transformed images are written to the build dir at that point, This last "step" takes much more time than the rest of the build when in finishes. Emphasis on when because I've had to forcibly terminate the build a couple of times. Using versions above 0.5.5 (I think) dramatically worsens the effect.

On the latest version, this last "step" pushes my CPU to 100% and I've noticed it take as much as 9.5GB of RAM. (For reference I have a desktop computer with a 6 core/12 thread i7 8700K and 16GBs of RAM). Initially I thought it was an issue with Windows so I rebooted to my Linux install and tried it there.... It crashed my desktop environment and I got kicked back to the login screen.

Something of note, later versions also throw an error only on Windows when building basically anything else apart from avif files. I don't know if these are caused by the same issue (earlier versions are ok).

I'd also like to note what I'm trying to do:
I am working on a blog website that uses wordpress as a CMS. I'm currently only using the Picture component passing the src attribute as a URL. The blog currently has <50 images.

[RafidMuhymin]

Thank you for reporting the issue @skourismanolis ! I would like to ask you a few questions :

1. Is this problem specific to build only? Or, do you face similar issues in dev mode too?
2. Do you see less RAM usage on Windows or were you able to successfully build in Linux using v0.5.3?

[skourismanolis]

1. Dev mode seems fine
2. I haven't tried bulding on Linux but I'm using Netlify and it built successfully, so I suppose it's a general issue.

I can try to make an example setup demonstrating the bug should that be required.

Thanks a lot for your care and time, I really like astro-imagetools it's super useful for optimizing speed and SEO, and I plan on using it in production for this small blog website I'm working on!

[RafidMuhymin]

Thank you @skourismanolis, an example setup will really help to investigate the issue.

[skourismanolis]

I created this repo to explore and demonstrate the issue astro-imagetools-example. It's using the astro blog starter, with modifications in src/pages/index.astro. I'm basically fetching a bunch of images from the blog I'm working on and displaying them. I tried to simply display a single image from imgur multiple times but the build worked perfectly.

However with experimentation I've found the following:

• High resource usage still happens on 0.5.3, although it's not as bad as on 0.5.8. Builds actually finish
• Once a build finishes in 0.5.3, subsequent builds are very fast and the same effect is not observed. I suppose you're implementing some sort of cache.
• The problem seems to appear (or worsen) when having images with a large total size. This example fetches 64 images but by using Array.slice() I managed to reproduce the problem on much smaller counts of images.
• On windows, build fails on >=0.5.4 with Input file is missing error.
• I used gitpod to make this example, on 0.5.3 - 0.5.6 the first build finishes on ~160s while subsequent builds finish in ~5s. On 0.5.7 and 0.5.8 it never finishes. Here's a gitpod snapshot.

I changed versions on gitpod with:

rm -rf node_modules
yarn add astro-imagetools@version
yarn build

While on my windows pc I just removed node_modules/.cache since deleting and reinstalling the whole node_modules folder took a lot of time.

[RafidMuhymin]

Thank you for your efforts @skourismanolis and especially thanks for using GitPod. It'll help a lot to investigate the issue.

[RafidMuhymin]

On windows, build fails on >=0.5.4 with Input file is missing error.

It's very odd! Can you confirm if sharp is being installed correctly?

[skourismanolis]

It's very odd! Can you confirm if sharp is being installed correctly?

I saw sharp mentioned on yarn install's 'Building fresh packages' with no errors. I also have a sharp folder in my node_modules.
I made a new project, installed sharp and did a console.log(sharp.versions) which run successfully. Do I need to anything else to verify ?

[RafidMuhymin]

No, it's enough information @skourismanolis. Thank you @skourismanolis.

[IanVS]

I'm also seeing Input file is missing from sharp during the generating static routes part of a build. I'm using astro 1.0.0-beta.24, and astro-imagetools 0.6.3. I have my images in /public, if that matters. Edit: tried moving them to /src, but it didn't help. I'm on macOS.

[IanVS]

OK, I think I traced mine down to markdown files being processed, which included images like:

![Screenshot of editing an Unsafe Route](/images/docs/unsafe_routes.png)

Which is sent to sharp as .//images/docs/unsafe_routes.png. I think I see what's going on. I'll try to push up a PR soon.

[IanVS]

The reason the build is slow in this case is that there are 240 .avif files being written to disk, which is pretty slow with sharp: lovell/sharp#2597 lovell/sharp-libvips#97

If I change the format in the example repo to just 'jpg', the build completes in 50.94s, after writing 316 image files to the /dist/assets folder.

I'm not sure there's much more that can be done in this package to improve this, other than to find another tool that is faster than sharp, which would be a pretty major change.

[RafidMuhymin]

Good catch @IanVS ! I ran a few tests on GitPod and here are the results:

• jpg- it took 8s to generate jpg images for all the 240 images
• png- it took 11s to generate png images for all the 240 images
• webp- it took 15s to generate webp images for all the 240 images
• avif- it took 179s to generate avif images for all the 240 images

[RafidMuhymin]

Oops. Sorry @skourismanolis, I didn't see that you have locked astro-imagetools to 0.5.3. I upgraded it to 0.6.8 and now the build is failing. I am getting an error Processed image is too large for the WebP format after 1/2 minutes, and then after 15/20 seconds the build is failing with exit code 137. That means it ran out of memory. But I don't know why it's throwing the error Processed image is too large for the WebP format.

[RafidMuhymin]

After upgrading to [email protected] and running the same tests on GitPod:

• jpg- it took 59s to generate jpg images for all the 240 images
• png- it took 86s to generate png images for all the 240 images
• webp- took a long time and then exited with error code 1 saying Processed image is too large for the WebP format
• avif- took a long time and then exited with error code 137

[RafidMuhymin]

I am confirming that this issue starts from 0.5.7.

[RafidMuhymin]

@skourismanolis a fix for the regression has been merged and released in 0.6.9.

After upgrading to [email protected] and running the same tests on GitPod:

• jpg- it took 9s to generate jpg images for all the 240 images
• png- it took 13s to generate png images for all the 240 images
• webp- it took 20s to generate webp images for all the 240 images
• avif- it took 204s to generate jpg images for all the 240 images

Now, https://github.com/skourismanolis/astro-imagetools-example builds successfully on the latest release. It's taking 225s for a build to finish on GitPod.

Note: The increase in build time in 0.6.9 from 0.5.3 is not due to another regression. It's because the maximum default number of breakpoints was 3 in v0.5.3 and it's 11 since v0.5.7.

[RafidMuhymin]

Closing the issue @skourismanolis ! Feel free to reopen the issue if you are facing any issues. And thank you so much for creating such a detailed issue.

[oliverpool]

I just got bitten by this (converting an existing blog).

Disabling avif solved the issue.

Would it be acceptable to disable by default in format ? (["webp"])

Since it is known to be 10x slower than other conversions, it would be a much nicer experience for newcomers like me 😄