Background threshold sizes and resize throttling

[asterikx]

Before you submit:

• Please search through the existing issues (both open AND closed) to see if your question has already been discussed. Github issue search can be used for this: https://github.com/imgix/react-imgix/issues?utf8=%E2%9C%93&q=is%3Aissue

Question
I'm using Background on a div whose height is set relative to the browser's viewport. This is very inefficient because the image is cropped to the exact pixel height of the div:

• the height (and width) of the div differs for each browser viewport size, causing very rare cache hits
• while resizing the window, a new cropped image is fetched for each resize event (I can't notice any throttling here)

<Background src="hero_3.jpg">
  <Flex minH="calc(80vh - 64px)" justify="center" align="center">
    {/* ... */}
  </Flex>
</Background>

Pixel-precise images fetched while resizing the window (timeframe is ~2s):

Question

What's the best approach to:

1. Not creating a new cropped image for each exact viewport size, but still taking into account the varying height and width (smartphone vs 16:9 monitor)
2. Throttle/debounce image fetching during resizing

I appreciate any help

[sherwinski]

Hey @asterikx
Thanks for opening this issue. Unfortunately, I don't think there's a straightforward way handle these cases, short of changing the component behavior. However, I think this is great feedback that I'd like to take back to my team and discuss with them. I am sure there are ways that we can improve the interface and default behavior to provide a better experience here. I'll be sure to keep this issue updated once I know more.

[tremby]

I would class this as a major bug. I came here to open a similar ticket, having just tried the background version of this.

Off the top of my head, I'd suggest:

• Set some threshold which needs to be passed before the component decides to fetch a new version of the image. Perhaps a +25% change in area, or a ±25% change in aspect.
• Debounce or throttle the event handler
• Preload the new version and only swap it out when ready. At present I'm shocked with a white background if the browser window changes size.

[sherwinski]

@tremby Agreed that the experience can be better here. This is something our team will be looking at in the near future. Also if anyone is feeling up for it, we'd happily accept a PR as well 👍

[tremby]

As mentioned in another ticket just now, I actually wrote my own version of this yesterday. I'll ask my client if I can release the code. I'm quite happy with it.

• I use ResizeObserver to track the size
• I use that size to get the "ideal aspect" (which falls back to a "nominal aspect" provided in the props if the size is not yet ready such as in server-side rendering), and then I quantize this aspect ratio to one of n values which are evenly spaced throughout all possible aspect ratios, via a little formula I cooked up.
• I use that aspect ratio to generate a srcset (via the imgix-js-core lib), which offers several resolutions at that quantized aspect ratio, running up to the image's original width and no higher

The idea is that by quantizing the aspect ratios a new version will not be requested by the browser unless the aspect or size changes significantly enough to make it worthwhile.

I did not yet implement preloading the new version, if it is to switch.

[ericdeansanchez]

Hey all 👋 @asterikx @tremby I just wanted to take a moment to talk about some of the work we've done to improve the behavior of this component.

This work (already released) revolves around this core idea:

The idea is that...a new [image] version will not be requested by the browser unless the aspect or size changes significantly enough to make it worthwhile.

Below I will try to outline the changes we've already made and the changes to come.

Biggest Issues

The biggest issue I found with the pre v9.0.3 Background component was that it infinitely re-rendered starting from the first render. Oof, I know. Now, this component only re-renders when a larger image is needed, see #782. Which leads me to the main points that #782 was meant to address.

Main Points

• (reduce) keep re-rendering to a minimum
• (reuse) don't fetch another image if we have a larger one

PR #782 Commit Message
Comment on old vs new (re)rendering behavior

Better, But Not Best (Yet)

Resizing and Debouncing

• while resizing the window, a new cropped image is fetched for each resize event (I can't notice any throttling here)
• Debounce or throttle the event handler/Throttle/debounce image fetching during resizing

These issues have been addressed and the behavior should be much better, but it isn't the best it could be (see the next steps section).

Next Steps

While #782 improves things a lot, it doesn't go all the way in my opinion. For one, this is still an issue:

the height (and width) of the div differs for each browser viewport size, causing very rare cache hits

I'd also like to enforce this kind of thresholding:

Set some threshold which needs to be passed before the component decides to fetch a new version of the image. Perhaps a +25% change in area, or a ±25% change in aspect.

An iteration of the improved/improving Background component (that didn't land in main) had this kind of flow:

• shouldComponentUpdate?
• did we downsize? yes: okay, then scale the image we already have
• did we upsize? yes: imageDimensions *= 1.08 (these values ended up being really close to our default srcset width values)

Ideally, I'd like to connect the concept of providing some “breakpoint dimensions” (a set of values that help constrain image fetching) to the Background component and then do something like:

• Maybe(given a set of breakpoint dimensions):
  • if we've downsized, try to reuse the larger image, otherwise
  • if we've upsized, try to use our breakpoint dimension constraints, otherwise
  • if we've upsized larger than expected or if no breakpoint constraints were given:
  • try to fetch an image that's likely to have been served before (e.g. one of default 31 srcset dimensions that's greater than or equal to the dimensions needed):

    // Choose the first targetWidth that's greater than to the exact image width:
    let i = len(targetWidths) - 1;
    
    if x >= 8192, return 8192
    if x <= 100, return 100
    
    while x < targetWidths[i] {
      i -= 1;
    }
    
    // A long shot, but we could get lucky.
    if x == targetWidths[i], return targetWidths[i]
    
    // The most likely case:
    // The previous value, targetWidths[i+1], was greater
    // than x so return that since there's a chance it's been
    // served before.
    return targetWidths[i + 1];

The above can also be substituted for the findClosest function that get’s called within the Background component. As it stands, it would be at least a 2-3x improvement over the function call (for certain edge cases it’s 25x faster).

If we could do the above I think those changes will make doing something like the below either easier or decrease its appeal.

Preload the new version and only swap it out when ready. At present I'm shocked with a white background if the browser window changes size.

[tremby]

Honestly I think you're significantly overcomplicating it.

did we downsize? yes: okay, then scale the image we already have

Let the browser handle this! Chrome for one already does exactly this. It won't fetch a smaller item from the srcset if it already has a larger one.

All you need to give the browser is a srcset of all available resolutions for the current aspect ratio, and a hint about what the render size is via the sizes attribute. You don't need to give it a particular size; you let the browser choose which.

Ideally, I'd like to connect the concept of providing some “breakpoint dimensions”

This is what my scheme outlined above achieves by quantizing the current aspect ratio.

Everywhere you talk about target widths, you should instead be thinking about target aspect ratios. Lean on srcset! A change in aspect ratio is the only reason you should be changing the srcset.

Under the scheme I outlined in my previous comment, the component only ever changes its srcset if the aspect changes significantly. The browser chooses if and when to load new images in all other cases.

I've asked for permission to show my code. I'll post a codepen or something if I'm allowed, so you can see how it works in action. But in the mean time, and in case it's denied, I'll explain the aspect quantization I implemented more fully.

1. Take the current aspect ratio of the element you're rendering the background image on (or fall back to a nominal aspect ratio if you don't have the current one available yet). This is the ideal aspect ratio.
2. Quantize the ideal aspect ratio to a fixed set of proportionally-evenly-spaced aspect ratios.
3. Generate a srcset based on this quantized aspect ratio.

My quantization scheme works by first mapping the aspect ratio (which could be 0 all the way to infinity) to a finite range.

• For any aspect ratio less than or equal to 1 (that is, any portrait or square image), the mapped value is identical to the input value. So an aspect of 0.25 is mapped to 0.25, 0 to 0, 0.9 to 0.9, 1 to 1.
• For any aspect ratio greater than 1, we invert it (1 ÷ input) and then subtract the result from 2. So an aspect of 1 is mapped to 1, 2 to 1.5, 4 to 1.75, infinity to 2.

Now we have a number from 0 to 2 which describes all possible aspect ratios (and even some impossible ones, 0 and infinity). What's more, in this mapping portrait and landscape aspects are comparable: in this mapped range 0.1 is the portrait equivalent of 1.9: unmapped these would be aspect ratios 1:10 and 10:1. This means we can easily choose any number of proportionally-equally-spaced aspect ratios over the entire spectrum.

So we pick a number of steps. This is an accuracy vs performance question and may be unique to any deployment. It should be an option in the final component. I picked 12 steps, so 13 possible aspects. 0 and infinity are included in this and will never be truly used, so in reality I have 11 aspect ratios which might be asked for. 5 portrait, 5 landscape, and square.

So we take our ideal aspect ratio, map it to the 0..2 range as described above, then quantize it linearly within this space to one of the allowable steps, then unmap it again. The output from this is our target aspect ratio. This is the aspect ratio we use to generate the srcset.

I'll comment with code and/or a codepen if/when I get permission.

[ericdeansanchez]

@tremby When I say:

then scale the image we already have

I mostly mean that the heavy lifting is done by CSS/browser: backgroundImage: url(${src}).

I see you've mentioned using srcsets a lot; is there an example scenario you could give where a srcset is more desirable over a single src? Maybe I'm just misunderstanding things here. Can you use a srcset for background images?

Update:
There's image-set which looks something like this:

background-image: image-set( "foo.png" 1x,
                             "foo-2x.png" 2x,
                             "foo-print.png" 600dpi );

[ericdeansanchez]

I'm closing this issue for now as current releases have largely fixed this problem. That said, the Background component is far from perfect, so please feel free to continue the conversation here and we'll continue to work towards a more ideal solution.

[tremby]

is there an example scenario you could give where a srcset is more desirable over a single src?

There are myriad resources about srcset online; here's an example.

The browser knows much more than the developer about the current circumstances, in terms of the user's display, their zoom level, the viewport size, their bandwidth limitations, which assets it has already has downloaded and cached, and so on. The whole point of srcset is to let the browser pick an image size which fits the circumstances. Under the scheme I implemented, the markup and stylesheet do not need to be mutated on resizes, until a aspect ratio threshold has been hit. Additionally, with a srcset and a good guess at the initial aspect ratio we are much more likely to load the correct image at initial page load time (which is the subject of one section of the page I linked above).

Update:
There's image-set which looks something like this:

That will be lovely in future, but it is only a draft in the spec and browser support is not good enough yet; particularly it's Safari on both desktop and IOS whose support is inadequate. See caniuse but be sure to read the footnotes; the 93% global support metric is misleading.

For now, using an img element with srcset and setting object-fit: cover is in my opinion the only viable approach.

Hopefully full image-set support isn't too far in the future. Once it's ready, it'll certainly be the best fit, but the algorithm I'm describing above to choose the set of images via a quantized aspect ratio would still be 100% applicable, and that's what I'm trying to push here. It's a whole lot less code than your old or new versions of this component (the important parts of my implementation are just a few lines, and my whole component is less than 100 including comments), and I'd wager better-behaved and more efficient.

Perhaps you're dead set on using background-image, and I'd understand that. Perhaps there's a viable polyfill for image-set if that's the case.