Shader-based BLUR Filter Discrepancy in p5.js

[SableRaf]

Increasing Access

Ensuring a consistent visual output across p5 versions reduces confusion and lessens cognitive load for both learners and long time users.

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build Process
• Unit Testing
• Internalization
• Friendly Errors
• Other (specify if possible)

Feature enhancement details

Note: this is a follow up to the conversation on #6324

First of all, many thanks to @wong-justin, @aferriss, and @aceslowman for the outstanding work on integrating shader-based filters in p5.js.

However, I've noticed a difference in the output of the filter(BLUR). For clarity, I've put together a visual comparison using a small demo sketch available here. Additionally, here's the image derived from the demo:

Generated on Chrome Version 116.0.5845.140 (Official Build) (arm64), MacOS Ventura 13.4 (22F66)

The newly added shader for BLUR uses a box blur, as opposed to the Gaussian blur in CPU mode. Might be worth checking if we can use a Gaussian kernel in the shader and what that means for performance. It'd be great to have outputs that are consistent, especially for older projects that were built around the original blur and whose outputs will look different with the new shader-based method.

Looking forward to the community's insights on this.

[wong-justin]

I think blur is the hardest filter to match exactly with CPU mode.

The main difficulties trying to recreate the Gaussian blur were:

• making a kernel size that could change on any frame according to the user's parameter, which could mean hard coding many different kernels in one shader, or compiling on the fly in the middle of runtime

• alternatively using a small kernel and repeating passes for more blur, but the extra passes were slowing things down with all the back-and-forth between layers

Happy to hear other thoughts and approaches though. It would be nice to have a better blur.

[davepagurek]

Instead of having a hardcoded kernel, we could generate the kernel value dynamically per sample based on the distance from the center. I'm not sure how much of a performance impact this would have compared to using hardcoded values, but if it's not significant, it would avoid the complexity of managing different shaders.

To loop over a variable number of pixels, to circumvent the issue where loops have to have constant bounds, I generally see people loop up to a max value (max radius, in this case) and then break if the loop index equals the real loop limit.

[SableRaf]

I asked Patricio Gonzales-Vivo (creator of Lygia and co-creator of the Book of Shaders). He shared the following insights:

If matching what’s on the cpu is the goal you could upload the weight you use them to procedurally write the kernel convolution shader. Also for reference in lygia I calculate them on the fly https://lygia.xyz/filter/gaussianBlur/2D Is not incredibly performant but is not the worst on the world. Is the goal efficiency or accuracy?

I'd argue that accuracy should be essential since we're replacing the CPU blur with the GPU blur. I believe minimizing discrepancies would be beneficial. As for performance, we could test it and see how costly this type of blur is. I'm guessing it should still be significantly faster than CPU blur but some benchmarking would help 🤓

An example of all 2D shaders in Lygia, including 2D gausian blur, using p5.js can be found here: https://editor.p5js.org/patriciogonzalezvivo/sketches/XCkTzoyB3

[wong-justin]

Ah ok, thanks for the resources and ideas. I can take the next week or two to dig in and learn and see what I can come up with.

[SableRaf]

Just one more resource that may be helpful:

https://github.com/SableRaf/Filters4Processing/blob/master/sketches/GaussianBlur/data/gaussianBlur.glsl

Taken from: http://callumhay.blogspot.com/2010/09/gaussian-blur-shader-glsl.html

[wong-justin]

Here's an initial draft: https://editor.p5js.org/jwong/sketches/Cn2z2G-to
The weight function needs some adjusting, and I haven't thought about matching CPU output yet. Supposedly with the right math, I think it could be tuned pretty close to the original.

But this is a single shader, single pass gaussian blur using the break technique (I think I've wrapped my head around it now thanks to everybody's help)

Note that performance drops kinda quickly. Some rough manual benchmarks from my machine:

steps / kernel size / parameter -->	10	20	30	40
current shader BLUR, fps	60	60	57	50
new linked BLUR, fps	60	30	10	7

[aferriss]

I think the current implementation could be modified to continue using the two pass solution, which should keep things much more performant. Here's another implementation of the gaussian kernel setup for a two pass renderer

https://editor.p5js.org/aferriss/sketches/IS4P5RdAo

[SableRaf]

Thanks @wong-justin and @aferriss! How do these two approaches compare with the CPU blur in terms of quality?

[aferriss]

Here's a few comparisons I put together, with the GPU blur on top and CPU on the bottom of each image. None of them match perfectly, and to be honest I think we will have a pretty difficult time matching the cpu blur exactly. I think we may need to make a tradeoff here between performance and visual appeal. That being said we can keep tinkering away at this and hopefully get as close as possible. I think of the options presented here the one I shared yesterday seems the closest, but would love to hear other opinions!

[SableRaf]

@aferriss I agree that your version looks the best of the ones there.

Another approach suggested here is to use a single precalculated kernel and apply the blur multiple times with two ping-pong framebuffers.

I've made a quick attempt based on your sketch, setting the samples relatively low, but calling filter(s) multiple times per frame. The results seem really encouraging in terms of both performance and visual quality.

Edit: I also tried a version of the hardcoded kernel from the article: https://editor.p5js.org/SableRaf/sketches/NTy_zPyPX

[SableRaf]

Another useful reference. This article does a systematic exploration of GPU blur, with regard to performance and quality across platforms:
https://venturebeat.com/pc-gaming/an-investigation-of-fast-real-time-gpu-based-image-blur-algorithms/

[SableRaf]

I've done further exploration using a shader to visualize the lightness as a graph, inspired by the figures in the article from VentureBeat. You can find the test sketch here: https://editor.p5js.org/SableRaf/sketches/6it1LpFq8

We can see that Adam's blur is pretty close but it has a bias towards the bottom right direction that becomes more visible at higher samples.

The multipass blur is also quite close but it explodes at extreme number of passes, I suppose because of edge effects. (edit: I'm not sure about that image, looks like I may have made a mistake while generating it, I'll have to get back to it)

This is not a super rigorous benchmark but I hope it will be helpful.

By the way, I'm testing in Chrome Version 116.0.5845.187 on macOS Ventura 13.4.

CPU Blur filter(BLUR,20,false) (ground truth)

Current GPU Blur filter(BLUR,20)

Multipass blur (200 passes)

Adam's blur (60 samples)

Adam's blur (80 samples)

Multipass blur (400 passes)

[davepagurek]

I think the offset in Adam's might be from this line:

float x2 =  (i - samples * 0.5);

If we're using e.g. samples=3, then we get:

i	x2
0	-1.5
1	-0.5
2	1.5

If we instead use x2 = i - (samples-1.) * 0.5, we get:

i	x2
0	-1
1	0
2	1

This only works when samples is odd, but if we define samples = blurRadius * 2 + 1 (so a radius of 1 means it averages the pixel to the left and the pixel to the right with the current pixel) then this would always be true.

[SableRaf]

Hey y'all, with some help from @davepagurek and others, I've built a more advanced benchmark for blur. Feel free to play around with it to get a better feel for the parameter space and performance. Hold click and move the cursor to see the profile change based on the radius and number of passes.

Feel free to experiment and let me know what you think.

Notes

• The shader version is based on Adam's, with the proposed fix by Dave and using only odd sample counts.
• CPU blur only refreshes when the radius changes so don't use this sketch to evaluate the performance of CPU blur.

Interactive demo

https://editor.p5js.org/SableRaf/sketches/KFye28jYn

Instructions

• Press W to toggle between WebGL and classic CPU blur.
• Press G to toggle displaying the grid overlay
• Move your cursor left and right to change the blur radius
• Move your cursor up and down to change the number of passes (WebGL mode only)
• Left Click to toggle the luminosity profile graph (green pixels).

Examples

Adding extra passes helps with the smoothness of the shader blur. You can see that the sharp edges almost completely disappear when applying the blur twice.

More passes help with the shape of the blur.

The CPU blur does get a more rounded shape at higher radii but shows some discontinuities (unless those are caused by my graphing shader, but they don't appear on the GPU blur).

To get this kind of roundedness on the GPU a higher number of passes is necessary.

[davepagurek]

I dug a bit more into the code for the existing CPU blur:

• It looks like it's not actually doing a Gaussian blur at all -- the influence of each pixel is parabolic as you go away from the center
• It's doing a two-pass blur, one for each axis
• The radius you pass in is not the number of samples, nor the radius of the blur -- it's something closer to σ in a Gaussian blur, where the conventional wisdom is that at 3σ away, it's "basically" 0 influence, so you can stop your kernel there. Similarly, the existing CPU blur increases the kernel radius to 3.5 times the number you pass in (which means the number of samples per axis is double that.)

Putting all those together into @SableRaf's scaffolding, I got this: https://editor.p5js.org/davepagurek/sketches/x6wNXyzxQ

Comparison:

[davepagurek]

One more update: @SableRaf and I have been working on versions that don't drift to the side when the blur radius is greater than the max samples.

Here's his with even spacing for 64 samples (we can always turn this up though): https://editor.p5js.org/SableRaf/sketches/QpkhsI94y

Here's a fork of that that I added dithering to, still 64 samples: https://editor.p5js.org/davepagurek/sketches/3aD5WBynW

[SableRaf]

Thanks @davepagurek that looks great!

@aferriss and @wong-justin what do you think? Should we go with that last one?

[aferriss]

Looks great to me! The multiple pass solution was something we had also explored early on but abandoned in favor of something simple, which we could later revisit. It does seem like its more difficult to achieve parity with the CPU blur when using more than 1 pass.

I like the dithering solution as well but it does add some barely perceptible noise. Personally I don't mind it but wonder how others feel.

For the dithering shader, I've had better luck with this snippet in making more natural looking noise that doesn't create artifacts and doesn't use a sin function. Also can we move that random calculation outside the for loop and save some calculations?

float random(vec2 p){
	vec3 p3  = fract(vec3(p.xyx) * .1031);
    p3 += dot(p3, p3.yzx + 33.33);
    return fract((p3.x + p3.y) * p3.z);
}
float r = random(gl_FragCoord.xy); // no need to normalize coords

Nice work everyone!!

[davepagurek]

I like the dithering solution as well but it does add some barely perceptible noise. Personally I don't mind it but wonder how others feel.

Do we have any easy ways of passing more than one parameter to a filter shader? I think letting people use more samples could make sense (e.g. for exports), but I'm not sure that we have a great API for it yet.

[davepagurek]

Also I'll try adding your dithering changes after work 🙂

[aferriss]

Do we have any easy ways of passing more than one parameter to a filter shader? I think letting people use more samples could make sense (e.g. for exports), but I'm not sure that we have a great API for it yet.

Currently you can only pass a single value to the shader, but there's no reason why it couldn't be extended to receive an object of parameters. I think the function signature is already a little complicated...

filter(customShader);
filter(FILTER_ENUM);
filter(FILTER_ENUM, parameter);
filter(FILTER_ENUM, parameter, useWebGL);

@wong-justin wrote this function to deal with the parameters

p5.js/src/image/pixels.js

Line 627 in 63f94b0

function parseFilterArgs(...args) {

[davepagurek]

Here's an updated dither using @aferriss's random function: https://editor.p5js.org/davepagurek/sketches/fth7UYyXI

I also experimented with making better use of the random function being inside the loop and actually using a different random number per sample, but honestly it didn't look any better. That's convenient at least, because we can take the random function out of the loop and have it run a bit faster!

[SableRaf]

Looks great! I'd say the original issue is fully addressed by the latest shader version. Should we roll with that and leave the parameter discussion for a separate issue if necessary?

[davepagurek]

I think that makes sense. @wong-justin were you already working on adding blur filter changes into your updates? If not I can make a PR with these changes.

[wong-justin]

The latest demo looks fantastic! @davepagurek no I didn't have a PR for blur in progress, feel free to make it.

[patriciogonzalezvivo]

Hi! Sorry for jumping a bit late to the party.
I gave it a try to a Bartlett (bilinear filter) and the diff is pretty minimal

bartlett

difference

#ifdef GL_ES
precision mediump float;
#endif

uniform vec2        u_resolution;
uniform vec2        u_mouse;

uniform sampler2D   u_tex0;
uniform vec2        u_tex0Resolution;

uniform sampler2D   u_tex1;
uniform vec2        u_tex1Resolution;

varying vec2        v_texcoord;

#include "lygia/filter/bartlett.glsl"

void main (void) {
    vec3 color = vec3(0.0);
    vec2 pixel = 1.0/u_resolution.xy;
    vec2 tex0Pixel = 1.0/u_tex0Resolution;
    vec2 st = gl_FragCoord.xy * pixel;
    vec2 uv = v_texcoord;

    vec2 mouse = u_mouse * pixel;

    vec4 blur = bartlett(u_tex0, st, tex0Pixel, 13);

    vec4 control = texture2D(u_tex1, st);
    vec4 delta = abs(FNC - control);

    color = mix(blur.xyz, delta.xyz, step(mouse.x, st.x));

    gl_FragColor = vec4(color,1.0);
}