Image Light Feature - GSOC 2023 #6255

AryanKoundal · 2023-07-06T11:36:55Z

This PR is for the work under the GSOC 2023 project "Improving p5.js WebGL/3d functionality by adding support for image-based lighting in p5.js WebGL mode."

Project Description:

This project aims to add lightning to a 3D object, using the surrounding environment as a single light source, which is generally called Image-Based Lightning. In simple words, one can very quickly drop in an image from real life to use as surrounding lights, rather than continuously tweaking the colors and positions of ambient, point, etc lights.

Changes:

This includes the whole progress broken down into steps. The boxes will be checked off according to the progress made.

CubeMap Convolution
PBR and Irradiance Lighting
Pre-filtering environment map
Pre-computing the BRDF

Screenshots of the change:

PR Checklist

npm test
npm run build passes
npm run lint passes
Inline documentation is included/updated
Examples included/updated
Functions Descriptions and Documentation is included/updated

The project is completed and here are some screenshots and videos demonstrating the work.

Image of Example 1

Video of Example 1
https://github.com/processing/p5.js/assets/77334487/44b30c77-33c1-41d0-ada5-282424978832

Image oF Example 2

Video of Example 2
https://github.com/processing/p5.js/assets/77334487/a0a6b3f9-b25b-451f-961e-b2970cb9e907

davepagurek

Looking good so far!

davepagurek · 2023-07-14T00:13:57Z

src/webgl/shaders/imageLight.frag

+
+const float PI = 3.141;
+
+vec2 normalToEquirectangular( vec3 v){


Do we use this function?

Nope, I actually re-wrote the function in line 22. The function N2E is the one being used for now. I think I should remove the function on line 10 and rename the function at line 22.

davepagurek · 2023-07-14T00:14:12Z

src/webgl/shaders/imageLight.frag

+    // Turn it into -theta, but in the 0-2PI range
+    theta = 2.0 * PI - theta;
+  }
+  theta = theta / (2.0 * 3.14159);


Maybe where we use 3.14159 right now we can use the const float PI we defined earlier, and add more decimals to that?

Sure, I'll clean up this file. It has some extra code as well.

… it is getting used

davepagurek · 2023-08-07T14:06:06Z

src/webgl/p5.RendererGL.js

@@ -446,6 +448,17 @@ p5.RendererGL = class RendererGL extends p5.Renderer {
    this.spotLightAngle = [];
    this.spotLightConc = [];

+    // repetition
+    this.textures = new Map();


I think we don't need to declare this any more now that we have blurryTextures below

Sure, I will remove this. I was going to ask you about this, that's the reason I marked it as repetition.

davepagurek · 2023-08-07T14:12:02Z

src/webgl/light.js

@@ -494,6 +494,33 @@ p5.prototype.pointLight = function(v1, v2, v3, x, y, z) {
  return this;
 };

+p5.prototype.imageLight = function(img){
+  // ! if the graphic is always true, do we really need this section?


If we're going to use img as a light source, it means we'll need the blurry version of its texture in order to pass into the shader. If this.blurryTextures(img) is defined, then we've already made the blurry texture in a previous frame and don't need to do anything more here. If not though, that's when we need to run the commented out code below to make it, and then after line 512 we'll need to store it with this.blurryTextures.set(img, newGraphic)

We could alternatively do this if statement right before we do shader.setUniform('equiRectangularTextures', this.blurryTextures.get(this.activeImageLight)); later on. The idea is that we need that texture to exist by that point

davepagurek · 2023-08-07T14:16:18Z

src/webgl/p5.RendererGL.js

+  // used in imageLight
+  // create a new texture from the img input
+  getBlurryTexture(input){
+    const blurrytex = new p5.Texture(this, input);


I think the main difference between the blurry textures we store here and the textures already used by RendererGL is that for normal textures, we just make a new p5.Texture directly from the input. For this though, the input is non-blurry, and we want to store the blurry version, so somewhere (maybe in here actually?) we should run the shader to generate the blurry image onto a graphic and store that instead.

davepagurek · 2023-08-07T14:20:01Z

src/webgl/p5.RendererGL.js

+    if (this.activeImageLight) {
+      // this.activeImageLight has image as a key
+      // look up the texture from the blurryTexture map
+      shader.setUniform('equiRectangularTextures', this.blurryTextures.get(this.activeImageLight));


If we decide to make getBlurryTexture compute and store the blurry texture if it doesn't yet exist, instead of doing blurryTextures.get() here, we could do this.getBlurryTexture(this.activeImageLight)

TanviKumar · 2023-08-08T10:00:32Z

src/webgl/p5.RendererGL.js

+  // used in imageLight
+  // create a new texture from the img input
+  getBlurryTexture(input){
+    const blurrytex = new p5.Texture(this, input);


nit: blurryTexture or inputBlurryTexture

We actually refactored this function completely in which this line was removed.

TanviKumar · 2023-08-08T10:24:04Z

src/webgl/shaders/lighting.glsl

 const float specularFactor = 2.0;
 const float diffuseFactor = 0.73;
+const float PI = 3.141;


Can this be used from another common constant file? Would be better to have a value closer to real PI, since 3.14159 is closer to 3.142

I don't think we have a file for constants because mostly constants are taken via the "MATH" object but it isn't available in shader languages according to my knowledge.

We can do #define PI 3.141592 at the top of this file.
Another possible option is to do #define PI 3.141592 in the \webgl\shaders\webgl2Compatibility.glsl file.

I am not sure which one is better.

For now since we only use this here and we don't use pi anywhere else in a shader yet (to my surprise, haha) I think we can keep it in this file for now, and if we find we need to use it in another file too (maybe we need it for the specular shader?) we can factor it out into a new shader constants file at that point

TanviKumar · 2023-08-08T10:25:08Z

src/webgl/shaders/lighting.glsl

@@ -67,6 +75,44 @@ LightResult _light(vec3 viewDirection, vec3 normal, vec3 lightVector) {
  return lr;
 }

+float map(float value, float min1, float max1, float min2, float max2) {


nit: A clearer name for this if possible

The map(vale, min1, min2, max1, max2) function is converting the range of value from [min1 to max1] to [min2 to max2].

So should we make it something like convertRange() or mapRange() maybe?

This function mirrors https://p5js.org/reference/#/p5/map so maybe we could link to that reference page + add a comment description of what it does?

TanviKumar · 2023-08-08T10:26:07Z

src/webgl/shaders/lighting.glsl

+    theta = acos( normal.x / sin(phi) );
+  }   
+
+    vec2 newTexCoor = vec2(


Identation is off

Thanks, corrected it :)

Co-authored-by: Dave Pagurek <[email protected]>

davepagurek · 2023-10-26T11:34:41Z

src/webgl/shaders/lighting.glsl

+#endif
+  // this is to make the darker sections more dark
+  // png and jpg usually flatten the brightness so it is to reverse that
+  return pow(outColor.xyz, 10.0);


Looks like tests are failing because pow needs both parameters to be the same type :') So maybe this needs to be:

Suggested change

return pow(outColor.xyz, 10.0);

return pow(outColor.xyz, vec3(10.0));

src/webgl/light.js

davepagurek · 2023-10-26T17:03:10Z

src/webgl/light.js

@@ -494,6 +494,12 @@ p5.prototype.pointLight = function(v1, v2, v3, x, y, z) {
  return this;
 };

+p5.prototype.imageLight = function(img){
+  // setting activeImageLight so that the setUniform is executed


Maybe we can be a bit more specific here and say that this is what _setImageLightUniforms looks at to see if it needs to send uniforms to the fill shader

davepagurek · 2023-10-26T17:07:01Z

src/webgl/p5.RendererGL.js

@@ -468,6 +473,17 @@ p5.RendererGL = class RendererGL extends p5.Renderer {
    this.spotLightAngle = [];
    this.spotLightConc = [];

+    // Null if no image light is set, otherwise, it is set to a p5.Image


Maybe we can rephrase this into sentences so it's a bit easier to read? Also, for each map, we should say what the keys and values are. E.g. for this one, it maps a p5.Image used by imageLight() to a p5.Graphics containing the combined light it sends to each direction.

davepagurek · 2023-10-26T17:08:41Z

src/webgl/p5.RendererGL.js

+    // this is to lookup image from blurrytexture Map rather from a
+    // texture map
+    this.diffusedTextures = new Map();
+    // map to store the created textures to prevent mis calculation


This one maps the input p5.Image to a p5.MipmapTexture. Worth mentioning that we use a mipmap texture because it can contain light from multiple roughness levels.

src/webgl/p5.RendererGL.js

davepagurek · 2023-10-26T17:20:19Z

src/webgl/shaders/imageLight.vert

@@ -0,0 +1,34 @@
+// add precision


Did we want to copy the precision from the fragment shader to here?

davepagurek · 2023-10-26T17:21:25Z

src/webgl/shaders/imageLightDiffused.frag

@@ -0,0 +1,82 @@
+precision mediump float;


oh actually we probably want this to be highp; this basically gets ignored by the webgl driver on desktop, but on mobile it's actually used, and in my experience any time we have normals, the precision makes a big difference on the visual fidelity

Suggested change

precision mediump float;

precision highp float;

davepagurek · 2023-10-26T17:21:46Z

src/webgl/shaders/imageLightDiffused.frag

+const float PI = 3.14159265359;
+
+// not needed
+vec2 normalToEquirectangular( vec3 v){


We can remove this function since we aren't using it

davepagurek · 2023-10-26T17:29:05Z

src/webgl/shaders/lighting.glsl

+  return min2 + (value - min1) * (max2 - min2) / (max1 - min1);
+}
+
+vec2 mapTextureToNormal( vec3 normal ){


oh should we be using your implementation of nTOE from the other fragment shaders instead of this? If I recall correctly, this is the older implementation that has a bug where it only uses the 0-PI range, which you fixed in your updated nTOE.

I think using nTOE would also mean we can remove the map function above since this is all that uses it.

Done, i'm testing after the changes. If nToE works, which it will very likely. I'll remove the mapTextureToNormal one.

done, changed the nToE name to mapTextureToNormal for better understanding name.

…Light examples to not run for the npm run test, those examples have been manually verified to be working

davepagurek

The documentation is looking great! just some very minor punctuation things for that.

I did a bit of debugging of the nToE function. It looks like it has to be a bit different from the other files to work properly? but it seems like there's a value that makes it work.

I also left some suggestions about the power-of-10 thing for diffuse light, let me know what you think!

davepagurek · 2023-10-27T16:21:13Z

src/webgl/light.js

+ *
+ * The image light stimulates light from all the directions.
+ * This is done by using the image as a texture for a relatively
+ * very huge light in a form of a sphere. This sphere contains


nit: "relatively very huge light in a form of a sphere" could maybe be "an infinitely large sphere light"

davepagurek · 2023-10-27T16:21:54Z

src/webgl/light.js

+ * It will have different effect for varying shininess of the
+ * object in the drawing.
+ * Under the hood it is mainly doing 2 types of calculations,
+ * first one is creating an irradiance map from the


Suggested change

* first one is creating an irradiance map from the

* the first one is creating an irradiance map from the

davepagurek · 2023-10-27T16:22:03Z

src/webgl/light.js

+ * Under the hood it is mainly doing 2 types of calculations,
+ * first one is creating an irradiance map from the
+ * environment map of the input image.
+ * Second one is managing reflections based on the shininess


Suggested change

* Second one is managing reflections based on the shininess

* The second one is managing reflections based on the shininess

davepagurek · 2023-10-27T16:22:19Z

src/webgl/light.js

+ *
+ * Note: The image's diffuse light will be affected by fill()
+ * and the specular reflections will be affected by specularMaterial()
+ * and shininess()


Suggested change

* and shininess()

* and shininess().

davepagurek · 2023-10-27T16:25:26Z

src/webgl/light.js

+ *   imageLight(img);
+ *   // This will use specular once we add it
+ *   specularMaterial(20);
+ *   // shininess(slider.value());


We can take out the commented out lines in this sketch I think

src/webgl/light.js

davepagurek · 2023-10-27T17:24:37Z

src/webgl/shaders/lighting.glsl

+  theta = theta / (2.0 * 3.14159);
+  phi = phi / 3.14159 ;
+
+  vec2 angles = vec2( phi, theta );


Looks like something might be up still:

I noticed that we had a note for diffused light to use the word normal instead of R. When I use that, it looks like this:

I tried using a two-toned background to check against and it seems like it's kind of flipping it:

That looks a bit more correct, but it looks like the green is on the side? I think that might be because in other files, we used theta and phi differently, but here, we want to return vec2(theta, phi) instead of the other way around (I think that makes sense that the x axis is theta here since that's the one whose range is 2*PI. That's different than in the other shaders, which is a little odd, but at least we tested those independently and they seem fine.) That looks like this:

so that still seems 90 degrees off, but in a different axis this time. I tried cycling -90 degrees in the theta axis. This seems to look right, and that's using vec2 angles = vec2( fract(theta + 0.75), 1.0 - phi );

With just diffuse:

With specular:

Suggested change

vec2 angles = vec2( phi, theta );

vec2 angles = vec2( fract(theta + 0.75), 1.0 - phi );

Changed the suggested line, but do i have to change the line

// use worldNormal instead of R vec2 newTexCoor = mapTextureToNormal( R );

Did you use worldNormal or R in this function?

Did you use worldNormal or R in this function?

I think that one should be worldNormal.

I double checked the suggested code and I think maybe I was testing without that by accident though -- I think when using the world normal, to get the example with the stripes to work, it would be this:

vec2 angles = vec2( fract(theta + 0.25), 1.0 - phi );

(Basically, 0.25 instead of 0.75, which is like rotating by pi.)

davepagurek · 2023-10-27T17:44:33Z

src/webgl/shaders/lighting.glsl

+  // return texture.xyz;
+  // this is to make the darker sections more dark
+  // png and jpg usually flatten the brightness so it is to reverse that
+  return pow(texture.xyz, vec3(10.0));


I tried changing imageLight() with lights() in a sketch to see if they could be roughly equivalent. I think 10 might be a bit too aggressive for the diffuse light unfortunately. I tried playing around with some other curves:

Lighting call
pow( texture.xyz, vec3(10.0) );

texture.xyz;

smoothstep( vec3(0.), vec3(0.8), texture.xyz );

imageLight(img);

lights()

So although it's not physically accurate or anything, maybe doing this will help make it easier for people to switch from lights() to imageLight()? This maps texture values from 0-0.8 to 0-1 along a smooth curve:

Suggested change

return pow(texture.xyz, vec3(10.0));

return smoothstep(vec3(0.0), vec3(0.8), texture.xyz);

Done for caluculateImageDiffuse, should I do this for calculateImageSpecular are well?

I think specular looks ok how it is, the inconsistency is fine as long as it looks ok.

src/webgl/light.js

davepagurek

The examples look great!

davepagurek · 2023-10-28T11:16:28Z

src/webgl/light.js

+/**
+ * Creates an image light with the given image.
+ *
+ * The image light stimulates light from all the directions.


Suggested change

* The image light stimulates light from all the directions.

* The image light simulates light from all the directions.

davepagurek · 2023-10-28T11:18:56Z

src/webgl/shaders/lighting.glsl

+  // return texture.xyz;
+  // this is to make the darker sections more dark
+  // png and jpg usually flatten the brightness so it is to reverse that
+  return pow(texture.xyz, vec3(10.0));


I think specular looks ok how it is, the inconsistency is fine as long as it looks ok.

davepagurek · 2023-10-28T11:32:29Z

src/webgl/shaders/lighting.glsl

+  theta = theta / (2.0 * 3.14159);
+  phi = phi / 3.14159 ;
+
+  vec2 angles = vec2( phi, theta );


Did you use worldNormal or R in this function?

I think that one should be worldNormal.

I double checked the suggested code and I think maybe I was testing without that by accident though -- I think when using the world normal, to get the example with the stripes to work, it would be this:

vec2 angles = vec2( fract(theta + 0.25), 1.0 - phi );

(Basically, 0.25 instead of 0.75, which is like rotating by pi.)

davepagurek

One last thing, do you think for the example image we use, we should use an actual sphere map image that wraps correctly so users know what kind of input this is intended for? e.g. something from https://polyhaven.com/hdris since the license for these is free and requires no attribution?

davepagurek · 2023-10-30T14:08:00Z

src/webgl/shaders/lighting.glsl

+  vec3 lightDirection = normalize( vViewPosition - worldCameraPosition );
+  vec3 R = reflect(lightDirection, worldNormal);
+  // use worldNormal instead of R
+  vec2 newTexCoor = mapTextureToNormal( R );


I think this still needs to be changed to worldNormal (and I believe that means we can remove lines 113 and 114 here too.)

Done, I have commented out the lines to test it. If the tests pass then I'll completely remove this section.

AryanKoundal · 2023-10-31T09:40:42Z

One last thing, do you think for the example image we use, we should use an actual sphere map image that wraps correctly so users know what kind of input this is intended for? e.g. something from https://polyhaven.com/hdris since the license for these is free and requires no attribution?

I agree with you. Those sphere map images would give a better understanding of the feature.
But also would that create confusion among users? They might feel that they need a sphere map image for the background they want to use(But in reality even a simple image would work fine as well). What do you think?

davepagurek · 2023-10-31T12:15:55Z

We talked about this on our call, but just adding it again here for anyone who might be reading: we've decided to use a sphere map for one example and a regular image for another.

davepagurek · 2023-10-31T13:15:29Z

Maybe we can scale down the sphere map a bit? Right now it's 8000px wide, we probably don't need quite so many pixels 😅

AryanKoundal added 5 commits July 6, 2023 16:45

Added functions for diffused image light

50a0c14

Added functions for diffused image light

dfa600c

Merge branch 'main' into imageLight

0521694

Merge branch 'main' into imageLight

b155bac

Added shaders and code to read them

34e6438

davepagurek reviewed Jul 14, 2023

View reviewed changes

AryanKoundal and others added 12 commits July 20, 2023 16:15

Merge branch 'main' into imageLight

916f5be

refactoring the ImageLight fragment shader

c59ae6c

added definition of map function in lightingglsl

8b1e19c

removed errors

5287a4f

error correction

b05bca6

Merge branch 'main' into imageLight

f6899a8

corrected errors

8cd5775

added comments

3c9bf12

added comments

701cbee

Merge branch 'main' into imageLight

4c63de5

made the changes from the build file test

b9837dc

create a new map for storing the images and modifying the files where…

5546093

… it is getting used

davepagurek reviewed Aug 7, 2023

View reviewed changes

TanviKumar reviewed Aug 8, 2023

View reviewed changes

AryanKoundal added 5 commits August 8, 2023 17:46

modifications according to the reviews

98af50f

Merge branch 'main' into imageLight

8dbbff5

applied all the changes after debugging

711c5f5

Merge branch 'main' into imageLight

52c4d42

made changes according to the reviews

0121439

Qianqianye added the Area:WebGL label Aug 18, 2023

AryanKoundal and others added 2 commits September 8, 2023 08:03

Merge branch 'main' into imageLight

8e9cd8a

Convert existing shaders to support GLES 300

7cf062d

AryanKoundal and others added 2 commits October 26, 2023 12:15

Updated the outcolor

b37afe5

Co-authored-by: Dave Pagurek <[email protected]>

added comments

69008c0

davepagurek reviewed Oct 26, 2023

View reviewed changes

resolved pow error and added documentation

d713b35

davepagurek reviewed Oct 26, 2023

View reviewed changes

AryanKoundal added 4 commits October 27, 2023 17:03

added documentation and example

6656e90

resolved imageLight documentation example error

374051b

modified timeout to 25sec for imageLight examples to work

b964403

modified timeout to 25sec for imageLight examples to work

0e7c3bc

AryanKoundal marked this pull request as ready for review October 27, 2023 12:21

AryanKoundal added 2 commits October 27, 2023 17:53

changed settimeout back to original 4sec limit and changed the image …

14aed59

…Light examples to not run for the npm run test, those examples have been manually verified to be working

Merge branch 'main' into imageLight

5d99712

davepagurek reviewed Oct 27, 2023

View reviewed changes

slight modifications on the suggestions

4fe2890

davepagurek reviewed Oct 28, 2023

View reviewed changes

resolved suggestion

6fec78a

davepagurek reviewed Oct 30, 2023

View reviewed changes

AryanKoundal added 2 commits October 31, 2023 15:02

used WorldNormalInstead of R in lighting.glgl

94e639b

changed angles in lighting.glsl

f2c0f1e

added images in examples and removed unnecessary code

e68f9f0

AryanKoundal added 2 commits November 3, 2023 15:03

Merge branch 'main' into imageLight

6df91ec

replaced asset with smaller size

fe5e0db

davepagurek merged commit cfe48f5 into processing:main Nov 3, 2023
2 checks passed

This was referenced Nov 6, 2023

Image light performance improvements #6531

Closed

Solves issue #6502 #6503

Merged

inaridarkfox4231 mentioned this pull request Dec 11, 2023

Fixed example broken by #6216 and added information for new camera default setting #6620

Merged

3 tasks

inaridarkfox4231 mentioned this pull request Jan 5, 2024

orbitControl() Compatibility with imageLight() #6688

Closed

17 tasks


		const float PI = 3.141;

		vec2 normalToEquirectangular( vec3 v){

	return pow(outColor.xyz, 10.0);
	return pow(outColor.xyz, vec3(10.0));

	* first one is creating an irradiance map from the
	* the first one is creating an irradiance map from the

	* Second one is managing reflections based on the shininess
	* The second one is managing reflections based on the shininess

	vec2 angles = vec2( phi, theta );
	vec2 angles = vec2( fract(theta + 0.75), 1.0 - phi );

	return pow(texture.xyz, vec3(10.0));
	return smoothstep(vec3(0.0), vec3(0.8), texture.xyz);

	* The image light stimulates light from all the directions.
	* The image light simulates light from all the directions.

Image Light Feature - GSOC 2023 #6255

Image Light Feature - GSOC 2023 #6255

Conversation

AryanKoundal commented Jul 6, 2023 • edited Loading

Project Description:

Changes:

Screenshots of the change:

PR Checklist

davepagurek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AryanKoundal Aug 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AryanKoundal Aug 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davepagurek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AryanKoundal Oct 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davepagurek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davepagurek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AryanKoundal commented Oct 31, 2023

davepagurek commented Oct 31, 2023

davepagurek commented Oct 31, 2023

AryanKoundal commented Jul 6, 2023 •

edited

Loading

AryanKoundal Aug 11, 2023 •

edited

Loading

AryanKoundal Aug 11, 2023 •

edited

Loading

AryanKoundal Oct 28, 2023 •

edited

Loading