Remove camera influence on image() in WebGL in p5.js

[inaridarkfox4231]

Increasing Access

Currently, by using the image function in WebGL of p5.js, you can paste images created with createGraphics onto the canvas.
However, since it is realized by pasting it as a texture to the rectangle drawn in immediate mode, it will be affected by the camera with the current specifications.

function setup() {
  createCanvas(600, 400, WEBGL);
  const gr = createGraphics(600, 400, WEBGL);
  gr.background(128);
  noStroke();
  camera(400,400,400,0,0,0,0,0,-1);

  clear();
  image(gr, -300, -200, 600, 400, 0, 0, 600, 400);

  directionalLight(255, 255, 255, 0, 0, -1);
  ambientLight(64);
  ambientMaterial(255);

  fill(255,200,200);
  plane(800, 600);
  translate(0, 0, 100);
  fill(0,128,255);
  sphere(80);
}

Current Output

There are several ways to use this gray graphic as a background, but if you want to move the camera and use it as a background like this, you have to switch cameras one by one.
In addition, the object will fall behind the background if the depth is not turned off.

If you do not use the image function to set the background and use it as it is even if there is an effect of the camera, it is the same as pasting the texture to the object drawn with rect, so it seems not very useful I think (subjectively).

Therefore, as for the behavior of the image function in WebGL, even if the method of specifying the second and third arguments follows the coordinate system with the center of the origin and the y-axis pointing downward, it is desirable that the rest be the same as in 2D.

Expected Output:

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build Process
• Unit Testing
• Internalization
• Friendly Errors
• Other (specify if possible)

Feature enhancement details

To achieve this, when drawing rectangles in immediate mode, set the camera back to default and turn depth off while drawing.

src/webgl/3d_primitives.js
in p5.RendererGL.prototype.image
before:

this.beginShape();
/*~~~~~~~~~~*/
this.endShape(constants.CLOSE);

after:

const gl = this.GL;
gl.disable(gl.DEPTH_TEST);
this._curCamera._setDefaultCamera();
this.beginShape();
/*~~~~~~~~~~*/
this.endShape(constants.CLOSE);
gl.enable(gl.DEPTH_TEST);

But this alone is not enough. This is because calling _setDefaultCamera on _curCamera does not set the camera to its default state.
The cause is that information about the default state is not registered when making a copy of _curCamera in the push() that is performed before this.

So we do this by calling _computeCameraDefaultSettings in the camera's copy() function.

src/webgl/p5.Camera.js
in p5.Camera.prototype.copy
before:

  return _cam;
};

after:

  _cam._computeCameraDefaultSettings();
  return _cam;
};

[davepagurek]

I think this is intended behaviour? It's a little different than 2D mode because in 2D mode we don't have a concept of a camera in 2D mode, but a valid use of image() in 3D mode is to add a textured plane at a position in space (e.g. when people make virtual art galleries), so changing this behaviour would be a breaking change for those sketches.

Is the main use case here to be able to add an image background in WebGL? maybe this could be better solved by addressing that in a different way? It's currently achievable by creating two cameras and switching to one just for drawing the background, so we could make an example showing that.

[inaridarkfox4231]

I see.
Using image() to set the background can be achieved by changing the depth test before and after calling image() like so:
image_sample0

let bg, gl, defaultCam, cam

function setup() {
  createCanvas(400, 400, WEBGL);
  gl = this._renderer.GL;
  bg = createGraphics(400, 400);
  bg.noStroke();
  for(i=0;i<400;i++){bg.fill(0,0,i*255/400);bg.rect(0,i,400,1)}
  
  defaultCam = createCamera();
  cam = createCamera();
  cam.camera(300,300,300,0,0,0,0,0,-1);
}

function draw(){
  // do like this
  setCamera(defaultCam);
  gl.disable(gl.DEPTH_TEST);
  image(bg, -200, -200);
  gl.enable(gl.DEPTH_TEST);
  setCamera(cam);
  
  myLights();
  noStroke();
  fill(255);
  plane(400);
  translate(0,0,80);
  fill(255,128,0);
  sphere(20);
}

function myLights(){
  directionalLight(255,255,255,0,0,-1);
  ambientLight(64);
  ambientMaterial(128);
}

However, running current p5.js does not do this. This is because uMVMatrix is ​​not set to cameraMatrix in setCamera(), so it is not possible to draw with multiple cameras in the same loop. So rewrite setCamera() like that:

 */
 _main.default.prototype.setCamera = function (cam) {
   this._renderer._curCamera = cam; // set the projection matrix (which is not normally updated each frame)
   this._renderer.uMVMatrix.set(cam.cameraMatrix.mat4[0], cam.cameraMatrix.mat4[1], cam.cameraMatrix.mat4[2], cam.cameraMatrix.mat4[3], cam.cameraMatrix.mat4[4], cam.cameraMatrix.mat4[5], cam.cameraMatrix.mat4[6], cam.cameraMatrix.mat4[7], cam.cameraMatrix.mat4[8], cam.cameraMatrix.mat4[9], cam.cameraMatrix.mat4[10], cam.cameraMatrix.mat4[11], cam.cameraMatrix.mat4[12], cam.cameraMatrix.mat4[13], cam.cameraMatrix.mat4[14], cam.cameraMatrix.mat4[15]);
   this._renderer.uPMatrix.set(cam.projMatrix.mat4[0], cam.projMatrix.mat4[1], cam.projMatrix.mat4[2], cam.projMatrix.mat4[3], cam.projMatrix.mat4[4], cam.projMatrix.mat4[5], cam.projMatrix.mat4[6], cam.projMatrix.mat4[7], cam.projMatrix.mat4[8], cam.projMatrix.mat4[9], cam.projMatrix.mat4[10], cam.projMatrix.mat4[11], cam.projMatrix.mat4[12], cam.projMatrix.mat4[13], cam.projMatrix.mat4[14], cam.projMatrix.mat4[15]);
};

It can also be used to display texts by doing the same drawing process at the end of the loop.
image_sample_1

  /* ~~~~~~ */
  sphere(20);
  
  // do like this
  setCamera(defaultCam);
  gl.disable(gl.DEPTH_TEST);
  image(info, -200, -200);
  gl.enable(gl.DEPTH_TEST);
  setCamera(cam);
}

Additionally, it can be run in combination with orbitControl.
image_sample_2

So I would like to do something like this:

• I want to make it as easy as possible to use image() for background drawing etc., so I would like to change setCamera() like this.
• I will launch a new pull request to make the above changes.
• I would like to close pull request #5948.

[inaridarkfox4231]

Oh, the third line of setCamera() can be resetMatrix()... (because I modified it that way).

Also, I prepared a sketch that uses image() to set the background. So, if possible, if you have an example of a sketch to create virtual art galleries using image(), could you please show me...?
It's fine if you can't. I am aware that this is a breaking change, so I am withdrawing this proposal.

[davepagurek]

Ah good catch about uMVMatrix. Looking into it again, I think the reason why it currently doesn't do that is because it would mean losing the current transform since the model matrix and camera matrix aren't separate (#5287). If we wanted to preserve just the model matrix when switching cameras, it would involve multiplying the inverse camera matrix of the first camera and then multiplying the second again I think (or, a bigger change, separating this one matrix into two.) Unfortunately neither of these methods seem like quick changes, but if you're up for working on them, I think it would be a worthwhile change!

For virtual galleries, while I think this was in Three.js, a few years ago NYU had a virtual gallery showcase for some of its student projects, where one could walk around and chat with other users, and also see "framed" images: https://youtu.be/XBPbyMKjbDw?t=7457 One could also do this with a textured plane, but currently image() also works, and benefits from the fit/fill APIs on image.

[inaridarkfox4231]

I think that the reason why only uPMatrix is ​​set inside setCamera() is that it is not supposed to draw with multiple cameras in the same loop. As long as uMVMatrix is ​​set to _curCamera's camera matrix in _update, it can be undeniable that in some cases it will be a hassle, but it is much easier than working on a large change rather than separating the matrix. think. In any case, if it is impossible to change the content proposed in this issue, it will be difficult no matter what means you use, so if you think about a simpler solution, I think modifying setCamera() is your best bet.

Dropping text on graphics in webgl is tricky. It doesn't have to be 3D, and if you want to paste text drawn on a 2D graphic, you have to mess with the depth. It is a high hurdle for beginners. It would be much easier if it was pasted without any special fuss. I have to reset the camera as well. Just to drop the text. Of course, the effect of depth and camera cannot be ignored even if you drop 3D text, it's the same thing.

Then what I'm asking is what kind of p5.js webgl image() is used to place an image in space, is it appropriate to use, and is it convenient? For example, it's about explaining with source code, not about what a virtual art gallery is like. There are many ways to do this (three.js, unity, blender, etc). Is p5.js the right way to go?

Thank you for your reply. thanks.

[davepagurek]

I think modifying setCamera is the way to go too. The thing with the matrices would be to handle this case:

translate(200, 0, 0)
setCamera(someCamera)
box()

Since translate modifies uMVMatrix, if we reset uMVMatrix in setCamera, the translation will be lost. I think if we don't handle that case for now it's OK, we'll just need to leave a note in the docs maybe, because one could always do this to make sure the translate doesn't get lost:

setCamera(someCamera)
translate(200, 0, 0)
box()

If we do want to handle it, splitting the matrix into the camera component and the model component would be one way, but more work. Since uMVMatrix is equivalent to modelTransforms * cameraTransforms (the model transforms get left-multiplied on top of the camera matrix, if we wanted to make setCameraturn it intomodelTransforms * newCameraTransforms, we'd need to right-multiply by the inverse * the new transforms: modelTransforms * cameraTransforms * cameraTransforms^-1 * newCameraTransforms`

In p5 code, that would be something like:

this._renderer.uMVMatrix = newCamera.cameraMatrix.copy()
  .mult(new p5.Matrix().invert(this._renderer._curCamera.cameraMatrix))
  .mult(this._renderer.uMVMatrix)

This is still more work than just resetting uMVMatrix but it's maybe worth testing? But if there are complications, I think it's still OK using your proposed update to setCamera without modification as long as we document this side effect.

Dropping text on graphics in webgl is tricky. It doesn't have to be 3D, and if you want to paste text drawn on a 2D graphic, you have to mess with the depth. It is a high hurdle for beginners. It would be much easier if it was pasted without any special fuss. I have to reset the camera as well. Just to drop the text. Of course, the effect of depth and camera cannot be ignored even if you drop 3D text, it's the same thing.

For this, I wonder if it makes the most sense to use createGraphics for the 3D parts, so that one can use true 2D mode to add 2D overlays? I think the ability to combine 2D and 3D so easily is one of p5's best features, so maybe that should be the recommended way to achieve this?

[inaridarkfox4231]

It's not a breaking change to the conventional case of using only the same camera in the same loop, and it's a simple process, so I don't understand why it can't be accepted. I can't handle it myself, so I want to leave it to someone else.
I would like to close this issue.

Also, no comment about virtual art gallery. Until the end, I didn't understand the usefulness of image(), which is affected by 3D cameras and depth tests (In the first place, if it was accepted, it shouldn't have been talked about modifying setCamera...). It's a pity that we disagree. It was a fun discussion. Thank you very much.

Shouldn't we appeal the usefulness not with words but with the source code? That's more persuasive. It is impossible to convince with words alone. I was claimed usefulness in the form of shortening the source code and dramatically reducing the number of global variables to be prepared, but why does it claim that such a specification has value without writing even one source code? Is it possible? I didn't understand.

[davepagurek]

It's not a breaking change to the conventional case of using only the same camera in the same loop, and it's a simple process, so I don't understand why it can't be accepted. I can't handle it myself, so I want to leave it to someone else.
I would like to close this issue.

oh I still think we can accept the change as you described, because like you mentioned, it would only be a breaking change for a small case, and we can always address that edge case in a follow-up later. I just wanted to explain what my suggestion was first in case it looked like it might be feasible, but I think we can still open up #5953 again!

Also, no comment about virtual art gallery. Until the end, I didn't understand the usefulness of image(), which is affected by 3D cameras and depth tests (In the first place, if it was accepted, it shouldn't have been talked about modifying setCamera...). It's a pity that we disagree. It was a fun discussion. Thank you very much.

Sorry I haven't had the time to dig through sketches yet to find concrete code examples, unfortunately code search on places like OpenProcessing isn't really a thing yet so it's a little time consuming to do.

Shouldn't we appeal the usefulness not with words but with the source code? That's more persuasive. It is impossible to convince with words alone. I was claimed usefulness in the form of shortening the source code and dramatically reducing the number of global variables to be prepared, but why does it claim that such a specification has value without writing even one source code? Is it possible? I didn't understand.

I'm not sure what you mean just yet about specifications vs source code, would you mind elaborating a bit? But anyway there's always a balance to strike between making the source code small and maintainable, supporting existing behaviour, and making things easier to use. I'm just trying to explore the possibilities first before committing to one method, but it's just a discussion, we still might decide on implementing one of the first ideas after discussing some other ones 🙂

[inaridarkfox4231]

After much thought, I came to the following conclusions.

When preparing the background and text, it is better to combine texture() and plane() than relying on image()

I tried to make the following demo, but when I ran it on my computer, I could draw the background and text using image() or combining texture() and plane(), but I tried using a smartphone As a result, I found that using image() causes problems.
texturePlaneTest
I don't know the cause. But it still seems inappropriate to use image(). The processing from push() to pop() is also heavy, so I decided to avoid using image() for such purposes.

No need to change the specification of setCamera()

setCamera() is mostly used in applications where only the same camera is used in the same loop, as in the reference demo.
So I've come to the conclusion that drawing with multiple cameras in the same loop is rare, and in those cases using resetMatrix() as appropriate will give the desired result.
Duplicate processing with _update is also undesirable, so I thought it would be better.

Image() as a texture plane subject to camera and depth testing can be very useful depending on how you use it.

After that, I thought about what a virtual art gallery would be like. This led me to create a simple demo like this:
season img gallery

By using orbitControl(), you can see pictures of seasonal scenery drawn by image() on four walls. If you combine translate() and rotate() properly, you can use it like this, so I thought it would be a waste to be tied to 2D. Such a change would certainly be destructive and undesirable.

The image() modifications presented at the beginning of this issue are not adapted to 2D.

If you think calmly, 2D image() is affected by translate() and rotate(). However, such a modification would not even be affected by it. I thought it was certainly inconvenient and an undesirable change.

For the above reasons, we believe that such changes are inappropriate. Sorry...it was fun to discuss, thank you!

[davepagurek]

Thanks for taking the time to write up your conclusions! There's definitely a lot of overlap between ways to add an image in WebGL mode, so tests like yours are really helpful for guiding docs and future API design choices.

I found that using image() causes problems

This is interesting, I found toggling to an image() version in your OpenProcessing sketch would cause the spinning shapes to stop appearing, but when I tried copying the code into the p5 editor, that behaviour seems to stop happening: https://editor.p5js.org/davepagurek/sketches/RNQdMwuS7 Any ideas on what might be different between the two environments that might cause that?

The processing from push() to pop() is also heavy

Agreed, I think there's some work we can do there to profile what part takes the most time and try to optimize those calls (maybe when modifying a property, we can record that it's been modified, so we only save modified properties when pushing/popping?)