[p5.js 2.0] State machines and renderer refactoring

[limzykenneth]

Renderer state machine

The base p5.Renderer class will provide a states property that will keep track of all states within the renderer that are affected by push() and pop(). The base version looks like the following:

this.states = {
      doStroke: true,
      strokeSet: false,
      doFill: true,
      fillSet: false,
      tint: null,
      imageMode: constants.CORNER,
      rectMode: constants.CORNER,
      ellipseMode: constants.CENTER,
      textFont: 'sans-serif',
      textLeading: 15,
      leadingSet: false,
      textSize: 12,
      textAlign: constants.LEFT,
      textBaseline: constants.BASELINE,
      textStyle: constants.NORMAL,
      textWrap: constants.WORD
};

Each renderer that extends p5.Renderer should add any additional states to the states properties when it should be affected by push() and pop(), eg. this.states.someState = 'someValue'. The base implementation of push() and pop() will keep track of the states with push() pushing a copy of the current state into an array then pop() restores the last saved state in the array to the renderer's current state. (See line comments for a bit more details below)

The individual renderer should call super.push() and super.pop() in their own implementation of push() and pop() in addition to acting on the necessary changes in states.

OOP hierachy

The previous hierachy has p5.Element as the base class which p5.Renderer extends and p5.Renderer2D/GL again extends. This means all renderer are assumed to have a p5.Element backing. Part of the changes made here is to remove this assumption so p5.Renderer can be extended into renderers that does not render to a HTML element.

Eventually, the individual renderers p5.Renderer2D and p5.RendererGL will have their own reference to p5.Element that they can operate through.

Overall event handling is still something that needs to be think through for this case though.

Global mode

_setProperty() is no longer used. Global variables will now all be using getters that refers to the value in the p5 instance.

Pending global functions to be attached in the same way.

[limzykenneth]

@davepagurek Following on what we discussed regarding the renderer, I've set up an implementation of the renderer state machine here. For 2D most of the things are working since it is relatively simple, for WebGL however it seems there are some states not being tracked correctly, causing tests to fail.

[davepagurek]

oops I resolved the merge conflicts with the dev-2.0 updates, but that also brought in some more bits that I need to convert to use states. I'll push another commit soon

[limzykenneth]

@davepagurek I don't want to change too much of WebGL at the moment seeing that you and others are also working on it so I can't fix all the WebGL tests. The main failure is likely linked to p5.Framebuffer and possibly also the WebGL filters.

The overall code is still a bit too spaghetti, I'll try to get things a bit more streamlined but have a busy period coming up so I'll see how much I can complete.

I've added some notes below regarding things that eventually should be implemented, let me know if they don't make sense. Let me know if discussing over a call is easier as well.

• WebGL attributes should be set per renderer instead of per p5 instance: Fixes p5.RendererGL -> webglVersion -> works on p5.Graphics unit test
• WebGL filter for 2D canvas not working because the copy helper using the RendererGL image() method expects this._pInst to be in WebGL mode but since the renderer for the real p5 instance is 2D, it cannot call WebGL methods (eg. noLights(). Possible solution is to move these methods to be implemented within p5.RendererGL, with the instance method acting as wrapper of renderer method.
• p5.Graphics should not be treated as a valid p5 instance, ie. it should not be used where internal this._pInst is expected. Either use its _renderer property where necessary or the actual _pInst instead.

[limzykenneth]

I just thought of a possible idea for p5.Graphics, perhaps I can leverage the new module API where it is a function taking in the signature of function(p5, fn) where fn is p5.prototype, instead I can attach modules and methods to p5.Graphics so that it can become a pseudo p5 instance where necessary without elements that don't make sense.

[limzykenneth]

p5.Graphics has almost all the needed methods now, with the exception of blendMode() and clearDepth() (clearDepth() may make sense to move to the webgl module, blendMode() need a new home that is also included in p5.Graphics already blendMode() now lives in color/setting.js along side functions like background()).

Also WebGL methods are not attached to p5.Graphics yet as they need to use the new module syntax, @davepagurek do you have an idea of around when would be a good point to do that conversion for WebGL? Probably need to align with other things going on so we don't convert in the middle of someone's work.

Another thing is that not everything make sense to attach to p5.Graphics, with a general guideline being only renderer related stuff should be attached, other environment related stuff should use the p5 instance version (eg. constants, framerates, data loading, interactions, objects such as p5.Color so probably also p5.Texture etc as well).

[davepagurek]

@limzykenneth Luke's project should be wrapping up this week, so that could potentially be a good point to start? But also if we're mostly renaming properties and maybe moving where they're defined, I can resolve any conflicts that come up with Luke's and Garima's projects if this is in the way of more refactor work.

[davepagurek]

took a while to find the cause of the masking bugs, but the cause of that one was the ordering of _setAttributeDefaults and _initContext. The default attributes have to be set before initializing the context in order for us to pass the right values into canvas.getContext(). Without that, it falls back on the browser defaults, which does not include a stencil buffer (while p5 asks for one by default to make clipping work.) I don't think moving it back to be before _initContext failed any new tests, but let me know if there are other reasons for having it lower down in initialization and I can think about how to move around the necessary defaults another way.

It turned out to not be the underlying cause, but I also refactored fill and stroke a bit to have them also set states.doFill and such. No tests were broken because of that, but it might be a cause of a future subtle bug if internal fill doesn't enable filling while external fill does.

[limzykenneth]

I don't think _initContext was moved specifically but if it needs to be somewhere else feel free to move it.

[davepagurek]

Sounds good, I've moved it earlier so I'll just keep it there.

What do you think about having a separate pixelsState between the main canvas and graphics? In addition to pInst, should we add an argument to the Renderer constructor to also give it a pixel state object (the same as pInst for the main canvas; the graphic for a graphic?)

[limzykenneth]

From what I can see in the implementation, pixelState is an alias of pInst so I'm not sure what the difference is meant to be or rather just what pixelState is meant to do.

[davepagurek]

Right now it acts kind of like an interface that anything that supports loadPixels conforms to to be able to share a 2D loadPixels implementation. So right now it's kind of like this:

interface SupportsPixels {
  pixelsState: {
    _pixelDensity: number
    width: number
    height: number
    drawingContext: CanvasRenderingContext2D
    pixels: Uint8ClampedArray

    imageData?: ImageData // Cached image data after loadPixels called
  }
}

And then things that have this reuse the 2D renderer's loadPixels, updatePixels, set(), and get(). e.g. p5.Image:

p5.js/src/image/p5.Image.js

Lines 457 to 460 in 60dbf86

	loadPixels() {
	p5.Renderer2D.prototype.loadPixels.call(this);
	this.setModified(true);
	}

So if we want to keep using that pattern, pixelsState will be different between the main canvas and a graphic. Previously this was also the case, pInst in a graphic confusingly refers to something other than the main instance, which is what necessitated the weird gymnastics in the filter rendering:

p5.js/src/core/p5.Renderer2D.js

Lines 21 to 28 in 60dbf86

	getFilterGraphicsLayer() {
	// create hidden webgl renderer if it doesn't exist
	if (!this.filterGraphicsLayer) {
	// the real _pInst is buried when this is a secondary p5.Graphics
	const pInst =
	this._pInst instanceof p5.Graphics ?
	this._pInst._pInst :
	this._pInst;

...but now, I think pInst always refers to the main instance? Which I think makes more sense for most things overall, but does break the assumptions used in pixelsState. It might be time for a new design for that anyway since 2D and WebGL things implement it differently anyway.

Since graphics and the main canvas both have a Renderer, these could just all be on the renderer and not in a separate pixelsState object and that would be simpler. The slight complication is that p5.Image also wants to have these methods, and it doesn't have a renderer. Maybe we can split that functionality out into a class PixelsState2D?

[limzykenneth]

The pixelState interface basically are all covered by the 2D renderer and calling it there will be easier than using the instance. For anything called pInst going forward, it should refer to the actual p5 instance for consistency and simplicity, and also it centralizes the instance states like color mode.

For p5.Image it has most of the pixelState interface implemented as well, if the implementation of things like loadPixel() don't differ with the 2D renderer's implementation, we can consider making them mixins for the different class to share code as long as they themselves have pixelState interface.

For p5 instance, it won't need to implement pixelState as it will call the renderer's loadPixel() etc when called on the instance, the same as how we handle the rest of the renderer's interface.

[davepagurek]

I think one thing we'll need is a way for the renderer to "reach out" to the thing that owns it to set its pixels property. I see that p5.Graphics adds getters for all non-private renderer properties, which means we can just have the renderer own pixels if the main instance makes getters too.

I've added that, and caught a few cases where the renderer had a property with the same name as a main instance function (e.g. textureMode) that I had to change (textureMode should really be in states anyway), and a few cases in dom.js where this.elt was being assigned to the last created element in createSelect() and a few others (I don't think it was being used anyway and has been in the codebase forever it seems, so I just removed it.)

There are a few tests still failing that have to do with updatePixels in WebGL not working so I'll take a look at those next.

[davepagurek]

Actually, this approach works for now but it means a lot more things are now leaked into the global scope that probably shouldn't be, e.g. this.zoomVelocity, this.prevTouches, etc in RendererGL. I guess that was already true on p5.Graphics but it matters less there since people aren't usually defining their own properties on those, whereas stuff on p5 becomes global.

I'm going to just make it expose pixels for now.

[limzykenneth]

I don't know how much of the renderer we need to expose to the instance beyond pixels, but if possible and there aren't too many, we should manually expose them.

Basically what I think would be nice is that global variables like width, height, pixels are all getters to the renderer's own width, height, pixels etc but manually expose them instead of looping though all the renderer's properties. ie. Remove the need for the renderer to need to reach to the instance as much as possible and let the renderer handle changes to its own state, which will then reflect into the global state through the getters.

I believe pixels is the only case where we may need to consider a setter and all other variables are read only.

[davepagurek]

Turns out the updatePixels problems were because setting the imageMode to CENTER does nothing when calling this.image as opposed to the user-exposed image(). I think that's fine, and actually probably for the best that our internal implementations depend less on state, but we should watch out for more cases like this left over from before.

[davepagurek]

Technically we also don't need this line since it'll use corner mode regardless. Might be worth keeping for clarity, although then it's misleading because it looks like something that you can change when you actually can't. Maybe I should remove it?

[limzykenneth]

Yeah, that should be removed if it's doing nothing, I kept it around because I didn't know if it was actually needed or not and I just translated from what was before.

[davepagurek]

Ok last webgl test was because it was accessing a constant from a graphic, which is now undefined. Tests are all green on my end now!

[davepagurek]

Basically what I think would be nice is that global variables like width, height, pixels are all getters to the renderer's own width, height, pixels etc but manually expose them instead of looping though all the renderer's properties. ie. Remove the need for the renderer to need to reach to the instance as much as possible and let the renderer handle changes to its own state, which will then reflect into the global state through the getters.

Do you think there would be a case where a new renderer, e.g. SVG, would want to add new state visible to the end user? If so, it might make sense to make a way for the renderer itself to define what gets exposed (could be as simple as doing this.exposedProperties = ['width', 'height', 'pixels'] and then having the renderer loop over those.)

[limzykenneth]

Two failed on my end that looks like single pixel and off by one error

p5.RendererGL
default stroke shader
coplanar strokes match 2D
AssertionError: expected [ +0, +0, +0, 255 ] to deeply equal [ +0, +0, 255, 255 ]
 - /test/unit/webgl/p5.RendererGL.js:123:14
- Expected
+ Received

  Array [
    0,
    0,
-   255,
+   0,
    255,
  ]
color interpolation
bezierVertex() should interpolate curFillColor
AssertionError: expected [ 255, 128, 128, 255 ] to deeply equal [ 255, 129, 129, 255 ]
 - /test/unit/webgl/p5.RendererGL.js:1900:14
- Expected
+ Received

  Array [
    255,
-   129,
-   129,
+   128,
+   128,
    255,
  ]

If you can do a rebase and push it should run here on CI then we can check here.

[limzykenneth]

Do you think there would be a case where a new renderer, e.g. SVG, would want to add new state visible to the end user? If so, it might make sense to make a way for the renderer itself to define what gets exposed (could be as simple as doing this.exposedProperties = ['width', 'height', 'pixels'] and then having the renderer loop over those.)

That could be a thing but probably need to be a slightly more elaborate API as the renderer implementer might want to set both the getter and setter.

[limzykenneth]

Rebasing now

[limzykenneth]

@davepagurek All tests passed in CI!

Shall we merge this in then you can start working on additional WebGL refactoring while I work on some more details stuff to make internal API clearer?

[davepagurek]

Awesome! Sounds good, thanks for doing that rebase!