Updating surfaces in "real time" for persistance

[eniv]

Hi,
I'm using WebglSquare as a surface for a dynamic xy scatter scatter plot as suggested in #72.
I'm trying to create a persistence function where points from previous frames will not be cleared when a new frame is drawn.

The most naive way of doing that is holding a FIFO (first in first out) type, fixed size, buffer in CPU RAM which will hold the contents of all the points from the current frame and previous frames. This buffer can then be transferred to webgl-plot for rendering after all the surfaces are removed (Adding removeSurfaces, equivalent to removeXXXXLines to webgl-plot would be nice).
Using this technique I can preserve about ~4000 points before the frame rate starts dropping.

Can you recommend a better way of doing this? It seems redundant to buffer the same data to the GPU over and over again until it is removed from the FIFO.

[danchitnis]

Hello, thanks for getting in touch. I see your point, persistence makes sense for scatter graphs. As you mentioned, the problem is with overloading. There needs to be a kind of rolling buffer so that it can retain the last 4000 points. The problem is Javascript doesn't have a native rolling buffer, and it is hard to make one because it doesn't have access to low-level memory as C/C++ does. I will do some tests to come up will a workable solution. The GPU method is better. Each object needs an id. I haven't tried removing objects in OpenGL. Again need to do some tests.

[danchitnis]

@eniv Okay, the WebGL code was more challenging than I expected. But I managed to do it. At the moment, it is single-size, single-color squares. I plotted 1M squares with 100k square updates per frame at 60fps. I think that is beyond your 4000 sq limit! I still need to do more work to integrate it in the webglplot and create API etc. The rolling line graph will also get a new code. To be clear, it has a MaxSquare capacity, and when you add a new square, it overwrites the oldest one when it reaches capacity.

[eniv]

To be clear, it has a MaxSquare capacity, and when you add a new square, it overwrites the oldest one when it reaches capacity.

Exactly, just like a shift register.

This is awesome!
I'm looking forward to trying it out.

[danchitnis]

Here is an example that I made. The new version is at webgl-plot@next. Feel free to play around. Notice that in this example, the maxSquare should be a multiple of newDataSize, otherwise some data will be lost. The API is at a very early stage, so it may change.

[eniv]

Excellent!
I modified the example for maxSquare of 1e6 squares and newDataSize of 1e5 squares and everything runs smoothly. This looks very very promising.

[eniv]

Hi,
I experimented with this more today.
First of all, thank you!

These are a couple of items which I've noticed so far:

1. It will be really nice if there was a clear function in WebglScatterAcc for clearing the buffer. It is useful for switching persistence OFF.
2. WebglScatterAcc requires its own canvas and therefore can't be easily integrated with other shapes (line, polar, ...). This means that adding a grid for example, will become more complex. Sharing a single canvas under a single context as was previously done in WebglPlot was advantageous and allowed more sophisticated plots

-Eyal

[danchitnis]

Thanks. I had already made an update. There were a few bugs in the previous version, also added color now. You can see the old example updated, and also this new example. On my desktop (i7 9th gen with RTX A4000), I could go up to 6M maxSquares and 100k newData at 60 fps. There are two bottlenecks. The maxSquare is determined by GPU cores (not memory), and the new data is determined by the CPU thread. The data rate of the newData is equal to 528 Mbps, and the GPU core utilization is 50% which more than some of the games out there! It would be interesting to see how the python and C++ versions of this perform (they are in the works!)

[danchitnis]

It will be really nice if there was a clear function in WebglScatterAcc for clearing the buffer. It is useful for switching persistence OFF.

Yes, I can see a way to do that temporarily. But the best is to set the maxSquare equal to the newData

WebglScatterAcc requires its own canvas and therefore can't be easily integrated with other shapes (line, polar, ...). This means that adding a grid for example, will become more complex. Sharing a single canvas under a single context as was previously done in WebglPlot was advantageous and allowed more sophisticated plots

yes, this is the next priority. I am working on it. All of this will be part of the next version so it will be a re-write of the entire library.

[danchitnis]

Updated for the "auxiliary" lines. Example here.

[rhard]

Hi @danchitnis, I tried the scatter plot from your latest example here: https://codesandbox.io/s/webglscatteracc2-psutjy?file=/src/index.ts

I have two issues with it:

1. There is a white rectangle in the middle of the plot, visible in the first ~1 sec of rendering. You can also see the issue in your example:

2. The second issue is more interesting: as soon the initial rectangle goes away, my plot slows down to some 0.5 frames per second and remains slow. I don't see any performance degradation in general, and my drawing callback still executed fast. In the first second, the plot updates normally (~15 fps).

Maybe these two issues have the same origin?

Here is my drawing code. I use Tauri + SvelteKit frameworks. The version of webgl-plot is 1.0.2.

<script lang="ts">
	import { WebglPlot, WebglScatterAcc, WebglAux, ColorRGBA } from 'webgl-plot';
	import { onMount } from 'svelte';
	import { listen, emit } from '@tauri-apps/api/event';

	const horizontal_resolution = 15;
	const vertical_resolution = 11;

	let canvas: HTMLCanvasElement;
	let wglp: WebglPlot;
	let aux: WebglAux;
	let sqAcc: WebglScatterAcc;

	const maxSquare = 165;
	let screenRatio: number;
	let sqSize = 0.06;
	let h_scale: number;
	let w_scale: number;
	let h_offset: number;
	let w_offset: number;

	 const pos = new Float32Array(horizontal_resolution * vertical_resolution * 2);
	 const colors = new Uint8Array(horizontal_resolution * vertical_resolution * 3);

	onMount(async () => {
		canvas = document.getElementById('my-canvas') as HTMLCanvasElement;
		const devicePixelRatio = window.devicePixelRatio || 1;
		canvas.width = canvas.clientWidth * devicePixelRatio;
		canvas.height = canvas.clientHeight * devicePixelRatio;
		screenRatio = canvas.width / canvas.height;

		wglp = new WebglPlot(canvas);
		wglp.gScaleX = screenRatio;
		wglp.gScaleY = 1;
		h_scale = 2 - sqSize * 2;
		w_scale = 1/screenRatio * (2 - sqSize*2*screenRatio);
		h_offset = (1 - sqSize);
		w_offset = (1/screenRatio - sqSize);
		sqAcc = new WebglScatterAcc(wglp, 1650);
		sqAcc.setSquareSize(sqSize);
		sqAcc.setColor(new ColorRGBA(255, 255, 0, 255));
		sqAcc.setScale(1/screenRatio, screenRatio);

		const unlisten = await listen('frame', (event) => {
			let current_data_idx = 0;
			let countX = 0;
			let countY = 0;

			for (let vi = 0; vi < vertical_resolution; vi++) {
				for (let hi = 0; hi < horizontal_resolution; hi++) {
					countX = hi / (horizontal_resolution - 1) * h_scale - h_offset;
    				        countY = vi / (vertical_resolution - 1) * w_scale - w_offset;
					pos[current_data_idx * 2] = countX;
					pos[current_data_idx * 2 + 1] = countY;
					colors[current_data_idx * 3] = (event.payload as Array<number>)[current_data_idx]/10;
    				        colors[current_data_idx * 3 + 1] = (event.payload as Array<number>)[current_data_idx]/10;
    				        colors[current_data_idx * 3 + 2] = (event.payload as Array<number>)[current_data_idx]/10;
					current_data_idx++;
				}
			}
			sqAcc.addSquare(pos, colors);
			wglp.clear();
 			sqAcc.draw();
		});
	});
</script>

<div class="h-full flex items-center">
	<canvas class="border-8 h-600 border-black w-818" id="my-canvas" />
</div>

[danchitnis]

@rhard Thanks for the feedback

There is a white rectangle in the middle of the plot, visible in the first ~1 sec of rendering. You can also see the issue in your example:

Correct, I need to investigate this.

The second issue is more interesting: as soon the initial rectangle disappears, my plot slows down to some 0.5 frames per second and remains slow. I don't see any performance degradation in general, and my drawing callback still executed fast. In the first second, the plot updates normally (~15 fps).

This doesn't happen on my side. If you are already running at 15 fps, that means you are severely resource-limited. Reduce the buffer size in the line below until you get the frame rate of your monitor, e.g. 60 fps. Monitor your resources to see if you are not out of RAM and if GPU is below 50%. It is expected that you run this on a high-end CPU and mid-range GPU.

const sqAcc = new WebglScatterAcc(wglp, 10000);

[rhard]

@danchitnis Thank you for your prompt response.

By 15 fps, I mean this is how fast my component receives the new data inside the event listener here, and I redraw the rectangles:

const unlisten = await listen('frame', (event) => {}

I also call wglp.clear() and sqAcc.draw() functions inside this handler. This is different from your examples, where you use requestAnimationFrame loop. But I also tried to make the same loop as yours and just add the rectangles inside my event handler, but the results were pretty the same.

How can I output the real rendering FPS? I don't feel real fps is degraded, and it looks like just the new data for drawing became lost or something. Furthermore, the slowdown happens exactly at the moment when the white rectangle in the middle disappears.

[rhard]

I've also tried to reduce/increase the buffer with different numbers. I even tried to make it to be x1 with just 165 pixels needed for one single frame. This doesn't make any visible difference.

[rhard]

@danchitnis I've created an example to mimic my use case here: example

It woks OK, but it doesn't have the white rectangle at the beginning. Something strange.

[danchitnis]

By 15 fps, I mean this is how fast my component receives the new d

It is a best practice that you run the plotting function in Javascript's render loop, otherwise may have unexpected results. To check the frame rate in Chrome, follow the instructions here and then Rendering -> Frame Rendering Stats

I am not familiar Svelte framework, but these usually modify the states and rendering loops, so it could be the issue is there.

[danchitnis]

@danchitnis I've created an example to mimic my use case here: example

It woks OK, but it doesn't have the white rectangle at the beginning. Something strange.

The grey square, in the beginning, is not an issue. This is a rolling buffer, which is initialized with grey squares at position (0,0). I will make an API to move it around and look nicer, but it will always be there, even if it is hidden. This has no impact on performance. Your example runs just fine in vanilla scripts, so you have to check what Svelte is doing to it.

[rhard]

@danchitnis I found the issue in my code. I was using different numbers for maxSquare and the actual number of data points. When I put the same number, everything starts working.

But now I can recreate the issue in the example I sent you before: I set the maxSquare to be 10x more than actual data point number (which I think is the correct use case for persistence), and it starts freezing: example

[rhard]

Ok, I think I misinterpreted your comment here and did the opposite:

So, the maxSquare could not be bigger than new data point arrays.

[danchitnis]

But now I can recreate the issue in the example I sent you before: I set the maxSquare to be 10x more than actual data point number (which I think is the correct use case for persistence), and it starts freezing: example

It depends on what you want to do. None of these examples is incorrect. You are allocating a buffer size of 1650 and then, in each loop, updating 165. So, of course, it will update 10x slower! As I mentioned, this is a rolling buffer, so it may be a bit confusing if you aren't familiar with this concept.

Also, you don't need to create pos and color arrays. This defeats the purpose of the library. Just calculate your square positions and add it using addSquare() in the nested for loop. Once you do all your modifications, do a draw() call. I re-wrote your example in a cleaner form here.

[rhard]

@danchitnis Thank you for clarification and support!

In my understanding the rolling buffer is just a normal circular or ring buffer. I was expecting I can redraw it's content as soon as I call sqAcc.draw();, but from your last message looks like the new content will be ready to be drawn only when I fill the whole buffer length?

Also, if I understand you correctly, it is better to call addSquare inside the loop one by one. Is it really better from performance point of view then to fill the whole buffer and call addSquare at the end? I receive the whole new data in a single packet anyway.

[danchitnis]

@rhard

I was expecting I can redraw it's content as soon as I call sqAcc.draw();, but from your last message looks like the new content will be ready to be drawn only when I fill the whole buffer length?

The buffer gets updated whenever you call addSquare(). The canvas gets updated whenever you call draw(). That is also when the GPU work is done.

Also, if I understand you correctly, it is better to call addSquare inside the loop one by one. Is it really better from performance point of view then to fill the whole buffer and call addSquare at the end? I receive the whole new data in a single packet anyway

As compiler people say, don't try to outsmart the compiler! In your case simply use addSquare() wherever you want. Then WebGLPlot will deal with how to draw it.

[rhard]

@danchitnis

The buffer gets updated whenever you call addSquare(). The canvas gets updated whenever you call draw(). That is also when the GPU work is done.

Correct, why then canvas do not update when I add just 165 squares to the 1650 circular buffer in the issue with “freezing” above? example. Every bunch of 165 squares has the same coordinates.

I don't mean to bother you, but having 15-year programmer experience in C/C++, I still can get the full understanding.

[danchitnis]

@rhard, when having 165 fixed positions and drawing 1650 squares, ten squares are on top of each other, and you see only the top square changing colour, not the one underneath. See this example, which slightly randomizes their position. Nothing to do with C++ experience. OpenGL implementation is complex and unclear in some situations.

[eniv]

Hi,
Just curious if you are planning to release v2 branch to npmjs at some point?

[danchitnis]

@eniv Hi, it has been there for a while. See here and use the next tag. I will not make a full release until I decide on the API functions. Meanwhile, I can keep patching the next version.

[eniv]

Thanks, I completely missed it.

[eniv]

It will be really nice if there was a clear function in WebglScatterAcc for clearing the buffer. It is useful for switching persistence OFF.

Yes, I can see a way to do that temporarily. But the best is to set the maxSquare equal to the newData

WebglScatterAcc requires its own canvas and therefore can't be easily integrated with other shapes (line, polar, ...). This means that adding a grid for example, will become more complex. Sharing a single canvas under a single context as was previously done in WebglPlot was advantageous and allowed more sophisticated plots

yes, this is the next priority. I am working on it. All of this will be part of the next version so it will be a re-write of the entire library.

Hi,
Would you mind explaining in a bit more detail how to clear WebglScatterAcc? My understanding of the suggestion above is to simply allocate it again with a size equal to newDataSize, however, this causes the rendering to hang.

[danchitnis]

@eniv You do not need to reinitialise as that would disrupt the WebGL flow. Instead, re-assign the position of squares in the buffer. For example, if all are set to (0,0), they will overlap on the centre of the coordinate. If you want them to disappear from the screen once you reset, you can set the position of the squares to a location outside your viewport. In that way, they are not visible once you reset them.

[eniv]

That means that I have to keep track of all the surfaces I've already given to WebglScatterAcc, correct?.
Ultimately, I'm looking for a way to dynamically reset and resize the rolling buffer, but I believe resizing would hurt performance. Perhaps it is possible to draw only the first N squares, where N can be an input argument to WebglScatterAcc::draw()?

[danchitnis]

@eniv Ideally if you re-assign the const sqAcc = new WebglScatterAcc(wglp, maxSquare); it should destroy the previous object and create a new one. Most likely, the webgl object and canvas needs to be reassigned too. If you using this in a web app then React should take care of that. Otherwise, the buffer is not dynamic, hence the acceleration. Feel free to create an example and I will debug.

Perhaps it is possible to draw only the first N squares, where N can be an input argument to WebglScatterAcc::draw()?

yes, this is easy, I think the function to assign the n-th square should already be there. I need to check!

[eniv]

Thank you.

This const sqAcc = new WebglScatterAcc(wglp, maxSquare); worked, however, there is an FPS performance toll when persistence is toggled between ON & OFF . It sounds like this can be avoided by WebglScatterAcc::draw() taking a parameter as mentioned above, but probably not a big deal.

Also I think the setScale(scaleX, scaleY) & setOffset(offsetX, offsetY) APIs(very useful for an oscilloscope application, particularly when persistence is ON) are a bit confusing (at least to me). Although It is possible that I simply misunderstood how to use them. I assumed that a square centered at (x, y) will be transformed to (x*scaleX + offsetX, y*scaleY + offsetY) in clip space and that the size will not be affected at all. Can you describe what you had in mind?
Does this make sense in the vertex shader?

gl_Position = vec4(u_size * squareVertices[gl_VertexID] + (u_scale * position) + u_offset, 0.0, 1.0);