Should deltaTime account for throttling of unfocused tab?

[stalgiag]

Hi! This came up while fixing a problem with duplicate GIF timing (see discussion in #4824 ). Wanted to ask a larger question about timing and deltaTime here before throwing a workaround into the GIF timing.

My question is: Should a coder expect there to be unpredictable spikes in deltaTime when coming back to a sketch after switching tabs?

This sketch illustrates this. If you click into the sketch, switch tabs, and come back after a few seconds, you will probably see a number that is nowhere near 1000 next to 'deltaTime' in the console (depends on how 'busy' your CPU currently is)

I know this mostly comes down to a combination of window.performance.now and browser throttling behavior with unfocused tab animation callbacks.

This is less of a problem with millis() where a coder would expect the value to correlate to the time since the sketch started. But with deltaTime, the animation frame throttling makes the difference in 'real-time' between frames very unexpected even if it is true to the space between frames.

Should we account for this in deltaTime in some way? My instinct is no, but this isn't clearcut for me because sketches that rely on deltaTime will be very fragile to tab switching.

For me, this brings up larger questions about whether we should manually throttle unfocused sketches in a controlled way by default to make behavior predictable and consistent across browsers/hardware? What do others think? @limzykenneth @montoyamoraga @outofambit @akshay-99 @owenbmcc

Most appropriate sub-area of p5.js?

• Timing

[limzykenneth]

I'm thinking of this in terms of physics engine update, would the expectation be that the physics stops on unfocus? Also I'm not 100% sure the animation frame pausing on unfocus behaviour is consistent enough to not cause sync issues.

From my experience (ie. may be wrong) unfocus event also fires when the window is not active but the tab is on top, in which case the animation frame do continue to execute, which again can cause sync issues. Need to test this bit to be sure if any implementation would be possible.

[outofambit]

Should we account for this in deltaTime in some way? My instinct is no, but this isn't clearcut for me because sketches that rely on deltaTime will be very fragile to tab switching.

I agree with this. Maybe the thing to do here is add to the documentation for deltaTime to communicate this unique case?

From a quick look around the internet, it doesn't seem like there are ways to influence browser-level refresh throttling when tabs aren't foregrounded. (Unless you played audio file in the webpage, which should prevent the throttling.)

I also found that there's events tied to "Page Visibility", which could wrap in p5 so that creators could subscribe to this event if they needed to do something special to account for the expected jump in deltaTime. Something like onWebPageHidden onWebPageShown? But I'm not sure how helpful that really is. What do y'all think?

[limzykenneth]

I haven't come across the Page Visibility events before and a quick read through it I'm still not sure does it ties to the pausing of requestAnimationFrame? I'm a bit more reluctant to add additional APIs like onWebPageHidden/Shown, it would be best if this can be handled by the library itself.

[owenbmcc]

Fwiw, I use three.js as well, so out of curiosity I did a test on the three.js clock which has a getDelta method that is similar to deltaTime in p5 and the tests I did have the same issue as p5 when pausing the animation or changing window focus, the value of getDelta() spikes and the animation either jumps or gets stuck.

p5 has events for window blur and focus which appears to be triggered by changing the tab as well:
https://github.com/processing/p5.js/blob/main/src/core/main.js#L565

That could be used to pause the draw loop or offset the value of deltaTime. I haven't tested it much but seems to work okay. However, it's also triggered by things like opening the developer console or switching to another program window with the browser still visible.

But it seems like a pause in rendering at any point kind of makes calculating deltaTime the way it's intended impossible? I think it might make sense in this case to set deltaTime to 0 to prevent a big spike or a guess.

I made an attempt to fix this issue:
owenbmcc@00d461e

I don't know if this is the right approach, but it seems to fix things for the example I originally created:
https://editor.p5js.org/owenroberts/sketches/LO8vidvA2

This isn't directly related to the unfocused tab issue, but the issue I first noticed with GIF animations getting delayed is also caused by using loop and noLoop in a sketch.

I'm not familiar with the thinking that went into the animation loop for p5, so I'm curious looking at the _draw function in main.js, why is it that the requestAnimationFrame(this._draw) is only called if _loop is set to true?
https://github.com/processing/p5.js/blob/main/src/core/main.js#L409

Would it make sense to continue calling draw just to update the _frameRate, _lastFrameTime and deltaTime properties and then not calling redraw() if _loop is false? That would prevent the spike in deltaTime caused by using noLoop.

[limzykenneth]

@owenbmcc I wouldn't recommend tying it to blur and focus event as you mentioned switching focus while the window is still visible will fire the blur event as well and I find that potentially unusable especially since I can't pop the developer console out and monitor behaviour there. The page visibility API as mentioned above could be a solution but without trying it I don't know what its actual behaviour is like in relation to requestAnimationFrame

The implementation of the draw loop as you can imagine goes back a long time so I can't say much about it as I don't know that much. I think the idea of making deltaTime 0 (by controlling _lastFrameTime) is worth exploring, I'm not sure there's that much value in updating deltaTime when noLoop() as that will provide a non 0 value to deltaTime when redraw() is called which doesn't make sense to me.

[owenbmcc]

@limzykenneth I agree that makes sense.

I should have explained more clearly, the solution I came up with for the unfocused tab issue wouldn't pause the sketch completely if you open the console or focus another program. My approach was to set deltaTime to 0 only in the frame directly following a blur event, so if you switch tabs and requestAnimationFrame is no longer running, when you return to the tab, it will have been paused, but since requestAnimationFrame seems to continue running as long as the page is visible, it will only pause for 1 frame. I'm still not sure that it is the right approach, but just wanted to clarify.

[weslord]

For what it's worth, as a user I expect deltaTime to be the actual time between frames. If something causes a big drop in framerate, I'd hope to see a big spike in deltaTime.

Maybe the thing to do here is add to the documentation for deltaTime to communicate this unique case?

I would much prefer a warning in the documentation explaining that browsers may stop rendering animations when backgrounded, than p5 assuming that I want an artificially small/consistent deltaTime.

[limzykenneth]

@weslord I think that is a reasonable expectation as well.

[stalgiag]

I feel differently about this but not enough to advocate for a change.

In general, I do expect deltaTime to be the actual time between frames. But I guess I don't ever expect the time between frames to be inconsistent & unpredictable. Coming from Unity, where I primarily work and the original inspiration for deltaTime I believe, deltaTime can be relied on to never spike in a way that breaks the code unless the hardware is incapable of running the project. If a Unity project is unfocused it is frozen (including deltaTime). In Unity, if a frame is fired 15 milliseconds pass and then the project loses focus, deltaTime upon regaining focus will be 15 ms. If you want it to run its update cycle when unfocused you can set the project settings to do so, but then it demands all of the necessary resources to run consistently (ie it doesn't throttle). We don't have either of these options because our hand is forced by the browser. This means that deltaTime cannot be used in the same way that one could use it in Unity (or any game engine) because using the 'real-time' logic of deltaTime will break even simple sketches when the browser tab is switched.

Bouncing ellipse tied to real-time

This simple sketch breaks immediately upon tab switching. This wouldn't happen if we were doing the standard x += speed frame-by-frame because the throttling would constrain the values. But since deltaTime isn't constrained by the throttling, the value quickly jumps the ellipse offscreen.

In my opinion, this makes the most instinctive uses of 'real time' very fragile to tab-switching. The useful connection between frame-by-frame rendering and 'real-time' is made unusable by the throttling. I am not suggesting that we do anything other than mention this in the documentation and get rid of all internal uses of deltaTime but I do think this is a limitation that we need to work into our understanding of p5's relationship to 'real-time'.

[Qianqianye]

Relates to #6785, need to test