Add a _network_process() method to complement _process() and _physics_process()

[Calinou]

• Related to Implement semi-fixed timestep / manual stepping #236, Implement support for using different physics update rates at the same time #439, Fixing physics timestep vs refresh rate problems (jitter) #1936, and Networking API changes for 4.0 #1939.

Describe the project you are working on

The Godot editor 🙂

Describe the problem or limitation you are having in your project

There is no built-in way to send network data at a different rate than the physics update rate. However, you may want to do that to have accurate/low-latency client-side prediction but keep the network bandwidth usage low.

While 60 Hz network updates are a good baseline nowadays, large-scale/MMO games may have to settle for 30 Hz or even 20 Hz updates to keep the bandwidth usage manageable.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Like _process(delta) and _physics_process(delta), call _network_process(delta) every 1 / network_fps seconds if it's present in a node script.

Exposing the network FPS as a global value also allows easily changing the value to accomodate server load or playing conditions. This is often done in battle royale games to increase network smoothness as the number of alive players decreases.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Add Engine.network_fps (integer) to complement Engine.physics_fps. Also add the associated project setting like for physics FPS.*

*: If we rename Engine.physics_fps to Engine.physics_ticks_per_second, we should name Engine.network_fps Engine.network_ticks__per_second instead.

The default value would be 60 (identical to the physics FPS in 3.2.x). Unlike physics FPS, it will not vary depending on the monitor's refresh rate (if support for that is implemented in 4.0).

If this enhancement will not be used often, can it be worked around with a few lines of script?

Yes, but it's not exactly trivial to do so over a whole codebase. For instance, you could use the following code:

func _physics_process(delta):
    # Only send our player position every two frames (30 Hz if physics run at the default 60 Hz).
    if Engine.get_physics_frames() % 2 == 0:
        rset("position", Vector2(12, 34))

On top of that, the above code snippet will break if variable physics update rate is implemented in 4.0.

Is there a reason why this should be core and not an add-on in the asset library?

This is core engine functionality that can't be implemented at low-level by an add-on.

[barbaros83]

is it maybe possible to define your own process like functions where they update at a custom rate, that way we could implement our own loops, i mean you could do it with a timer but maybe doing it this way its more eficient.
think about a scenario like, updating the gui once a second instead of 60 times per second, could save a lot of proccesing power

[Calinou]

think about a scenario like, updating the gui once a second instead of 60 times per second, could save a lot of proccesing power

Updating a GUI in _process() is almost never a bottleneck. If it turns out to be, you can implement throttling/debouncing using a Timer node when you need it.

[jonbonazza]

This would be very useful for multiplayer games, especially server-side. I will rake a stab at this.

[AndreaCatania]

Usually, however, just a custom _process with a different tick Hz is not enough, for networking games. When you have prediction, and so a rewind mechanism, you still need to control the emission of that process; so you are still forced to create your own mechanism tick, making this new feature useless.

There are many ways to support this feature in GDScript. For example: you can register a singleton (where the network magic is computed) that emits a signal network_process. The singleton will emit such signal at fixed Hz or when it needs to do so, or both. At this point any Node that want to process at such rate, can just connect a function to that node.

Rather, I think would be more useful a Node, where you can specify the desired tick Hz. It will emits a signal at that specified Hz. In this way, this Node, can be used to process the networking, but also to process the HUD as @barbaros83 said, or anything else the user needs.

[Calinou]

Rather, I think would be more useful a Node, where you can specify the desired tick Hz. It will emits a signal at that specified Hz

Isn't that a Timer node? 🙂

The only difference is that time is specified in seconds rather than Hz, but it's only a different way to specify the time unit.

[AndreaCatania]

Yes, exactly, something like the Timer with the same mechanism the core uses to tick _physics_process at fixed Hz.

Then, you can simply make that a singleton; The nodes that needs it, can just do:

func _ready():
    MyNetworking.connect("process", self, "_network_process")

func _network_process(delta):
    # Make sure it's ticking at 60Hz.
    assert((1 / delta) == 60)

[jonbonazza]

@AndreaCatania i don't entirely agree with your assessment here.

In mant networked games such as MMOs that use basic client side prediction and server reconciliation, emmission rates from the client are irrelevant. The only rate that matters really is the tick rate of the server, as this is what determines the amount of egress data used. For other clients, basic entity enterpolation is enough because seeing them x ms behind is fine.

In some other more advanced approaches sure, this might not be enough, but i wouldnt go as far as to call it useless.

The question is, do we want to try to kill two birds with one stone here or treat them as separate problems

Edit: reading @AndreaCatania response again perhaps i am misunderstabding. Are you saying the flaw with this approach is not being able to adjust the tick rate on the fly?

[AndreaCatania]

What I'm trying to say is that I feel it like too much restrictive for a networking tick, to be really useful.

• You can't decide if tick it before or after the physics step (This is a key concept for the sync mechanism).
• You can't step it within the physics step, (if you need to interact with physics (network a RigidBody) you need to tick it within the physics).
• You can't manually control its tick (to perform reconciliation).
• This is always active even for single player game that doesn't use it, (which is a useless waste).

The node, as proposed here #2020 (comment), is much more easy to implement and doesn't suffer of the above issues, making it more useful.

So putting on the scales the _network_process implementation complexity and its versatility vs the complexity of the Node and its versatility; I think that the Node will be a much useful tool on the hands of the devs (and in the worst case it will not impact the processing when not used).

[jonbonazza]

Hmm... Ill try to reply in the order of your concerns

1. Im not following why the order matters here. I think that just ensuring that _network_tick occurs after _physics tick on frames where they both occur would be enough. Perhaps you could elaborate a bit on why this seems ti be a concern for you?
2. This one is a legitamate concern, though synchronizing rigid bodies is a lot more complicated than just doing it in a physics tick and likely wouldn't be supported by this feature regardless.
3. Im not sure what you mean by manually comtrolling the tick and how this would affect reconciliation. Can you elaborate a bit more on this one?
4. Well, id imagine just like _process or _physics_process, if it's not implemented by a Node, it won't be called.

In your singleton idea, whow would the signal emission be controlled? That is, when would the singleton emit the signal?

I think the goal here is to provide a simple, out of the box mechanism that solves this common problem (one novic multiplayer devs usually don't even know they have) in a convenient way for the 80% case.

As complicated as network cide is, i dont think we'll ever get sometging that works for 100% of use cases, and in those outluer cases, they can do something more custom.

[AndreaCatania]

If I got it right, what you are trying to achieve with this proposal is an out of the box mechanism for a simple problem, call rpc at a fixed Hz.

Your intention is to just use _network_process for that; and so:

• Collect and compress the inputs and the outputs.
• Sync the Characters at different rate depending the distance between each other.
• etc...
  are all things that you will implement in custom iterator and not using _network_process.

To me this, doesn't solve the 80% of the issues that you have to solve when you deal with a networking game that usually use more complex algorithms than just sending inputs at a fixed rate.

_network_process would just be used for 1 thing, so it's more restrictive than a node that can process at any rate, and it's much more customizable and so usable for many things.

In your singleton idea, how would the signal emission be controlled? That is, when would the singleton emit the signal?
The idea is to provide a node where you can set:

• Tick Hz
• Process mode (Physics / Idle (after physics) / Manual)
• Active
  Each time it has to tick, it emits a signal that the interested nodes can connect to it and perform operation at that Hz.

One important thing to highlight, is that while you can have what _network_process does #2020 (comment), you can also use it for complex stuff:

• The client has a singleton instance that uses to send inputs at fixed rate to the server.
• The server has one that is used to send the server status at fixed Hz to the clients.
• The server has one additional node for each peer, where the tick rate Hz change dynamically depending on the character distance; so faraway characters are updated with a lower precision for that specific client.
• This node can also be used for other things, and it's not tie specifically for networking. (I was thinking about the Dark Souls: where the faraway enemies' animation are processed with a lower Hz).

[jonbonazza]

@AndreaCatania i think there might be some misunderstanding of what this proposal is for. This proposal is intended more for the server side than the client side. It doesn't have much to do with sending rpcs, but instead, it provides a way for servers to tick at a constant rate that is different from the physics interval.

This could be useful, for example, when processing input messages from a client. You usually only want to process the serverside input buffer and replicate back to clients at something like 15fps, for instance to save on network bandwidth.

Edit: if this was a part of Node instead of a singleton, it can still distinguish between server and client to react accordingly, if for some reason you did want to use it on client (i still dont know why you would though)

Also, it fits better than a singleton. There are currently no existing out of the box autoloads and having a node that only works as a autoload would be not only unconventional but also difficult document. There's also a UX problem to solve there that is non-trivial.

[AndreaCatania]

This could be useful, for example, when processing input messages from a client. You usually only want to process the serverside input buffer and replicate back to clients at something like 15fps, for instance to save on network bandwidth.

If, for process inputs you mean read them, pack in a buffer, and provide it as inputs for the following _physics_process definitely _network_process is not going to help you (#2020 (comment)).

It's not as simple as just sync (/process) an input buffer at fixed rate and I've the impression that the scope in which this feature will be useful is really little. But, I don't really know how you will use it, so I'm probably wrong, so can you please explain (using pseudo code) how it will be used?

Edit: if this was a part of Node instead of a singleton, it can still distinguish between server and client to react accordingly, if for some reason you did want to use it on client (i still dont know why you would though)
Also, it fits better than a singleton. There are currently no existing out of the box autoloads and having a node that only works as a autoload would be not only unconventional but also difficult document. There's also a UX problem to solve there that is non-trivial.

The engine should not provide support for not well-defined workflow, as core features. Rather, it should provide the tools in the form of Nodes, so the dev can create its own workflow; so, anyone can use these for easy and complex stuff. The proposed Node would not be much different from a Timer or a Twean, which are perfectly documented.

[jonbonazza]

Ah! I see what you are saying now @AndreaCatania . I believe you are correct in this in fact.

Hmm instead of a singleton node, i wonder if we could just expose something in NetworkPeer or MultiplayerAPI that could be hooked into from gdscript.

[jordo]

I think this proposal may actually just lead to more confusion than anything else.

This is core engine functionality that can't be implemented at low-level by an add-on.

I don't think the above is true. This could be implemented with a global timer, or controller node with _process(delta time) and an accumulator for example.

Server tick rate (simulation rate) and broadcast (send or sync) rate are often different. These two things should be independent. They can be the same, but often they are not.

Often it's desirable to have server tick rate be relatively high (the server can receive input at any time asynchronously. So higher tick rate means smaller worse-case additional latency between receiving a client input, and processing the input. 60hz would be a maximum delay of 16ms if a input has unfortunate timing and is received just slightly after input processing on the server).

And send/sync/broadcast-rate is sometimes not desirable to have fixed globally (same for each client connection). If an individual client has a low bandwidth, you may not want to exacerbate a congestion problem by throwing the same amount of data at that client (more than they can handle). In which case the game server for that particular client connection may want to have some basic congestion avoidance which could bring that particular clients send/sync/broadcast rate down.

Exposing the network FPS as a global value also allows easily changing the value to accomodate server load or playing conditions. This is often done in battle royale games to increase network smoothness as the number of alive players decreases.

I have not heard of the above being done. I'm not sure this makes sense to me. What does 'network FPS' actually mean in the above context? If it means actual game 'simulation rate' (tick rate), then the benefit of reducing it dynamically is just to lower CPU overheard (at the cost of additional latencies introduced, and the complexity of syncing this change in real-time among clients and server). If it means 'network send rate', then I don't understand how decreasing this or increasing it changes 'network smoothness' (this term in of itself needs a much better definition in order for discussion to happen) for clients or servers unless egress bandwidth on your game server is an issue, which it's not likely to be in the suggested scenario.

I just think #2020 (comment) basically covers this proposal no?

[Calinou]

I have not heard of the above being done. I'm not sure this makes sense to me. What does 'network FPS' actually mean in the above context? If it means actual game 'simulation rate' (tick rate), then the benefit of reducing it dynamically is just to lower CPU overheard (at the cost of additional latencies introduced, and the complexity of syncing this change in real-time among clients and server). If it means 'network send rate', then I don't understand how decreasing this or increasing it changes 'network smoothness' (this term in of itself needs a much better definition in order for discussion to happen) for clients or servers unless egress bandwidth on your game server is an issue, which it's not likely to be in the suggested scenario.

Here, "Network FPS" refers to both the server physics simulation rate and network update rate. The main goal is to save on CPU resources, so maybe lowering only the server physics simulation rate would make more sense.

[jknightdoeswork]

A common use case for us in previous games has been to have an extremely high physics tick rate (ie 240hz) and a low network broadcast rate (ie 30hz). I don't think have a seperate process function is right here. It is very simple to build your own time accumulator in any given _process. We have done that already and likely would not use network_process simply because we would have to code review the whole system to determine if it has any problems, and if it did, we would then have to engage in the upstreaming process.

My opinion on this is that this is too simple of a feature to add value. If a team is sophisticated enough to implement dedicated server based multiplayer, they are simple enough to figure out how to call a function at regular intervals.

[DanielKinsman]

I don't want a _network_process, i just want rpc latency to not be tied to framerate (or at least tied to _physics_process rate instead of _process rate). As it stands in both godot 3 and 4 if you do something like this:

extends Spatial


func _ready():
    Engine.iterations_per_second = 120
    Engine.target_fps = 2

    var peer = NetworkedMultiplayerENet.new()
    if "--server" in OS.get_cmdline_args():
        peer.create_server(50000)
    else:
        peer.create_client("localhost", 50000)

    peer.transfer_mode = NetworkedMultiplayerPeer.TRANSFER_MODE_UNRELIABLE
    get_tree().network_peer = peer


func ping():
    rpc_id(1, "pong", OS.get_system_time_msecs())


remote func pong(time):
    var latency = OS.get_system_time_msecs() - time
    print("one way latency %s ms" % latency)


func _physics_process(delta):
    if get_tree().get_network_unique_id() != 1:
        ping()

and then run the server and client at some a random internal apart (bash sleep "3.$(($RANDOM % 10))s" ) it becomes clear that your rpc calls are fitting into a "window" of the _process loop on the server. You get a consistent number between 0-500ms whereas I would expect it to never go as high as 500ms on the loopback interface.

[Calinou]

@DanielKinsman I don't think there is a way around this. Setting target_fps to the same value as iterations_per_second should resolve this. Also, remember that Godot limits the number of simulated physics iterations to 8 per processed ("rendered") frame.

To reduce CPU usage, you can still reduce target_fps/iterations_per_second dynamically depending on server conditions (e.g. when there are no players on the server).

[DanielKinsman]

I'm not actually running at 2fps, it was just to highlight the issue by taking it to the extreme. In my testing I am running a 100Hz monitor and a 60Hz physics tick. Due to timing of those windows on average it means that sending rpcs from client _physics_process to server _physics_process and back again sees a round trip time of 3 physics ticks (50 ms) on the same computer via loopback. Turn off vsync (my scene is just a few cubes right now) and it drops to 1 tick.

In reality the multiple game clients and non-dedicated servers will all have their own weird and wonderful and varying refresh rates and gpu capabilities and setting target fps or tinkering with iterations per second is not really an option.

In my very naive thinking I am imagining that there is an rpc "message pump" somewhere in the engine code, and would like an option to have that pump run in step with physics process instead of process, or to have it disengaged entirely from either and (and let the user deal with their own thread safety issues).

[Calinou]

In my very naive thinking I am imagining that there is an rpc "message pump" somewhere in the engine code, and would like an option to have that pump run in step with physics process instead of process

Running physics on a separate thread might achieve this – there's an option for this in the project settings, but it's not 100% reliable. See also #1288.

Due to timing of those windows on average it means that sending rpcs from client _physics_process to server _physics_process and back again sees a round trip time of 3 physics ticks (50 ms) on the same computer via loopback. Turn off vsync (my scene is just a few cubes right now) and it drops to 1 tick.

For comparison, what round trip time do you get if you set Engine.target_fps to 60 or 120? Alternatively, you can set your monitor to 60 Hz and enable V-Sync for the 60 FPS test.

[DanielKinsman]

It is basically the same at 60Hz. I made a quick test project for comparing syncing physics via rpc or udp at https://github.com/DanielKinsman/godot_latency_test

[Faless]

I don't want a _network_process, i just want rpc latency to not be tied to framerate (or at least tied to _physics_process rate instead of _process rate).

You can already disable automatic polling and poll the multiplayer API manually in _physics_process.

Doing that in a separate thread (completely unrelated to engine iterations) is much more complex, and quite a corner case because you almost always want your network code (RPCs/replication) to happen in sync with either the physics or frame state.

For these reasons, by default the connection is polled during process.
That means that, even with no network latency, you have an average 8 ms delay at 60 fps (16ms round-trip) (you can try out https://github.com/Faless/gd-pinger).
I don't think there's much we can do here but I'm open to suggestions.
This is also why, in many games, only bots pings 0, a player connected to the LAN still tend to ping at least 6/7, and while some games do report lower pings, I tend to believe that when the server is implemented using the same engine as the client in most cases they are "cheating", i.e. subtracting the process delta frame from the the reported latency.

In this sense a _network_process could potentially help in those cases where you want to force both a low graphics and physics steps, but still retain a higher main iteration step, but I'm not really sure this is indeed a very common case (it won't magically create extra processing power when you experience FPS drops).

As explained above by others, achieving a lower network tick is already possible via a Timer and manual polling.

That said, I believe the current implementation of target_fps (OS::add_frame_delay) and platforms vsync add forced delay to the main iteration and that physics frames are in fact called in rapid succession with slices of the iteration delta (so that _physics_process actual execution time is always dependent to target_fps and/or vsynced FPS).

[DanielKinsman]

Thanks, I wasn't aware of disabling automatic polling and that is exactly what I needed! When enabling for both client and server it gets down to 1 frame. There still seems to be some interaction with frame rate but it is acceptable, especially given how much easier it is to use the inbuilt high level networking.

physics.mp4

[WantToSignUp]

_physics_process sadly tied to frame rate(input processing too), its behaves more like substepping than something being called in regular intervals. I wanted to used it to reduce/untie input and network latency from rendering, but sadly its not designed that way.

If you have 10 FPS, and 100 physics rate, the engine will run the physics in a loop 10 times immediately after the rendering without any delay, then idle wait 10 ms until the next frame can be rendered. The physics frames will be spaced out with whatever time it took to calculate physics, which can be close to 0 ms.

It would be nice to have a way to enable evenly spaced out physics steps instead of the current way. Or even have some kind of busy waiting timer(current timer is also tied to the framerate/gametime if the physics gets ahead more than 8 frames) or a constantly running node, and let us manually call physics process + network polling from that.

[Calinou]

_physics_process sadly tied to frame rate(input processing too), its behaves more like substepping than something being called in regular intervals. I wanted to used it to reduce/untie input and network latency from rendering, but sadly its not designed that way.

For input processing, this is being tracked in #1288.

Running network processing at a higher rate than rendering is interesting, as it can allow for significantly lower ping on 60 Hz monitors with V-Sync enabled. That said, disabling V-Sync and targeting higher framerates will give you similar benefits, on top of also reducing input lag (at least until #1288 is implemented). This is what most competitive players do anyway 🙂

Or even have some kind of busy waiting timer(current timer is also tied to the framerate/gametime if the physics gets ahead more than 8 frames) or a constantly running node, and let us manually call physics process + network polling from that.

See #1893. #2821 may also be handy here, as it'll let you simulate as many physics ticks per frame as you'd like.

[Calinou]

Closing, as I no longer think the original proposal is so useful now that we have MultiplayerSynchronizer. Also, the workaround is only 2 lines of code.