-
Notifications
You must be signed in to change notification settings - Fork 171
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve pipeline backoff #1097
Improve pipeline backoff #1097
Conversation
@yellowhatter can you please help in reviewing this PR? |
@Mallets looks nice! I've been in parallel working on a little bit different design. Our queue implementation uses
My idea is that the producer thread should take care about always having "one batch behind": after each serialization check atomic count of batches in |
* Add NOTE for LowLatency transport. (#1088) Signed-off-by: ChenYing Kuo <[email protected]> * fix: Improve debug messages in `zenoh-transport` (#1090) * fix: Improve debug messages for failing RX/TX tasks * fix: Improve debug message for `accept_link` timeout * chore: Fix `clippy::redundant_pattern_matching` error * Improve pipeline backoff (#1097) * Yield task for backoff * Improve comments and error handling in backoff * Simplify pipeline pull * Consider backoff configuration * Add typos check to CI (#1065) * Fix typos * Add typos check to CI * Start link tx_task before notifying router (#1098) * Fix typos (#1110) * bump quinn & rustls (#1086) * bump quinn & rustls * fix ci windows check * add comments * Fix interface name scanning when listening on IP unspecified for TCP/TLS/QUIC/WS (#1123) Co-authored-by: Julien Enoch <[email protected]> * Enable releasing from any branch (#1136) * Fix cargo clippy (#1145) * Release tables locks before propagating subscribers and queryables declarations to void dead locks (#1150) * Send simple sub and qabl declarations using a given function * Send simple sub and qabl declarations after releasing tables lock * Send simple sub and qabl declarations after releasing tables lock (missing places) * Update async-io * Update base64 dependency * Update event-listener dependency * Update jsonschema dependency * Update keyed-set dependency * Update console-subscriber dependency * Update pnet dependency * Update rcgen dependency * Update tokio-tungstenite dependency * Update thread-priority dependency * Fix typos * Fix typos * Add Unicode-3.0 to allowed licenses --------- Signed-off-by: ChenYing Kuo <[email protected]> Co-authored-by: ChenYing Kuo (CY) <[email protected]> Co-authored-by: Mahmoud Mazouz <[email protected]> Co-authored-by: Luca Cominardi <[email protected]> Co-authored-by: Tavo Annus <[email protected]> Co-authored-by: JLer <[email protected]> Co-authored-by: Julien Enoch <[email protected]>
* Add NOTE for LowLatency transport. (#1088) Signed-off-by: ChenYing Kuo <[email protected]> * fix: Improve debug messages in `zenoh-transport` (#1090) * fix: Improve debug messages for failing RX/TX tasks * fix: Improve debug message for `accept_link` timeout * chore: Fix `clippy::redundant_pattern_matching` error * Improve pipeline backoff (#1097) * Yield task for backoff * Improve comments and error handling in backoff * Simplify pipeline pull * Consider backoff configuration * Add typos check to CI (#1065) * Fix typos * Add typos check to CI * Start link tx_task before notifying router (#1098) * Fix typos (#1110) * bump quinn & rustls (#1086) * bump quinn & rustls * fix ci windows check * add comments * Fix interface name scanning when listening on IP unspecified for TCP/TLS/QUIC/WS (#1123) Co-authored-by: Julien Enoch <[email protected]> * Enable releasing from any branch (#1136) * Fix cargo clippy (#1145) * Release tables locks before propagating subscribers and queryables declarations to void dead locks (#1150) * Send simple sub and qabl declarations using a given function * Send simple sub and qabl declarations after releasing tables lock * Send simple sub and qabl declarations after releasing tables lock (missing places) * feat: make `TerminatableTask` terminate itself when dropped (#1151) * Fix bug in keyexpr::includes leading to call get_unchecked on empty array UB (#1208) * REST plugin uses unbounded flume channels for queries (#1213) * fix: typo in selector.rs (#1228) * fix: zenohd --cfg (#1263) * fix: zenohd --cfg * ci: trigger * Update zenohd/src/main.rs --------- Co-authored-by: Luca Cominardi <[email protected]> * Fix failover brokering bug reacting to linkstate changes (#1272) * Change missleading log * Fix failover brokering bug reacting to linkstate changes * Retrigger CI --------- Co-authored-by: Luca Cominardi <[email protected]> * Code format * Fix clippy warnings * Code format * Fix Clippy errors from Rust 1.80 (#1273) * Allow unexpected `doc_auto_cfg` flag * Keep never-constructed logger interceptor * Ignore interior mutability of `Resource` * Fix typo * Resolve `clippy::doc-lazy-continuation` errors * Upgrade `[email protected]` to `[email protected]` See time-rs/time#693 * Update Cargo.toml (#1277) Updated description to be aligned with what we use everywhere else * Merge ci.yaml --------- Signed-off-by: ChenYing Kuo <[email protected]> Co-authored-by: ChenYing Kuo (CY) <[email protected]> Co-authored-by: Mahmoud Mazouz <[email protected]> Co-authored-by: Tavo Annus <[email protected]> Co-authored-by: JLer <[email protected]> Co-authored-by: Julien Enoch <[email protected]> Co-authored-by: OlivierHecart <[email protected]> Co-authored-by: Yuyuan Yuan <[email protected]> Co-authored-by: Diogo Matsubara <[email protected]> Co-authored-by: OlivierHecart <[email protected]> Co-authored-by: kydos <[email protected]>
* Add NOTE for LowLatency transport. (#1088) Signed-off-by: ChenYing Kuo <[email protected]> * fix: Improve debug messages in `zenoh-transport` (#1090) * fix: Improve debug messages for failing RX/TX tasks * fix: Improve debug message for `accept_link` timeout * chore: Fix `clippy::redundant_pattern_matching` error * Improve pipeline backoff (#1097) * Yield task for backoff * Improve comments and error handling in backoff * Simplify pipeline pull * Consider backoff configuration * Add typos check to CI (#1065) * Fix typos * Add typos check to CI * Start link tx_task before notifying router (#1098) * Fix typos (#1110) * bump quinn & rustls (#1086) * bump quinn & rustls * fix ci windows check * add comments * Fix interface name scanning when listening on IP unspecified for TCP/TLS/QUIC/WS (#1123) Co-authored-by: Julien Enoch <[email protected]> * Enable releasing from any branch (#1136) * Fix cargo clippy (#1145) * Release tables locks before propagating subscribers and queryables declarations to void dead locks (#1150) * Send simple sub and qabl declarations using a given function * Send simple sub and qabl declarations after releasing tables lock * Send simple sub and qabl declarations after releasing tables lock (missing places) * feat: make `TerminatableTask` terminate itself when dropped (#1151) * Fix bug in keyexpr::includes leading to call get_unchecked on empty array UB (#1208) * REST plugin uses unbounded flume channels for queries (#1213) * fix: typo in selector.rs (#1228) * fix: zenohd --cfg (#1263) * fix: zenohd --cfg * ci: trigger * Update zenohd/src/main.rs --------- Co-authored-by: Luca Cominardi <[email protected]> * Fix failover brokering bug reacting to linkstate changes (#1272) * Change missleading log * Fix failover brokering bug reacting to linkstate changes * Retrigger CI --------- Co-authored-by: Luca Cominardi <[email protected]> * Code format * Fix clippy warnings * Code format * Fix Clippy errors from Rust 1.80 (#1273) * Allow unexpected `doc_auto_cfg` flag * Keep never-constructed logger interceptor * Ignore interior mutability of `Resource` * Fix typo * Resolve `clippy::doc-lazy-continuation` errors * Upgrade `[email protected]` to `[email protected]` See time-rs/time#693 * Update Cargo.toml (#1277) Updated description to be aligned with what we use everywhere else * fix: typos (#1297) * Replace trees computation tasks with a worker (#1303) * Replace trees computation tasks with a worker * Address review comments * Remove review comments * zenohd-default config error #1292 (#1298) * Zenohd panic when tring load file When zenohd trying load file, if it have a problem it crash cause another treat was "unwrap", and it return to a type config. So, it crash and cause painic. * zenohd default config error #1292 When tring load config file defined by -c option. With haver any problema "unwrap" has been to Config type. I treat it return a Default Config whe it happen * If file fail when try load configs If file fail when try load configs * Update main.rs * Resolve typos at comment Resolve typos at comment * fix: typos (#1297) * zenohd-default config error #1292 (#1298) * Zenohd panic when tring load file When zenohd trying load file, if it have a problem it crash cause another treat was "unwrap", and it return to a type config. So, it crash and cause painic. * zenohd default config error #1292 When tring load config file defined by -c option. With haver any problema "unwrap" has been to Config type. I treat it return a Default Config whe it happen * If file fail when try load configs If file fail when try load configs * Update main.rs * Resolve typos at comment Resolve typos at comment * Replace trees computation tasks with a worker (#1303) * Replace trees computation tasks with a worker * Address review comments * Remove review comments * revering fix #1298 --------- Signed-off-by: ChenYing Kuo <[email protected]> Co-authored-by: ChenYing Kuo (CY) <[email protected]> Co-authored-by: Mahmoud Mazouz <[email protected]> Co-authored-by: Luca Cominardi <[email protected]> Co-authored-by: Tavo Annus <[email protected]> Co-authored-by: JLer <[email protected]> Co-authored-by: Julien Enoch <[email protected]> Co-authored-by: OlivierHecart <[email protected]> Co-authored-by: Yuyuan Yuan <[email protected]> Co-authored-by: Diogo Matsubara <[email protected]> Co-authored-by: OlivierHecart <[email protected]> Co-authored-by: kydos <[email protected]> Co-authored-by: brianPA <[email protected]> Co-authored-by: Tiago Neves <[email protected]>
This PR improves the pipeline backoff implementation and reduces the possibility of spinning tasks.
Without this patch, it may happen before that in high-load scenario we could observe the following logs:
With this patch, in my tests I didn't observe those logs anymore.
What I expect is that this patch heavily mitigates the problem although it might not solve it in 100% of the cases.