Panic from libp2p-upnp after BN re-connects to the network

[jimmygchen]

Description

I am getting the following panic after my BN loses connection and re-connects:

message: mapping should exist, location: /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-upnp-0.2.1/src/behaviour.rs:450:38, module: lighthouse:534

Command:

lighthouse bn --listen-address 0.0.0.0 --port 9500 --disable-upnp --http --checkpoint-sync-url 'https://mainnet.checkpoint.sigp.io' --execution-endpoint <xxx> --execution-jwt ./jwt.hex --metrics

Note: The panic comes from the libp2p-upnp crate, so I've tried disabling upnp (--disable-upnp) but it didn't help. May be related to #5443 - upnp doesn't actually get disabled.

Version

• unstable (v5.1.0+)
• MacOS 14.2.1

Steps to reproduce

1. Start Lighthouse BN with checkpoint sync
2. After the node completes head sync, disconnect the internet (e.g. turn off Wifi)
3. Turn Wifi back on after a slot (not sure if the wait time matters, but I was able to re-produce consistently with my setup
4. Lighthouse crashes with this panic.

Additional Info

I'm seeing this log entry on startup (without --disable-upnp):

Mar 20 05:01:13.104 INFO Could not UPnP map Discovery port       error: Could not UPnP map port: 9500 on the gateway

Note that this log entry isn't present when I run it with --disable-upnp, but Lighthouse still panics, so I'm not sure if the above is relevant.

Full Backtrace:

Mar 20 05:06:44.805 CRIT Task panic. This is a bug!              advice: Please check above for a backtrace and notify the developers, backtrace:    0: std::backtrace_rs::backtrace::libunwind::trace
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1: std::backtrace_rs::backtrace::trace_unsynchronized
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2: std::backtrace::Backtrace::create
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/backtrace.rs:331:13
   3: lighthouse::run::{{closure}}
   4: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2029:9
   5: std::panicking::rust_panic_with_hook
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:783:13
   6: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
   7: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
   8: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   9: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
  10: core::panicking::panic_display
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:196:5
  11: core::panicking::panic_str
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:171:5
  12: core::option::expect_failed
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/option.rs:1980:5
  13: <libp2p_upnp::behaviour::Behaviour as libp2p_swarm::behaviour::NetworkBehaviour>::poll
  14: <lighthouse_network::service::behaviour::Behaviour<AppReqId,TSpec> as libp2p_swarm::behaviour::NetworkBehaviour>::poll
  15: libp2p_swarm::Swarm<TBehaviour>::poll_next_event
  16: lighthouse_network::service::Network<AppReqId,TSpec>::poll_network
  17: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  18: futures_util::future::future::FutureExt::poll_unpin
  19: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  20: task_executor::TaskExecutor::spawn_handle::{{closure}}
  21: tokio::runtime::task::core::Core<T,S>::poll
  22: tokio::runtime::task::harness::Harness<T,S>::poll
  23: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  24: tokio::runtime::scheduler::multi_thread::worker::Context::run
  25: tokio::runtime::context::runtime::enter_runtime
  26: tokio::runtime::scheduler::multi_thread::worker::run
  27: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  28: tokio::runtime::task::core::Core<T,S>::poll
  29: tokio::runtime::task::raw::poll
  30: tokio::runtime::blocking::pool::Inner::run
  31: std::sys_common::backtrace::__rust_begin_short_backtrace
  32: core::ops::function::FnOnce::call_once{{vtable.shim}}
  33: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  34: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  35: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  36: __pthread_joiner_wake
, message: mapping should exist, location: /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-upnp-0.2.1/src/behaviour.rs:450:38

[AgeManning]

This has been resolved in #5498 and #5501