Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Already shutting down, interrupt more to panic #25775

Closed
MareG76 opened this issue Sep 15, 2022 · 2 comments · Fixed by #25831
Closed

Already shutting down, interrupt more to panic #25775

MareG76 opened this issue Sep 15, 2022 · 2 comments · Fixed by #25831
Labels

Comments

@MareG76
Copy link

MareG76 commented Sep 15, 2022

Geth version: 1.10.25
OS & Version: Windows 10

Expected behaviour

Almost every version of get lately gives me this problem, after hitting ctrl+c it just won't stop, some times I wait half an hour and nothing

Actual behaviour

It gives me "Already shutting down, interrupt more to panic" and when i interrupt it more I get this:
https://pastebin.com/HpWBRxMp

Steps to reproduce the behaviour

Try to stop geth on Windows 10 with ctrl+c

@holiman
Copy link
Contributor

holiman commented Sep 16, 2022

The shutdown sequence has become stuck here:

goroutine 55628060 [semacquire, 5 minutes]:
sync.runtime_Semacquire(0x3?)
        runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc003207bc0?)
        sync/waitgroup.go:136 +0x52
github.com/ethereum/go-ethereum/eth.(*handler).Stop(0xc0003ea600)
        github.com/ethereum/go-ethereum/eth/handler.go:565 +0x89
github.com/ethereum/go-ethereum/eth.(*Ethereum).Stop(0xc0003ea500)
        github.com/ethereum/go-ethereum/eth/backend.go:548 +0x50
github.com/ethereum/go-ethereum/node.(*Node).stopServices(0xc0000f8460, {0xc0000b7690, 0x1, 0x28?})
        github.com/ethereum/go-ethereum/node/node.go:303 +0xbe
github.com/ethereum/go-ethereum/node.(*Node).Close(0xc0000f8460)
        github.com/ethereum/go-ethereum/node/node.go:223 +0x191
created by github.com/ethereum/go-ethereum/cmd/utils.StartNode.func1.1
        github.com/ethereum/go-ethereum/cmd/utils/cmd.go:92 +0x96

I.e :

        // Disconnect existing sessions.
        // This also closes the gate for any new registrations on the peer set.
        // sessions which are already established but not added to h.peers yet
        // will exit when they try to register.
        h.peers.close()
        h.peerWG.Wait() <-- HERE

The peerWG is added to once for runEthPeer and runSnapExtension. Here's an eth handler, stuck trying to deliver block headers:

goroutine 11159874 [select, 1276 minutes]:
github.com/ethereum/go-ethereum/eth/protocols/eth.(*Peer).dispatchResponse(0xc0a4863860, 0xc08cde7da0, 0xc0536ef678)
        github.com/ethereum/go-ethereum/eth/protocols/eth/dispatcher.go:172 +0x28f
github.com/ethereum/go-ethereum/eth/protocols/eth.handleBlockHeaders66({0x7ff77361d580?, 0xc0001dae40?}, {0x7ff7739b8800, 0xc08cde6240}, 0xc0c0aef5914f4320?)
        github.com/ethereum/go-ethereum/eth/protocols/eth/handlers.go:367 +0x178
github.com/ethereum/go-ethereum/eth/protocols/eth.handleMessage({0x7ff7739be900, 0xc0003ea600}, 0xc0a4863860)
        github.com/ethereum/go-ethereum/eth/protocols/eth/handler.go:230 +0x597
github.com/ethereum/go-ethereum/eth/protocols/eth.Handle({0x7ff7739be900, 0xc0003ea600}, 0xc0a4863860)
        github.com/ethereum/go-ethereum/eth/protocols/eth/handler.go:154 +0x3d
github.com/ethereum/go-ethereum/eth/protocols/eth.MakeProtocols.func1.1(0xc0a4863860?)
        github.com/ethereum/go-ethereum/eth/protocols/eth/handler.go:111 +0x27
github.com/ethereum/go-ethereum/eth.(*handler).runEthPeer(0xc0003ea600, 0xc0a4863860, 0xc262f38e40)
        github.com/ethereum/go-ethereum/eth/handler.go:475 +0x117a
github.com/ethereum/go-ethereum/eth.(*ethHandler).RunPeer(0x42?, 0xc26b531500?, 0x7ff7739b7090?)
        github.com/ethereum/go-ethereum/eth/handler_eth.go:41 +0x19
github.com/ethereum/go-ethereum/eth/protocols/eth.MakeProtocols.func1(0xc0fa3eaf00?, {0x7ff7739b7090, 0xc07419c0a0})
        github.com/ethereum/go-ethereum/eth/protocols/eth/handler.go:110 +0x122
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1()
        github.com/ethereum/go-ethereum/p2p/peer.go:415 +0x8c
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
        github.com/ethereum/go-ethereum/p2p/peer.go:413 +0xae

This is where it's been sitting for 21 hours:

// dispatchRequest fulfils a pending request and delivers it to the requested
// sink.
func (p *Peer) dispatchResponse(res *Response, metadata func() interface{}) error {
        resOp := &response{
                res:  res,
                fail: make(chan error),
        }
        res.recv = time.Now()
        res.Done = make(chan error)

        select {
        case p.resDispatch <- resOp:
                // Ensure the response is accepted by the dispatcher
                if err := <-resOp.fail; err != nil {
                        return nil
                }
                // Request was accepted, run any postprocessing step to generate metadata
                // on the receiver thread, not the sink thread
                if metadata != nil {
                        res.Meta = metadata()
                }
                // Deliver the filled out response and wait until it's handled. This
                // path is a bit funky as Go's select has no order, so if a response
                // arrives to an already cancelled request, there's a 50-50% changes
                // of picking on channel or the other. To avoid such cases delivering
                // the packet upstream, check for cancellation first and only after
                // block on delivery.
                select { 
                case <-res.Req.cancel:
                        return nil // Request cancelled, silently discard response
                default:
                        // Request not yet cancelled, attempt to deliver it, but do watch
                        // for fresh cancellations too
                        select { <--- STUCK ON THIS LINE
                        case res.Req.sink <- res:
                                return <-res.Done // Response delivered, return any errors
                        case <-res.Req.cancel:
                                return nil // Request cancelled, silently discard response
                        }
                }

@holiman
Copy link
Contributor

holiman commented Sep 16, 2022

I think this should fix it

diff --git a/eth/protocols/eth/dispatcher.go b/eth/protocols/eth/dispatcher.go
index 65a935d555..1486a70022 100644
--- a/eth/protocols/eth/dispatcher.go
+++ b/eth/protocols/eth/dispatcher.go
@@ -174,6 +174,8 @@ func (p *Peer) dispatchResponse(res *Response, metadata func() interface{}) erro
 				return <-res.Done // Response delivered, return any errors
 			case <-res.Req.cancel:
 				return nil // Request cancelled, silently discard response
+			case <-p.term:
+				return errDisconnected
 			}
 		}

That should fix the shutdown problem, but there's some other underlying error here which also needs to be fixed, namely -- why doesn't that request become cancelled earlier?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants