Experiment: looking for goroutine leaks triggered by tests

[k-wall]

Test failures on another PR caused me to notice some goroute leaks when running the tests. I fixed some by way of #2198. Three still remain which have a less obvious root cause:

To reproduce, with this PR do:

GOFLAGS="-tags=goleak" go test -c -o tests
for test in $(go test -list . | grep -E "^(Test|Example)"); do ./tests -test.run "^$test\$" &>/dev/null && echo -n "." || echo -e "\n$test failed"; done

gives:

........................................................
TestRefreshMetaDataWithDifferentController failed
................................
TestAsyncProducerIdempotentRetryCheckBatch failed
..............................................................
TestClientRefreshBrokers failed
................................................

Running the test individually gets the details:

./tests -test.run ^TestAsyncProducerIdempotentRetryCheckBatch$

kwall@Oslo sarama % ./tests -test.run ^TestRefreshMetaDataWithDifferentController$
PASS
goleak: Errors on successful test run: found unexpected goroutines:
[Goroutine 11 in state chan receive, with github.com/Shopify/sarama.(*Broker).responseReceiver on top of the stack:
goroutine 11 [chan receive]:
github.com/Shopify/sarama.(*Broker).responseReceiver(0xc0000b4e00)
	/Users/kwall/src/sarama/broker.go:1047 +0x74
github.com/Shopify/sarama.withRecover(0x0)
	/Users/kwall/src/sarama/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*Broker).Open.func1
	/Users/kwall/src/sarama/broker.go:245 +0xb65
]
kwall@Oslo sarama % ./tests -test.run ^TestAsyncProducerIdempotentRetryCheckBatch$
PASS
goleak: Errors on successful test run: found unexpected goroutines:
[Goroutine 38 in state select, with github.com/Shopify/sarama.(*brokerProducer).run on top of the stack:
goroutine 38 [select]:
github.com/Shopify/sarama.(*brokerProducer).run(0xc0002824e0)
	/Users/kwall/src/sarama/async_producer.go:807 +0x19a
github.com/Shopify/sarama.withRecover(0x0)
	/Users/kwall/src/sarama/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer
	/Users/kwall/src/sarama/async_producer.go:692 +0x25b

 Goroutine 39 in state chan receive, with github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer.func1 on top of the stack:
goroutine 39 [chan receive]:
github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer.func1()
	/Users/kwall/src/sarama/async_producer.go:699 +0x65
github.com/Shopify/sarama.withRecover(0x0)
	/Users/kwall/src/sarama/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer
	/Users/kwall/src/sarama/async_producer.go:695 +0x32f

 Goroutine 40 in state chan receive, with github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer.func2 on top of the stack:
goroutine 40 [chan receive]:
github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer.func2()
	/Users/kwall/src/sarama/async_producer.go:745 +0xcc
github.com/Shopify/sarama.withRecover(0x0)
	/Users/kwall/src/sarama/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer
	/Users/kwall/src/sarama/async_producer.go:741 +0x3c7

 Goroutine 33 in state select, with github.com/Shopify/sarama.(*brokerProducer).run on top of the stack:
goroutine 33 [select]:
github.com/Shopify/sarama.(*brokerProducer).run(0xc000112e40)
	/Users/kwall/src/sarama/async_producer.go:807 +0x19a
github.com/Shopify/sarama.withRecover(0x0)
	/Users/kwall/src/sarama/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer
	/Users/kwall/src/sarama/async_producer.go:692 +0x25b

 Goroutine 98 in state chan receive, with github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer.func1 on top of the stack:
goroutine 98 [chan receive]:
github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer.func1()
	/Users/kwall/src/sarama/async_producer.go:699 +0x65
github.com/Shopify/sarama.withRecover(0x0)
	/Users/kwall/src/sarama/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer
	/Users/kwall/src/sarama/async_producer.go:695 +0x32f

 Goroutine 99 in state chan receive, with github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer.func2 on top of the stack:
goroutine 99 [chan receive]:
github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer.func2()
	/Users/kwall/src/sarama/async_producer.go:745 +0xcc
github.com/Shopify/sarama.withRecover(0x0)
	/Users/kwall/src/sarama/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer
	/Users/kwall/src/sarama/async_producer.go:741 +0x3c7
]
kwall@Oslo sarama % ./tests -test.run ^TestClientRefreshBrokers$
PASS
goleak: Errors on successful test run: found unexpected goroutines:
[Goroutine 8 in state chan receive, with github.com/Shopify/sarama.(*Broker).responseReceiver on top of the stack:
goroutine 8 [chan receive]:
github.com/Shopify/sarama.(*Broker).responseReceiver(0xc00017dc00)
	/Users/kwall/src/sarama/broker.go:1047 +0x74
github.com/Shopify/sarama.withRecover(0x0)
	/Users/kwall/src/sarama/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*Broker).Open.func1
	/Users/kwall/src/sarama/broker.go:245 +0xb65

 Goroutine 27 in state select, with github.com/Shopify/sarama.(*client).backgroundMetadataUpdater on top of the stack:
goroutine 27 [select]:
github.com/Shopify/sarama.(*client).backgroundMetadataUpdater(0xc0001603f0)
	/Users/kwall/src/sarama/client.go:823 +0x12e
github.com/Shopify/sarama.withRecover(0x0)
	/Users/kwall/src/sarama/utils.go:43 +0x3e
created by github.com/Shopify/sarama.NewClient
	/Users/kwall/src/sarama/client.go:182 +0x3db
]

[k-wall]

@dnwe I raised PRs for leaks associated with TestClientRefreshBrokers and TestRefreshMetaDataWithDifferentController (both issues with the production code, I believe). I don't understand enough about the producer close implementation at the moment to make inroads into leaks appearing from TestAsyncProducerIdempotentRetryCheckBatch

[k-wall]

I spent a bit more time reasoning about the leak shown by TestAsyncProducerIdempotentRetryCheckBatch. I believe the defect fix is #2208