[fix][tests] Fix Mockito mocks memory leak

[lhotari]

Fixes #17714

Motivation

There's a memory leak when using Mockito mocks since all invocations get recorded.

Modifications

Add multiple solutions for making sure that mocks are cleaned up properly.
Modify MockedPulsarServiceBaseTest to clean up PulsarService and PulsarAdmin mocks since the detected memory leaks are caused by these mocks.

Documentation

• doc-required
  (Your PR needs to update docs and you will update later)

• doc-not-needed
  (Please explain why)

• doc
  (Your PR contains doc changes)

• doc-complete
  (Docs have been already added)

Matching PR in forked repository

PR in forked repository: lhotari#92

[tisonkun]

LGTM.

Since we already hard depend on org.mockito.internal.util.MockUtil, the comment above is not a blocker.

[tisonkun]

Perhaps we can report this OOM to the mockito community for a solution in idiom. Since they mark these methods as not recommended they could have some insights here :)

[lhotari]

Perhaps we can report this OOM to the mockito community for a solution in idiom. Since they mark these methods as not recommended they could have some insights here :)

There's nothing here that is caused by a Mockito bug.

[tisonkun]

There's nothing here that is caused by a Mockito bug.

I have some experience in asking upstream experts for best practices. But let's say I should volunteer to report the case instead of requesting others.

[lhotari]

There's nothing here that is caused by a Mockito bug.

I have some experience in asking upstream experts for best practices. But let's say I should volunteer to report the case instead of requesting others.

In this case, the reason for the memory leaks is the misuse of Mockito in the Pulsar internal test framework.
Mockito will record invocations by default. To clear the invocations, mocks should be resetted or invocations should be cleared if the instances are around so that it matters.
Unfinished verification or stubbing will also leak memory. Those are really issues in test code of Pulsar, but nobody has cared to fix the warning that are logged by the test interceptors that mitigate the bad practices by clearing the thread locals.

org.apache.pulsar.tests.MockitoThreadLocalStateCleaner will log warnings about "Invalid usage of Mockito detected on thread ..." to help find the locations of the misuse of Mockito in Pulsar tests.
Similarly it's now logging "Mock contains registered invocations that should be cleared". This is something that should be fixed in tests so that it doesn't leak.

I guess it would be helpful to have some support for checks like this directly in Mockito so that it would be possible to add such validation without using reflection.

@tisonkun Since you seem to care about this topic, it would be useful to fix issues in Pulsar test code one issue at a time.