scheduler: fix reconnecting allocations getting rescheduled

[mismithhisler]

In scenarios where a disconnected client reconnects with failed replacement allocs, the reconcile strategy for picking the correct allocation was not working. These changes allow evaluation of multiple replacement allocs, and the stopping of allocs that are in a client terminal status.

[tgross]

This is looking great @mismithhisler!

I ran a test coverage report via go test -v -count=1 -cover -coverprofile=$HOME/cover.out ./scheduler && go tool cover -html=$HOME/cover.out and found that we're missing coverage in two places in this method. Normally we're not sticklers for test coverage but this is clearly a very tricky area of the code. The two areas are:

• The block starting if stopReconnecting (ref)
• The block where keepAlloc == replacementAlloc. (ref).

Let's try to add a test that covers these two areas, to make sure we're not missing anything. And don't forget to run make cl to add a change log entry!

[tgross]

LGTM!