Use docker-compose to run the functional tests

[KJTsanaktsidis]

CC @bai @dnwe - I took a stab at changing how the functional tests run as we discussed in #1661. Let me know what you think!

This PR makes a few changes:

• Puts the functional tests behind a separate build tag. Running go test runs the unit tests, and running go test -tags=integration will run all tests, including the functional tests
• Replaces the vagrant test setup mechanism with a docker-compose file.
• The functional tests will automatically start/stop the docker-compose test environment if TOXIPROXY_ADDR isn't set
• Changed the github actions config to use the docker-compose set up as well.

Open questions:

• This is using the confluent platform docker images (confluentinc/cp-{kafka,zookeeper}), which have a different versioning scheme to the upstream apache kafka project. This leads to some pretty ugly mapping code to deduce the kafka version from the confluent platform version, and vice versa. Unfortunately this seems to be the "msot official" kafka image I could find. Open to other suggestions on what we should use though.
• This changed the API for test cases to interact with the test environment a bit (access to the toxiproxies, kafka address, etc is done via the FunctionalTestEnv now).
• I did experiment with just using the Docker Engine API directly for spinning up the containers instead of using docker-compose. It lead to a lot of extra code though so I think compose was ultimately a better solution.
• Maybe it doesn't matter, but we could run docker-compose up in the "install dependancies" part of the github action workflow, instead of relying on it to be automatically invoked as part of "run test suite"

[dnwe]

@KJTsanaktsidis this looks good

RE: the docker images and version mapping, I wonder if we should just use a Java base image for the containers and volume mount in the extracted Kafka tgz and the server.properties we were generating? I remember we had some custom (short) timeouts etc. to make the toxiproxy tests complete more quickly etc. so it might be easier to manage our own Kafka within the container

[KJTsanaktsidis]

I remember we had some custom (short) timeouts etc. to make the toxiproxy tests complete more quickly

Actually I replicated those configs in the compose file - the confluent kafka image transforms environment variables of the form KAFKA_* to server properties.

use a Java base image for the containers and volume mount in the extracted Kafka tgz

This could work but makes it much more difficult to run the test environment by hand (to persist it across runs when iterating on them for example). We could however build our own Kafka images from a Dockerfile - I think docker-compose might even be smart enough to build them once the first time someone writes a functional test and use the same image for subsequent runs.

[KJTsanaktsidis]

@dnwe @d1egoaz @bai just wondering if there was still an appetite to have the functional tests be run like this?

[bai]

I'll let @d1egoaz re-review this one but LGTM.

[KJTsanaktsidis]

sorry, I just looked at this a second time since I was on the page and noticed there was a spot where I'd hardcoded kafka 2.5.0 for debugging purposes 🤦 Fixed and pushed.