Spring testcontainer test hangs when trying to simulate no connection to Mongodb

[FlorinAlexandru]

I'm trying to simulate that the Mongo database is down when the query is run. I'm using testcontainers with Toxiproxy and Mongo.

In the below test class, test called withLatency work as expected but test withoutConnection is hanging.

Code:

@SpringBootApplication
public class ToxiproxyIssueApplication {
    public static void main(String[] args) {
        SpringApplication.run(ToxiproxyIssueApplication.class, args);
    }
}

@Document
public record Person(String name, int age) {
}

@Repository
public interface PersonRepository extends MongoRepository<Person, String> {
}

The properties file contains:

spring.data.mongodb.host=localhost
spring.data.mongodb.port=27017

The test:

@Testcontainers
@SpringBootTest(properties = {"spring.main.allow-bean-definition-overriding=true"})
class ToxiproxyIssueApplicationTests {

    @Autowired
    private PersonRepository personRepository;

    private static final Network network = Network.newNetwork();

    private static final DockerImageName dockerImageName = DockerImageName.parse("mongo:5.0.5").asCompatibleSubstituteFor("mongo");

    @Container
    static final MongoDBContainer mongoDbContainer = new MongoDBContainer(dockerImageName)
            .withNetwork(network)
            .withNetworkAliases("mongo");
//          .withReuse(true);

    @Container
    private static final ToxiproxyContainer toxiproxy = new ToxiproxyContainer("ghcr.io/shopify/toxiproxy:2.6.0")
            .withNetwork(network);

    private static Proxy mongoProxy;

    @DynamicPropertySource
    static void mongoProperties(DynamicPropertyRegistry registry) throws IOException {
        ToxiproxyClient toxiproxyClient = new ToxiproxyClient(toxiproxy.getHost(), toxiproxy.getControlPort());
        mongoProxy = toxiproxyClient.createProxy("mongodb", "0.0.0.0:8666", "mongo:27017");

        int port = mongoDbContainer.getMappedPort(27017);
        registry.add("spring.data.mongodb.host", toxiproxy::getHost);
        registry.add("spring.data.mongodb.port", () -> toxiproxy.getMappedPort(8666));
    }


    @Test
    void withLatency() throws IOException {
        mongoProxy.toxics().latency("mongo-latency", ToxicDirection.DOWNSTREAM, 2600).setJitter(100);
        var result = personRepository.findAll();
        assertThat(result).isEmpty();
    }

    @Test
    void withoutConnection() throws IOException {
        mongoProxy.toxics().bandwidth("no-connection-downstream", ToxicDirection.DOWNSTREAM, 0);
        mongoProxy.toxics().bandwidth("no-connection-upstream", ToxicDirection.UPSTREAM, 0);
        var result = personRepository.findAll();
        assertThat(result).isEmpty();

        mongoProxy.toxics().get("no-connection-downstream").remove();
        mongoProxy.toxics().get("no-connection-upstream").remove();
    }
}

When the test is hanging I see in the logs a timeout from the driver:

com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message
Is there something wrong in my config or is this a testcontainer bug?

[eddumelendez]

Well, you block the network and then call to findAll without handling failures. For that reason, it is hanging out. When you use latency, you know that after certain amount of time it will be gone but not when cutting down the connection. Look at the following example. In my example, I'm using reactor library.

[FlorinAlexandru]

I recognize the code from this example https://www.atomicjar.com/2023/03/developing-resilient-applications-with-toxiproxy-and-testcontainers/. I used it as reference.

I did not understood the "without handling failures" part. I expected that after I sent the query, an timeout exception should be thrown (based on the logs I think this is happening) and then the test should fail. Is there an example of handling failures but using non-reactive code?

[eddumelendez]

I'm not sure how the mongodb client library is handling it and that's what needs to be revisit. Looks like logs the time out but still retrying? What you need to do is once test that the connection cut down works and the client fails then capture the failure and restore the connection again. This is related to the client library itself.

[FlorinAlexandru]

Looks like logs the time out but still retrying?

Yes, based on the debug logs of mongo driver this is what is doing.

Yes, I agree that I need to revisit both spring data implementation and also the mongo driver. I suspect that this behaviour is from the java mongo driver.

Other idee is to set timeout to 5s and latency to 6s. This way I have my timeout and also the restoration of the network.

[FlorinAlexandru]

Based on the Toxiproxy documentation, there is the down toxic (https://github.com/Shopify/toxiproxy#down): by disabling the proxy no traffic will be routed.

In java Toxiproxy client library there are some methods to enable and disable the proxy: https://github.com/trekawek/toxiproxy-java/blob/7c5ffc5d7aa4780bd2ea73ad945f0227793a7e3d/src/main/java/eu/rekawek/toxiproxy/Proxy.java#L70-L76

Thus, the mentioned failed test will be rewritten as:

	@Test
	void withoutConnection() throws IOException {
		mongoProxy.disable();
		assertThatThrownBy(()->personRepository.findAll()).isInstanceOf(Exception.class);
		mongoProxy.enable();
		var result = personRepository.findAll();
		assertThat(result).isEmpty();
	}