Replication pool and cold / warm setup

[krmeljalen]

Idea is to fix single point of failure of memcache. For now we have two clusters, each with one node on our production. From time to time, some maintenance on a either memcache server is needed, so we would like to setup a config that:

1. replicates both clusters (A cluster has same data as B)
2. warmup of either cluster if one goes down (If A cluster goes down, we would need B to warmup data to A when A comes back and the other way around if B goes down, A should warmup B.)

I'm sure such redundant setup should be pretty useful for anyone using memcache and hopes to achieve redundancy, that memcache by default does not have.

For now we have came up with this config:

{
    "pools": {
        "cold": {
            "servers": [ "mc-db1.alen.mydomain.com:11211", "mc-db2.alen.mydomain.com:11211" ]
        },
        "warm": {
            "servers": [ "mc-db1.alen.mydomain.com:11211", "mc-db2.alen.mydomain.com:11211" ] 
        }
    },
    "route":{
        "type": "OperationSelectorRoute",
        "operation_policies": {
            "get": {
                "type": "WarmUpRoute",
                "cold": {
                    "type": "OperationSelectorRoute",
                    "operation_policies": {
                        "get": "LatestRoute|Pool|cold",
                        "set": "AllSyncRoute|Pool|cold"
                    },
                    "default_policy": "AllSyncRoute|Pool|cold"
                },
                "warm": "PoolRoute|warm"
            },
            "set": "AllSyncRoute|Pool|cold",
            "add": "AllSyncRoute|Pool|cold",
            "delete": "AllSyncRoute|Pool|cold"
        }
    }
}

Note: same server names used in cold and warm are intentionally, so that it replicates to both, no matter which one goes down.

This setup should work, but unfortunately, it doesn't work as expected.

As we shutdown A server, GET's for our keys continue to work. Then when A comes up, we keep running same GET's in order to warmup A that was down. After a while we decide to shutdown B since A should have the keys already warmed up in this time. When B shuts down, keys start failing.

Any help on this setup would be greatly appreciated. I'm sure it would help others too.

[krmeljalen]

I1231 12:31:28.847344 12391 main.cpp:338] /usr/bin/mcrouter --config-str={"pools":{"cold":{"servers":["mc-db1.alen.mydomain.com:11211","mc-db2.alen.mydomain.com:11211"]},"warm":{"servers":["mc-db1.alen.mydomain.com:11211","mc-db2.alen.mydomain.com:11211"]}},"route":{"type":"OperationSelectorRoute","operation_policies":{"get":{"type":"WarmUpRoute","cold":{"type":"OperationSelectorRoute","operation_policies":{"get":"LatestRoute|Pool|cold","set":"AllSyncRoute|Pool|cold"},"default_policy":"AllSyncRoute|Pool|cold"},"warm":"PoolRoute|warm"},"set":"AllSyncRoute|Pool|cold","add":"AllSyncRoute|Pool|cold","delete":"AllSyncRoute|Pool|cold"}}} -p 11211
I1231 12:31:28.847414 12391 main.cpp:472] 38.0.0-master mcrouter startup (12391)
I1231 12:31:28.847430 12391 main.cpp:364] Starting Memcache router
I1231 12:31:28.847455 12391 Server-inl.h:128] Spawning AsyncMcServer
I1231 12:31:28.847678 12391 FunctionScheduler.cpp:405] Starting FunctionScheduler with 0 functions.
I1231 12:31:28.849020 12391 CarbonRouterInstance-inl.h:486] started reconfiguring
I1231 12:31:28.849370 12391 CarbonRouterInstance-inl.h:503] reconfigured 1 proxies with 2 pools, 4 clients 46b61936a9a2017e6bcafd7ca6b41588)
I1231 12:35:53.679001 12392 ProxyDestination.cpp:524] mc-db2.alen.mydomain.com:11211 marked hard TKO. Total hard TKOs: 1; soft TKOs: 0. Reply: mc_res_connect_error
I1231 12:36:05.213534 12392 ProxyDestination.cpp:524] mc-db2.alen.mydomain.com:11211 unmarked TKO. Total hard TKOs: 0; soft TKOs: 0. Reply: mc_res_ok

This is the log we get.