Athens fails to start when deployed on Kubernetes using custom specifications

[amarruedo]

Describe the bug

Trying to deploy Athens on a Kubernetes cluster using an Athens deployment and service specification I've experienced the following issue:

If I first deploy the service and then the deployment, Athens fails to start. If I change the order and deploy first the deployment and then the service, then it works.

Error Message

$ kubectl get pods -w
NAME                      READY   STATUS             RESTARTS   AGE
athens-588fd7597f-wnxcj   0/1     CrashLoopBackOff   1          6s
athens-588fd7597f-wnxcj   0/1   Error   2     17s
athens-588fd7597f-wnxcj   0/1   CrashLoopBackOff   2     18s
$ kubectl logs athens-588fd7597f-wnxcj -f
buffalo: Unless you set SESSION_SECRET env variable, your session storage is not protected!
time="2019-01-10T14:42:19Z" level=info msg="Exporter not specified. Traces won't be exported"
buffalo: Starting application at tcp://10.3.48.153:80
buffalo: listen tcp: address tcp://10.3.48.153:80: too many colons in address
2019/01/10 14:42:19 context canceled

To Reproduce

athens-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: athens
spec:
  ports:
  - name: http
    port: 80
    targetPort: 3000
  selector:
    app: athens

athens-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: athens
spec:
  replicas: 1
  selector:
    matchLabels:
      app: athens
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: athens
    spec:
      containers:
      - name: athens
        image: docker.io/gomods/athens:v0.2.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 3000
        ports:
        - containerPort: 3000
        env:
        - name: ATHENS_STORAGE_TYPE
          value: disk
        - name: ATHENS_DISK_STORAGE_ROOT
          value: /var/lib/athens
        volumeMounts:
        - name: athens-data
          mountPath: /var/lib/athens
          readOnly: false
      volumes:
      - name: athens-data
        emptyDir: {}

First, apply service specification, then, deployment specification (order matters)

$ kubectl apply -f athens-service.yaml
$ kubectl apply -f athens-deployment.yaml

Expected behavior

It should work no matter the order you apply specifications.

Environment (please complete the following information):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", GitCommit:"0ed33881dc4355495f623c6f22e7dd0b7632b7c0", GitTreeState:"clean", BuildDate:"2018-09-28T15:18:13Z", GoVersion:"go1.11", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

[marwan-at-work]

@amarruedo what is the failure message that you get? Athens shouldn't know anything about a K8s Service object.

[amarruedo]

@marwan-at-work

buffalo: Unless you set SESSION_SECRET env variable, your session storage is not protected!
time="2019-01-10T14:42:19Z" level=info msg="Exporter not specified. Traces won't be exported"
buffalo: Starting application at tcp://10.3.48.153:80
buffalo: listen tcp: address tcp://10.3.48.153:80: too many colons in address
2019/01/10 14:42:19 context canceled

[marwan-at-work]

buffalo: listen tcp: address tcp://10.3.48.153:80: too many colons in address

looks like the colon at the end of the address is incorrect, is that somehow K8s's fault?

[amarruedo]

looks like the colon at the end of the address is incorrect, is that somehow K8s's fault?

I would say that this could be more an issue on Buffalo side. I've found this and I'm not sure if this means that there is some sort of problem with buffalo based apps (which I beleive is Athen's case) on Kubernetes. I'll research it a little bit more on monday.

[marwan-at-work]

@amarruedo the next Athens release won't have Buffalo as its router, so it might be great to retest with the upcoming release and see if that solves our problem. Nonetheless, I think this warrants a patch fix on v0.2.0 if we can get to the bottom of it. Thanks

[amarruedo]

Hi again!

Today I had some time to do some more research and finally found out whats happening. It has nothing to do with buffalo, it's a Kubernetes issue, more precisely a kubernetes environment variable injection issue.

From Kubernetes documentation:

When a Pod is run on a Node, the kubelet adds a set of environment variables for each active Service. It supports both Docker links compatible variables (see makeLinkVariables) and simpler {SVCNAME}_SERVICE_HOST and {SVCNAME}_SERVICE_PORT variables, where the Service name is upper-cased and dashes are converted to underscores.

Since I named my service athens, Kubernetes was injecting following environment variables to my athens Pod:

ATHENS_SERVICE_HOST=10.3.104.46
ATHENS_PORT=tcp://10.3.104.46:80
ATHENS_PORT_80_TCP=tcp://10.3.104.46:80
ATHENS_PORT_80_TCP_ADDR=10.3.104.46
ATHENS_PORT_80_TCP_PROTO=tcp
ATHENS_SERVICE_PORT=80
ATHENS_PORT_80_TCP_PORT=80
ATHENS_SERVICE_PORT_HTTP=80

The thing is that athens itself allready uses an environment variable named ATHENS_PORT for its configuration (that should have a numeric value i.e. ATHENS_PORT=3000). What's happening is that Kubernetes is writing that variable putting in it the wrong value ATHENS_PORT=tcp://10.3.104.46:80. When athens tries to start using that port, it fails with listen tcp: address tcp://10.3.104.46:80: too many colons in address.

Changing service name to athens-proxy:

apiVersion: v1
kind: Service
metadata:
  name: athens-proxy
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 3000
  selector:
    app: athens

then we've got following injected variables:

ATHENS_PROXY_PORT_80_TCP=tcp://10.3.218.66:80
ATHENS_PROXY_PORT_80_TCP_ADDR=10.3.218.66
ATHENS_PROXY_PORT=tcp://10.3.218.66:80
ATHENS_PROXY_PORT_80_TCP_PORT=80
ATHENS_PROXY_SERVICE_PORT=80
ATHENS_PROXY_SERVICE_PORT_HTTP=80
ATHENS_PROXY_SERVICE_HOST=10.3.218.66
ATHENS_PROXY_PORT_80_TCP_PROTO=tcp

and then everything works as expected, since there is no ATHENS_PORT variable collision.

There is a question in stackoverflow regarding this issue, and apparently the only thing that we can do is to be aware of this collision possibility.

[bcspragu]

Commenting from the future to thank you for saving me a lot of time debugging this. Another fix, if (like me) you want to keep the name athens, is to just set the env variable explicitly, e.g.

env:
- name: ATHENS_PORT
  value: "3000"