grpc error missing port in address(No Validation)

[Rader]

I have a server 10.183.105.89 runs a dgraph v0.7.5 with "-bindall" option. Now I try to add another dgraph v0.7.5 on 10.183.105.89 to the cluster. But I get an error.

I think "peer" has format "IP:Port". Am I right?

---

[email protected]:~/dgraph_bin$ ./dgraph_v0.7.5 --groups "2" --idx 3 --my "10.183.105.90" -peer "10.183.105.89:12345"

Dgraph version : v0.7.5
Commit SHA-1 : cea0622
Commit timestamp : 2017-04-12 18:44:27 +1000
Branch : release/v0.7.5

Starting commit routine.
Connection with "10.183.105.89:12345" successful.
2017/04/13 19:01:00 worker.go:90: Worker listening at address: 127.0.0.1:12345
2017/04/13 19:01:00 main.go:849: grpc server started.
2017/04/13 19:01:00 main.go:850: http server started.
2017/04/13 19:01:00 main.go:851: Server listening on port 8080

====== APPLYING MEMBERSHIP UPDATE: {NodeId:1 Addr:10.183.105.89 Leader:true RaftIdx:86}

Group: 1. List: [{NodeId:1 Addr:10.183.105.89 Leader:true RaftIdx:86}]

====== APPLYING MEMBERSHIP UPDATE: {NodeId:1 Addr:10.183.105.89 Leader:true RaftIdx:86}

Group: 1. List: [{NodeId:1 Addr:10.183.105.89 Leader:true RaftIdx:86}]
Group: 0. List: [{NodeId:1 Addr:10.183.105.89 Leader:true RaftIdx:86}]
Last update is now: 86
2017/04/13 19:01:00 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp: address 10.183.105.89: missing port in address"; Reconnecting to {10.183.105.89 }
2017/04/13 19:01:00 conn.go:99: While trying to connect to "10.183.105.89", got error: rpc error: code = Unavailable desc = grpc: the connection is unavailable
2017/04/13 19:01:00 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp: address 10.183.105.89: missing port in address"; Reconnecting to {10.183.105.89 }
2017/04/13 19:01:00 conn.go:99: While trying to connect to "10.183.105.89", got error: rpc error: code = Unavailable desc = grpc: the connection is unavailable

[janardhan1993]

After starting the first server with dgraph --bindall,
Running ./dgraph_v0.7.5 --groups "2" --idx 3 --my "10.183.105.90" -peer "10.183.105.89:12345"
should actually throw an error since it would try to use port 12345 for internal nodes communication which is already binded.
Please try the following command.
./dgraph_v0.7.5 --groups "2" --idx 3 --workerport 12346 -peer "10.183.105.89:12345"

--my flag can be used to tell other instances about the current dgraph instance's address.. The address should be valid with port, "10.183.105.90" is an invalid address. It should be ip:workerport or dns which points to workerport of this instance

[Rader]

@janardhan1993 But dgraph does not complain a invalid address for "--my" flag when I run this command to start the first instance:
./dgraph_v0.7.5 --groups "0,1" --idx 1 --my "10.183.105.89" -bindall

So "--my" flag is not checked when there is no "--peer" flag being set?

Update: after checking the document, I think I am doing right.

# Server handling only group 2.
dgraph --groups "2" --idx 3 --peer "<ip address>:<port>" --my "ip-address-others-should-access-me-at"

# Server handling groups 0, 1 and 2.
dgraph --groups "0,1,2" --idx 4 --peer "<ip address>:<port>" --my "ip-address-others-should-access-me-at"

# If running on the same server as other instances of Dgraph, do set the --port and the --workerport flags.

[janardhan1993]

@Rader On second thought, it's difficult to validate whether the address specified after --my"" points to the dgraph instance or not. (10.183.105.89 points to port 80, but since dgraph is listening on port 12345 by default, it didn't work).
You can specify either ip address, dns or even load balancer. Until the dgraph instance has been fully initialised, the health check would fail so the instance won't be attached under load balancer. So in this case we can't really ping and check whether the advertised address (--my) actually points to itself(my address is needed for server bootstrap).
I think user should ensure that the advertised address actually points to dgraph instance itself(to correct worker port).

[Rader]

@janardhan1993 At least a few things we need to check

1. "--my" flag needs a "port" part. The default port "12345" is not used for "--my" address.
2. "--bindall" should be set if we start dgraph with "--my" flag, or no other dgraph instance can connect to it.

Oh! Now I get the "missing port" error when I start the first dgraph instance, even I have set the IP and port for "-my" flag.

./dgraph_v0.7.5 -bindall -groups "0,1" -idx 1 -my "10.183.105.89:12345"

Dgraph version   : v0.7.5
Commit SHA-1     : cea0622
Commit timestamp : 2017-04-12 18:44:27 +1000
Branch           : release/v0.7.5

Starting commit routine.
2017/04/14 16:18:07 worker.go:90: Worker listening at address: 0.0.0.0:12345
2017/04/14 16:18:07 main.go:849: grpc server started.
2017/04/14 16:18:07 main.go:850: http server started.
2017/04/14 16:18:07 main.go:851: Server listening on port 8080
...
Connection with "localhost:12345" successful.
Connection with "localhost:12345" successful.
Connection with "localhost:12345" successful.
Connection with "localhost:12345" successful.
Connection with "localhost:12345" successful.
Connection with "localhost:12345" successful.
2017/04/14 16:18:08 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp: address 10.183.105.89: missing port in address"; Reconnecting to {10.183.105.89 <nil>}
2017/04/14 16:18:08 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp: address 10.183.105.89: missing port in address"; Reconnecting to {10.183.105.89 <nil>}
2017/04/14 16:18:08 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp: address 10.183.105.89: missing port in address"; Reconnecting to {10.183.105.89 <nil>}
2017/04/14 16:18:08 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp: address 10.183.105.89: missing port in address"; Reconnecting to {10.183.105.89 <nil>}
2017/04/14 16:18:08 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp: address 10.183.105.89: missing port in address"; Reconnecting to {10.183.105.89 <nil>}

[Rader]

More information about this issue. I found that if I set wrong address for "-my" or "-peer", e.g. forget the port, then Dgraph will fail to join the cluster even if I set the right address for "-my" and "-peer" when restart it.

The solution is to remove the "w" folder before restart Dgraph. I think the cluster config in WAL should be overwritten by the config set through command line.

[janardhan1993]

Thanks for pointing out the issues. We will fix them soon.