Bootstrapping OM High Availability

[mosleyit]

We are trying to setup a 3 node ozone cluster running both SCM and OM in HA.

We are running into problems bootstrapping the additional OM nodes. When we try and bootstrap the 2nd node it fails because it cannot communicate with the first node. I am not sure what we are missing.

Steps:

Node1
ozone scm --init
ozone scm --daemon start

Node2
ozone scm --bootstrap
ozone scm --daemon start

Node3
ozone scm --bootstrap
ozone scm --daemon start

**Node1
ozone om --init
ozone om --daemon start

Node2
ozone om --bootstrap

This is where everything fails as it cannot connect to OM on Node3 which we have not bootstrapped in yet.

Please find my ozone-site.xml (example.com is being used in place of personal identifiers for this post)

`

ozone.enabled true

<!-- SCM HA Configuration -->
<property>
    <name>ozone.scm.primordial.node.id</name>
    <value>scm1</value>
</property>
<property>
    <name>ozone.scm.service.ids</name>
    <value>scm-ha</value>
</property>

<property>
    <name>ozone.scm.nodes.scm-ha</name>
    <value>scm1,scm2,scm3</value>
</property>

<property>
    <name>ozone.scm.address.scm-ha.scm1</name>
    <value>scm.atx.example.com</value>
</property>

<property>
    <name>ozone.scm.address.scm-ha.scm2</name>
    <value>scm.sat.example.com</value>
</property>

<property>
    <name>ozone.scm.address.scm-ha.scm3</name>
    <value>scm.dfw.example.com</value>
</property>

<!-- Port Configuration -->
<property>
    <name>ozone.scm.client.port</name>
    <value>9860</value>
</property>

<property>
    <name>ozone.scm.block.client.port</name>
    <value>9863</value>
</property>

<property>
    <name>ozone.scm.datanode.port</name>
    <value>9861</value>
</property>

<!-- Storage Directories -->
<property>
    <name>ozone.metadata.dirs.permissions</name>
    <value>750</value>
</property>
<property>
    <name>ozone.metadata.dirs</name>
    <value>/data/ozone/scm/meta</value>
</property>
<property>
    <name>ozone.scm.db.dirs.permissions</name>
    <value>750</value>
</property>
<property>
    <name>ozone.scm.db.dirs</name>
    <value>/data/ozone/scm/db</value>
</property>
<property>
    <name>ozone.scm.ha.ratis.storage.dir</name>
    <value>/data/ozone/scm/ratis</value>
</property>

<!-- Security Configuration -->
<property>
    <name>ozone.security.enabled</name>
    <value>false</value>
</property>


<!-- Metrics Configuration -->
<property>
    <name>ozone.metrics.enabled</name>
    <value>true</value>
</property>

<property>
    <name>ozone.metrics.prometheus.endpoint</name>
    <value>prometheus.example.com</value>
</property>
<!-- OM HA Configuration --> 
<property>
    <name>ozone.scm.ratis.enable</name>
    <value>true</value>
</property>
<property>
    <name>ozone.om.ratis.enable</name>
    <value>true</value>
</property>
<property> 
    <name>ozone.om.service.ids</name> 
    <value>om-ha</value> 
</property> 
<property> 
    <name>ozone.om.nodes.om-ha</name> 
    <value>om1,om2,om3</value> 
</property> 
<property> 
    <name>ozone.om.address.om-ha.om1</name> 
    <value>om.atx.example.com</value> 
</property> 
<property> 
    <name>ozone.om.address.om-ha.om2</name> 
    <value>om.sat.example.com</value> 
</property> 
<property>
    <name>ozone.om.address.om-ha.om3</name>
    <value>om.dfw.example.com</value>
</property>
<property>
    <name>ozone.om.ratis.storage.dir</name>
    <value>/data/ozone/om/ratis</value>
</property>
<property>
    <name>ozone.om.ratis.snapshot.dir</name>
    <value>/data/ozone/om/snapshot</value>
</property>
<!-- Port Configuration --> 
<property> 
    <name>ozone.om.rpc-port</name> 
    <value>9862</value> 
</property> 
<property> 
    <name>ozone.om.http-port</name> 
    <value>9874</value> 
</property> 
<property> 
    <name>ozone.om.https-port</name> 
    <value>9875</value> 
</property> 
<property> 
    <name>ozone.om.ha-port</name> 
    <value>9876</value> 
</property> 
<!-- Storage Directories --> 
<property>
    <name>ozone.om.db.dirs.permissions</name>
    <value>750</value>
</property> 
<property> 
    <name>ozone.om.db.dirs</name> 
    <value>/data/ozone/om/db</value> 
</property> 
<!-- DataNode Configuration --> 
<property> 
    <name>hdds.datanode.dir</name> 
    <value>/data/ozone/dn/nvme0n1,/data/ozone/dn/nvme1n1,/data/ozone/dn/nvme2n1,/data/ozone/dn/nvme3n1,/data/ozone/dn/nvme4n1,/data/ozone/dn/nvme5n1,/data/ozone/dn/nvme6n1,/data/ozone/dn/nvme7n1,/data/ozone/dn/nvme8n1,/data/ozone/dn/nvme10n1</value>  
</property> 
<property> 
    <name>hdds.container.ratis.datanode.storage.dir</name> 
    <value>/data/ozone/dn/ratis</value> 
</property> 
<property> 
    <name>ozone.scm.datanode.id.dir</name> 
    <value>/data/ozone/dn/id</value> 
</property>
<property> 
    <name>hdds.datanode.container.db.dir</name> 
    <value>/data/ozone/dn/db</value> 
</property> 
<property> 
    <name>hdds.datanode.http-address</name> 
    <value>0.0.0.0:9882</value> 
</property> 
<property> 
    <name>hdds.datanode.https-address</name> 
    <value>0.0.0.0:9883</value> 
</property> 
<!-- Storage Configuration --> 
<property> 
    <name>hdds.datanode.storage.utilization.critical.threshold</name> 
    <value>0.95</value> 
</property> 
<property> 
    <name>hdds.datanode.storage.utilization.warning.threshold</name> 
    <value>0.85</value> 
</property> 
<property> 
    <name>ozone.scm.names</name> 
    <value>scm.atx.example.com,scm.sat.example.com,scm.dfw.example.com</value> 
</property>
    <!-- Recon Configuration -->
<property>
    <name>ozone.recon.http-address</name>
    <value>0.0.0.0:9888</value>
</property>

<property>
    <name>ozone.recon.https-address</name>
    <value>0.0.0.0:9889</value>
</property>    
<property>
    <name>ozone.recon.db.dir</name>
    <value>/data/ozone/recon/db</value>
</property>
<property>
    <name>ozone.recon.db.dirs.permissions</name>
    <value>750</value>
</property>
<property>
    <name>ozone.recon.om.db.dir</name>
    <value>/data/ozone/recon/omdb</value>
</property>
<property>
    <name>ozone.recon.prometheus.http.endpoint</name>
    <value>http://prometheus.example.com:9090</value>
</property>

`

[ChenSammi]

Can you paste the failure stack of OM bootstrap? BTW, are above "scm.atx.example.com" and "om.atx.example.com" actually the same node, but different hostname?

[mosleyit]

Yes they are the same node with different hostnames.

[nandakumar131]

@mosleyit

For fresh installations, the way we initialize SCM and OM differs a bit.

SCM
Run init on primordial node
scm --init
Run bootstrap on both the non-primordial nodes
scm --bootstrap

OM
For OM, run init on all the three nodes
om --init

When should we use bootstrap for OM?
OM bootstrap is used if there is running cluster and we want to replace one of the existing OM.
We should first decommission one of the OM and bootstrap the new OM node.

tldr; During fresh installation we have run init on all the three OMs.

[mosleyit]

Okay we have solved the issue. We were facing an issue that when doing it this way that one of the nodes would never participate in a vote causing recon to not show all the metrics properly. What we found is that once we run init on all the nodes we force leadership movement to a random node and that causes a full vote to take place.

I am not sure if this is the intended process or if this is a bug in recon. If we just deploy all three without doing the leadership movement then we get a null pointer exception due to prometheus metrics showing either 0 votes or -1 for one node.

[errose28]

@devmadhuu are you aware of any known issues with regards to Recon here? NPE is almost always a bug.

[devmadhuu]

@mosleyit can you please post the NPE error stack you faced in Recon ?

[mosleyit]

@errose28 @devmadhuu I will have to replicate the issue again. We are stress testing ozone now and will be tearing down our cluster this week and rebuilding. I will replicate the issue and provide you as much as I can.