Force Replication #7791

mosleyit · 2025-02-02T00:37:14Z

mosleyit
Feb 2, 2025

I have a HA Ozone Cluster and something has happened causing some datanodes to not have all the data.

Is there a way to force ozone to replicate or to check everything and repair?

errose28 · 2025-02-03T21:57:38Z

errose28
Feb 3, 2025
Collaborator

Can you elaborate on the issue a more? Are you seeing under-replicated containers? Ozone (SCM specifically) should be replicating those automatically if there are enough datanodes with space available. Some commands you can use to check:

ozone admin datanode list will show the datanodes currently up and running in the cluster
ozone admin container report will provide counts and examples of specific containers that may be in undesirable states like over/under replicated.
ozone admin container info <id> [--json] will give specific info on the replicas of a given container you may have seen referenced from the container report command above.

1 reply

mosleyit Feb 4, 2025
Author

@errose28 Effectively what we found is that through our stress testing of ozone that something happened causing us to only have two copies of the data on 2 of the three nodes. Through stress and failure testing it seems that it lost track of under replicated files.

We are configured with RATIS with a replication factor of three. We found this issue when taking down a datanode and then ensuring that we could still read all the files.

After doing this we found that some files faced errors when downloading. Inspecting further we found that the data only existed on two nodes yest did not show anything as under replicated.

When inspecting the container we received this error:

ozone@nas-atx:/opt/ozone/bin$ ./ozone debug container inspect
{
  "containerID": 1,
  "schemaVersion": "3",
  "containerState": "CLOSED",
  "currentDatanodeID": "e5d028ce-f2af-468a-b7e3-f41da849f7fa",
  "originDatanodeID": "e5d028ce-f2af-468a-b7e3-f41da849f7fa",
  "dBMetadata": {
    "#BLOCKCOUNT": 1502,
    "#BYTESUSED": 2903970822,
    "#PENDINGDELETEBLOCKCOUNT": null,
    "#delTX": null,
    "#BCSID": 95820
  },
  "aggregates": {
    "blockCount": 1502,
    "usedBytes": 2903970822,
    "pendingDeleteBlocks": 0,
    "pendingDeleteBytes": 0
  },
  "chunksDirectory": {
    "path": "/data/ozone/dn/nvme1n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/1/chunks",
    "present": true,
    "fileCount": 0
  },
  "correct": true,
  "errors": []
}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007990d2ffd552, pid=3847257, tid=3847320
#
# JRE version: OpenJDK Runtime Environment (11.0.25+9) (build 11.0.25+9-post-Ubuntu-1ubuntu124.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.25+9-post-Ubuntu-1ubuntu124.04, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [librocksdbjni16566112592457274601.so+0x5fd552]  rocksdb::ColumnFamilyOptions::Dump(rocksdb::Logger*) const+0x382
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /opt/ozone/bin/core.3847257)
#
# An error report file with more information is saved as:
# /opt/ozone/bin/hs_err_pid3847257.log
#
# If you would like to submit a bug report, please visit:
#   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

We also went further once we found a file that was missing and inspected the locations for chunks. We found:

ozone@nas-atx:/opt/ozone/bin$ ./ozone debug chunkinfo s3v/wordcount/2019/04/26/in-s-5082742415-20190426-221527-1556316927.47.wav
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.gson.internal.reflect.ReflectionHelper (file:/opt/ozone/share/ozone/lib/gson-2.9.0.jar) to field java.lang.ThreadLocal.threadLocalHashCode
WARNING: Please consider reporting this to the maintainers of com.google.gson.internal.reflect.ReflectionHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
{
  "KeyLocations": [
    [
      {
        "Datanode-HostName": "nas.sat.somewhere.co",
        "Datanode-IP": "192.168.11.31",
        "Container-ID": 20,
        "Block-ID": 115816896921600335,
        "Locations": {
          "files": [
            "/data/ozone/dn/nvme4n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block"
          ],
          "pipelineID": "f063c805-34b8-4852-810d-108bd84c6f23"
        }
      },
      {
        "Datanode-HostName": "nas.atx.somewhere.co",
        "Datanode-IP": "192.168.11.30",
        "Container-ID": 20,
        "Block-ID": 115816896921600335,
        "Locations": {
          "files": [
            "/data/ozone/dn/nvme1n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block"
          ],
          "pipelineID": "f063c805-34b8-4852-810d-108bd84c6f23"
        }
      },
      {
        "Datanode-HostName": "nas.dfw.somewhere.co",
        "Datanode-IP": "192.168.68.15",
        "Container-ID": 20,
        "Block-ID": 115816896921600335,
        "Locations": {
          "files": [
            "/data/ozone/dn/nvme5n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block"
          ],
          "pipelineID": "f063c805-34b8-4852-810d-108bd84c6f23"
        }
      }
    ]
  ]
}
ozone@nas-atx:/opt/ozone/bin$

And found that chunks were missing:

jeffmosley@nas-dfw:~$ ls -l /data/ozone/dn/nvme5n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block
ls: cannot access '/data/ozone/dn/nvme5n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block': No such file or directory

ozone@nas-sat:/opt/ozone/bin$ ls -l /data/ozone/dn/nvme4n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block
-rw-r--r-- 1 ozone ozone 118124 Jan 27 00:29 /data/ozone/dn/nvme4n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block

ozone@nas-atx:/opt/ozone/bin$ ls -l /data/ozone/dn/nvme1n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block
ls: cannot access '/data/ozone/dn/nvme1n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block': No such file or directory

Once we realized that this was the case and recon was not seeing this. We started going down the path of forcing replication. We have found that there is no way to force replication that we can find.

We attempted to use the freon tools but it seems they are meant for non production use.

Reviewing further we found that if we decommission a node it will check the replication status of everything. We attempted this and it just does nothing. Our belief is that the reason is it needs three active nodes. Since our testing cluster is only 3 when we take down the datanode it does not have another node to maintain a quorum and fails.

Our concern is that none of this is being reported by recon on container info. We also believe that this happened as we showed 1000s of under replicated files at one point in time (we did not log them but should have) and assumed that they just magically replicated but at the same time we were stress testing taking nodes out of service bringing them back etc.

In a perfect world it would be nice to have a utility that can iterate over everything and ensure replication is consistent.

Any feedback is MUCH appreciated and we are setting up a second cluster in order to do further testing.

We are using 1.4.1 binary for reference.

Current ozone-site.xml for reference:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <!-- Basic Configuration -->
    <property>
        <name>ozone.enabled</name>
        <value>true</value>
    </property>
    
    <!-- SCM HA Configuration -->
    <property>
        <name>ozone.scm.service.ids</name>
        <value>scm-ha</value>
    </property>
    
    <property>
        <name>ozone.scm.nodes.scm-ha</name>
        <value>scm1,scm2,scm3</value>
    </property>
    
    <property>
        <name>ozone.scm.address.scm-ha.scm1</name>
        <value>scm.atx.somewhere.co</value>
    </property>
    
    <property>
        <name>ozone.scm.address.scm-ha.scm2</name>
        <value>scm.sat.somewhere.co</value>
    </property>
    
    <property>
        <name>ozone.scm.address.scm-ha.scm3</name>
        <value>scm.dfw.somewhere.co</value>
    </property>
    
    <!-- Port Configuration -->
    <property>
        <name>ozone.scm.client.port</name>
        <value>9860</value>
    </property>
    
    <property>
        <name>ozone.scm.block.client.port</name>
        <value>9863</value>
    </property>
    
    <property>
        <name>ozone.scm.datanode.port</name>
        <value>9861</value>
    </property>
    
    <!-- Storage Directories -->
    <property>
        <name>ozone.metadata.dirs.permissions</name>
        <value>750</value>
    </property>
    <property>
        <name>ozone.metadata.dirs</name>
        <value>/data/ozone/scm/meta</value>
    </property>
    <property>
        <name>ozone.scm.db.dirs.permissions</name>
        <value>750</value>
    </property>
    <property>
        <name>ozone.scm.db.dirs</name>
        <value>/data/ozone/scm/db</value>
    </property>
    <property>
        <name>ozone.scm.ha.ratis.storage.dir</name>
        <value>/data/ozone/scm/ratis</value>
    </property>

    <!-- Security Configuration -->
    <property>
        <name>ozone.security.enabled</name>
        <value>false</value>
    </property>
    
    
    <!-- Metrics Configuration -->
    <property>
        <name>ozone.metrics.enabled</name>
        <value>true</value>
    </property>
    
    <property>
        <name>ozone.metrics.prometheus.endpoint</name>
        <value>prometheus.somewhere.co</value>
    </property>
    <!-- OM HA Configuration --> 
    <property>
        <name>ozone.scm.ratis.enable</name>
        <value>true</value>
    </property>
    <property>
        <name>ozone.om.ratis.enable</name>
        <value>true</value>
    </property>
    <property> 
        <name>ozone.om.service.ids</name> 
        <value>om-ha</value> 
    </property> 
    <property> 
        <name>ozone.om.nodes.om-ha</name> 
        <value>om1,om2,om3</value> 
    </property> 
    <property> 
        <name>ozone.om.address.om-ha.om1</name> 
        <value>om.atx.somewhere.co</value> 
    </property> 
    <property> 
        <name>ozone.om.address.om-ha.om2</name> 
        <value>om.sat.somewhere.co</value> 
    </property> 
    <property> 
        <name>ozone.om.address.om-ha.om3</name> 
        <value>om.dfw.somewhere.co</value> 
    </property> 
    <property>
        <name>ozone.om.ratis.storage.dir</name>
        <value>/data/ozone/om/ratis</value>
    </property>
    <property>
        <name>ozone.om.ratis.snapshot.dir</name>
        <value>/data/ozone/om/snapshot</value>
    </property>
    <!-- Port Configuration --> 
    <property> 
        <name>ozone.om.rpc-port</name> 
        <value>9862</value> 
    </property> 
    <property> 
        <name>ozone.om.http-port</name> 
        <value>9874</value> 
    </property> 
    <property> 
        <name>ozone.om.https-port</name> 
        <value>9875</value> 
    </property> 
    <property> 
        <name>ozone.om.ha-port</name> 
        <value>9876</value> 
    </property> 
    <!-- Storage Directories --> 
    <property>
        <name>ozone.om.db.dirs.permissions</name>
        <value>750</value>
    </property> 
    <property> 
        <name>ozone.om.db.dirs</name> 
        <value>/data/ozone/om/db</value> 
    </property> 
    <!-- DataNode Configuration --> 
    <property> 
        <name>hdds.datanode.dir</name> 
        <value>/data/ozone/dn/nvme0n1,/data/ozone/dn/nvme1n1,/data/ozone/dn/nvme2n1,/data/ozone/dn/nvme3n1,/data/ozone/dn/nvme4n1,/data/ozone/dn/nvme5n1,/data/ozone/dn/nvme6n1,/data/ozone/dn/nvme7n1,/data/ozone/dn/nvme8n1,/data/ozone/dn/nvme10n1</value>  
    </property> 
    <property> 
        <name>hdds.container.ratis.datanode.storage.dir</name> 
        <value>/data/ozone/dn/ratis</value> 
    </property> 
    <property> 
        <name>ozone.scm.datanode.id.dir</name> 
        <value>/data/ozone/dn/id</value> 
    </property>
    <property> 
        <name>hdds.datanode.container.db.dir</name> 
        <value>/data/ozone/dn/db</value> 
    </property> 
    <property> 
        <name>hdds.datanode.http-address</name> 
        <value>0.0.0.0:9882</value> 
    </property> 
    <property> 
        <name>hdds.datanode.https-address</name> 
        <value>0.0.0.0:9883</value> 
    </property> 
    <!-- Storage Configuration --> 
    <property> 
        <name>hdds.datanode.storage.utilization.critical.threshold</name> 
        <value>0.95</value> 
    </property> 
    <property> 
        <name>hdds.datanode.storage.utilization.warning.threshold</name> 
        <value>0.85</value> 
    </property> 
    <property> 
        <name>ozone.scm.names</name> 
        <value>scm.atx.somewhere.co,scm.sat.somewhere.co,scm.dfw.somewhere.co</value> 
    </property>
        <!-- Recon Configuration -->
    <property>
        <name>ozone.recon.http-address</name>
        <value>0.0.0.0:9888</value>
    </property>
    
    <property>
        <name>ozone.recon.https-address</name>
        <value>0.0.0.0:9889</value>
    </property>    
    <property>
        <name>ozone.recon.db.dir</name>
        <value>/data/ozone/recon/db</value>
    </property>
    <property>
        <name>ozone.recon.db.dirs.permissions</name>
        <value>750</value>
    </property>
    <property>
        <name>ozone.recon.om.db.dir</name>
        <value>/data/ozone/recon/omdb</value>
    </property>
    <property>
        <name>ozone.recon.prometheus.http.endpoint</name>
        <value>http://prometheus.somewhere.co:9090</value>
    </property>
    <property>
        <name>ozone.recon.address</name>
        <value>recon.atx.somewhere.co:9891</value>
    </property>
    <property>
        <name>hdds.container.ratis.enabled</name>
        <value>true</value>
    </property>
</configuration>

NOTE: edited 15 times for formating!
NOTE: and i cannot spell formatting.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Force Replication #7791

{{title}}

Replies: 1 comment 1 reply

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Force Replication #7791

mosleyit Feb 2, 2025

Replies: 1 comment · 1 reply

errose28 Feb 3, 2025 Collaborator

mosleyit Feb 4, 2025 Author

mosleyit
Feb 2, 2025

Replies: 1 comment 1 reply

errose28
Feb 3, 2025
Collaborator

mosleyit Feb 4, 2025
Author