HDDS-10926. Block deletion should update container merkle tree. #6875
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* HDDS-10239-container-reconciliation: HDDS-10372. SCM and Datanode communication for reconciliation (apache#6506) HDDS-10239. Storage Container Reconciliation. (apache#6121)
* HDDS-10887-mt-manager: Rename data tree to container tree
kerneltime
reviewed
Jun 28, 2024
...service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java
Outdated
Show resolved
Hide resolved
kerneltime
reviewed
Jun 28, 2024
...service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java
Outdated
Show resolved
Hide resolved
kerneltime
reviewed
Jun 28, 2024
...c/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java
Outdated
Show resolved
Hide resolved
|
Will give another look post addressing of review comments and rebase. |
…-delete * HDDS-10239-container-reconciliation: HDDS-10887. Implement a basic Merkle Tree Manager. (apache#6778) HDDS-10923. Container Scanner should still scan unhealthy containers. (apache#6809) Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java
errose28
commented
Jul 23, 2024
...tainer-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
Show resolved
Hide resolved
…-delete * HDDS-10239-container-reconciliation: (184 commits) HDDS-10373. Implement framework for capturing Merkle Tree Metrics. (apache#6864) HDDS-11188. Initial setup for new UI layout and enable users to switch to new UI (apache#6953) HDDS-11120. Rich rebalancing status info (apache#6911) HDDS-11187. Fix Event Handling in Recon OMDBUpdatesHandler to Prevent ClassCastException. (apache#6950) HDDS-11213. Bump commons-daemon to 1.4.0 (apache#6971) HDDS-11212. Bump commons-net to 3.11.1 (apache#6973) HDDS-11211. Bump assertj-core to 3.26.3 (apache#6972) HDDS-11210. Bump log4j2 to 2.23.1 (apache#6970) HDDS-11150. Recon Overview page crashes due to failed API Calls (apache#6944) HDDS-11183. Keys from DeletedTable and DeletedDirTable of AOS should be deleted on batch operation while creating a snapshot (apache#6946) HDDS-11198. Fix Typescript configs for Recon (apache#6961) HDDS-11180. Simplify HttpServer2#inferMimeType return statement (apache#6963) HDDS-11194. OM missing audit log for upgrade (apache#6958) HDDS-10389. Implement a search feature for users to locate open keys within the Open Keys Insights section. (apache#6231) HDDS-10561. Dashboard for delete key metrics (apache#6948) HDDS-11192. Increase SPNEGO URL test coverage (apache#6956) HDDS-11179. DBConfigFromFile#readFromFile result of toIOException not thrown (apache#6957) HDDS-11186. First container log missing from bundle (apache#6952) HDDS-10844. Clarify snapshot create error message. (apache#6955) HDDS-11166. Switch to Rocky Linux-based ozone-runner (apache#6942) ...
This will be handled properly once HDDS-11254 is complete.
|
For tests to pass I had to add the same metrics unregister workaround from #6945. |
|
Run on my fork is green, waiting for verification here as well. |
kerneltime
approved these changes
Aug 5, 2024
aswinshakil
approved these changes
Aug 6, 2024
errose28
merged commit d7f302e
into
apache:HDDS-10239-container-reconciliation
50 checks passed
|
Thanks @errose28 . Nice commit messages btw :D |
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Aug 13, 2024
…p-supervisor Merge conflicts are resolved but the change does not yet build. * HDDS-10239-container-reconciliation: (183 commits) HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945) HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024) HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043) HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040) HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036) HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003) HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724) HDDS-10926. Block deletion should update container merkle tree. (apache#6875) HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021) HDDS-11272. Statistics some node status information (apache#7025) HDDS-11278. Move code out of Hadoop util package (apache#7028) HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026) HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032) HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990) HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023) HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007) HDDS-11224. Increase hdds.datanode.handler.count (apache#7011) HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012) HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014) ... Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Aug 16, 2024
…rrupt-files * HDDS-10239-container-reconciliation: (183 commits) HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945) HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024) HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043) HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040) HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036) HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003) HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724) HDDS-10926. Block deletion should update container merkle tree. (apache#6875) HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021) HDDS-11272. Statistics some node status information (apache#7025) HDDS-11278. Move code out of Hadoop util package (apache#7028) HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026) HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032) HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990) HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023) HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007) HDDS-11224. Increase hdds.datanode.handler.count (apache#7011) HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012) HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014) ... Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
When a block is deleted from a container, it should be marked as deleted in the container merkle tree. This tells peer datanodes that the data be skipped if they have not yet processed the deletion but find there is an error with this block. The chunk, block, and container hashes in the tree should not change.
This test also moves some existing test methods to a test util class since more tests, including those in this patch, need to use them.
What is the link to the Apache JIRA
HDDS-10926
How was this patch tested?
Unit tests added