HDDS-11257. Ozone write does not work when http proxy is set for the JVM. #7036

sadanand48 · 2024-08-06T07:00:50Z

What changes were proposed in this pull request?

GRPC uses HTTP internally for its connections and due to this, if http proxy is configured for any Ozone process using grpc , it directs each call through the proxy even for grpc which is not desirable for performance. Hence disabling proxy for grpc connections that ozone uses.
This has been fixed in RATIS via RATIS-2133. For the complete fix we need to update Ozone's ratis version to one which has the fix but that would happen whenever the next ratis update happens.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11257

How was this patch tested?

Since current Ratis version in Ozone does not have the fix, used the flaky test check with custom ratis commit to point to latest commits in apache/ratis
Tested a unit test with this commit having proxy config set in the maven sure fire plugin

      <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <configuration>
        <argLine>-Dhttps.proxyHost=127.0.0.1</argLine>
      </configuration>
      </plugin>

https://github.com/sadanand48/hadoop-ozone/actions/runs/10261429774/job/28389549624

…JVM.

adoroszlai

Thanks @sadanand48 for the patch.

There is one more we might want to change:

ozone/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/GrpcOmTransport.java

Lines 145 to 148 in 96bb60e

    
           NettyChannelBuilder channelBuilder = 
        
               NettyChannelBuilder.forAddress(hp.getHost(), hp.getPort()) 
        
                   .usePlaintext() 
        
                   .maxInboundMessageSize(maxSize);

Tejaskriya

Thanks for the patch @sadanand48
The changes LGTM!

adoroszlai · 2024-08-07T08:49:35Z

Thanks @sadanand48 for the patch, @Tejaskriya for the review.

* master: (181 commits) HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024) HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043) HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040) HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036) HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003) HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724) HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021) HDDS-11272. Statistics some node status information (apache#7025) HDDS-11278. Move code out of Hadoop util package (apache#7028) HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026) HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032) HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990) HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023) HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007) HDDS-11224. Increase hdds.datanode.handler.count (apache#7011) HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012) HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014) HDDS-11226. Make ExponentialBackoffPolicy maxRetries configurable (apache#6985) HDDS-11260. [hsync] Add Ozone Manager protocol version (apache#7015) ... Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/audit/DNAction.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java hadoop-hdds/interface-client/src/main/proto/DatanodeClientProtocol.proto hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerReportHandler.java

…p-supervisor Merge conflicts are resolved but the change does not yet build. * HDDS-10239-container-reconciliation: (183 commits) HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945) HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024) HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043) HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040) HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036) HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003) HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724) HDDS-10926. Block deletion should update container merkle tree. (apache#6875) HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021) HDDS-11272. Statistics some node status information (apache#7025) HDDS-11278. Move code out of Hadoop util package (apache#7028) HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026) HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032) HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990) HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023) HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007) HDDS-11224. Increase hdds.datanode.handler.count (apache#7011) HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012) HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014) ... Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java

…rrupt-files * HDDS-10239-container-reconciliation: (183 commits) HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945) HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024) HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043) HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040) HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036) HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003) HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724) HDDS-10926. Block deletion should update container merkle tree. (apache#6875) HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021) HDDS-11272. Statistics some node status information (apache#7025) HDDS-11278. Move code out of Hadoop util package (apache#7028) HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026) HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032) HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990) HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023) HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007) HDDS-11224. Increase hdds.datanode.handler.count (apache#7011) HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012) HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014) ... Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java

HDDS-11257. Ozone write does not work when http proxy is set for the …

822879e

…JVM.

sadanand48 marked this pull request as draft August 6, 2024 07:00

remove config for tests

cbee623

sadanand48 marked this pull request as ready for review August 6, 2024 07:06

fix checkstyl

ffa687b

sadanand48 marked this pull request as draft August 6, 2024 07:20

sadanand48 marked this pull request as ready for review August 6, 2024 07:36

sadanand48 force-pushed the HDDS-11257 branch from 725f79a to ffa687b Compare August 6, 2024 07:37

adoroszlai reviewed Aug 6, 2024

View reviewed changes

address comment

ad8df33

Tejaskriya approved these changes Aug 6, 2024

View reviewed changes

sadanand48 requested a review from adoroszlai August 7, 2024 08:11

adoroszlai approved these changes Aug 7, 2024

View reviewed changes

adoroszlai merged commit 6569278 into apache:master Aug 7, 2024
39 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-11257. Ozone write does not work when http proxy is set for the JVM. #7036

HDDS-11257. Ozone write does not work when http proxy is set for the JVM. #7036

sadanand48 commented Aug 6, 2024 •

edited

Loading

adoroszlai left a comment

Tejaskriya left a comment

adoroszlai commented Aug 7, 2024

	NettyChannelBuilder channelBuilder =
	NettyChannelBuilder.forAddress(hp.getHost(), hp.getPort())
	.usePlaintext()
	.maxInboundMessageSize(maxSize);

HDDS-11257. Ozone write does not work when http proxy is set for the JVM. #7036

HDDS-11257. Ozone write does not work when http proxy is set for the JVM. #7036

Conversation

sadanand48 commented Aug 6, 2024 • edited Loading

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

adoroszlai left a comment

Choose a reason for hiding this comment

Tejaskriya left a comment

Choose a reason for hiding this comment

adoroszlai commented Aug 7, 2024

sadanand48 commented Aug 6, 2024 •

edited

Loading