Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RATIS-2206. Jars in Maven repo and binary tarball are not the same #1188

Merged
merged 1 commit into from
Dec 14, 2024

Conversation

adoroszlai
Copy link
Contributor

What changes were proposed in this pull request?

During release process, Ratis is currently re-built from scratch in several steps. Outputs of various steps are published to two different locations.

  1. prepare-src: builds everything, installs artifacts (jars, tarballs, etc.) in local repo
  2. prepare-bin: verifies that Ratis can be built from the contents of the source tarball created by prepare-src
  3. assembly: adds src tarball from step (1) and binary tarball from step (2) to Subversion
  4. publish-mvn: publish artifacts to the Maven repo

Problem: Both prepare-bin and publish-mvn re-build Ratis and install build artifacts to the local repo.

Result:

  • Binary tarball in Subversion and Maven repo are different, created by prepare-bin and publish-mvn, respectively.
  • Jars in binary tarball and Maven repo are different.
  • Time unnecessarily spent on re-build in publish-mvn.

This PR updates the release helper script to ensure the artifacts published to Maven repo and those in the binary tarball are the same.

  1. prepare-bin: build, but do not overwrite
  2. assembly: pick both tarballs created by prepare-src
  3. publish-mvn: deploy, but do not re-build

https://issues.apache.org/jira/browse/RATIS-2206

How was this patch tested?

Executed parts of the release process:

git checkout -b RATIS-2206-test

export RATISVERSION=3.2.0
export RC=-test
export CODESIGNINGKEY="..."

./dev-support/make_rc.sh prepare-src
./dev-support/make_rc.sh prepare-bin
./dev-support/make_rc.sh assembly
./dev-support/make_rc.sh publish-mvn

Extracted binary tarball:

cd ..
tar zxf svndistratis/3.2.0/test/apache-ratis-3.2.0-bin.tar.gz

Verified that artifacts in various locations are the same.

  1. tarballs in Subversion and local repo
diff -q svndistratis/3.2.0/test/apache-ratis-3.2.0-bin.tar.gz ratis.repository/org/apache/ratis/ratis-assembly/3.2.0/ratis-assembly-3.2.0-bin.tar.gz
diff -q svndistratis/3.2.0/test/apache-ratis-3.2.0-src.tar.gz ratis.repository/org/apache/ratis/ratis-assembly/3.2.0/ratis-assembly-3.2.0-src.tar.gz
  1. jars
# jars from tarball
for f in $(find apache-ratis-3.2.0-bin/jars -name "ratis-*-$RATISVERSION.jar" | sort); do
  comp=$(basename "$f" | sed -e "s/-$RATISVERSION.jar//")

  # match local repo
  diff -q "$f" "ratis.repository/org/apache/ratis/$comp/$RATISVERSION/$comp-$RATISVERSION.jar"

  # match project build dir (where make_rc.sh was run)
  diff -q "$f" "ratis/$comp/target/$comp-$RATISVERSION.jar"
done

Excerpts from outputs

prepare-src

Built and installed artifacts to local repo:

[INFO] --- install:2.5.2:install (default-install) @ ratis-common ---
[INFO] Installing ratis-common/target/ratis-common-3.2.0.jar to ../ratis.repository/org/apache/ratis/ratis-common/3.2.0/ratis-common-3.2.0.jar
[INFO] Installing ratis-common/pom.xml to ../ratis.repository/org/apache/ratis/ratis-common/3.2.0/ratis-common-3.2.0.pom
...
[INFO] --- install:2.5.2:install (default-install) @ ratis-assembly ---
[INFO] Installing ratis-assembly/pom.xml to ../ratis.repository/org/apache/ratis/ratis-assembly/3.2.0/ratis-assembly-3.2.0.pom
[INFO] Installing ratis-assembly/target/ratis-assembly-3.2.0-src.tar.gz to ../ratis.repository/org/apache/ratis/ratis-assembly/3.2.0/ratis-assembly-3.2.0-src.tar.gz
[INFO] Installing ratis-assembly/target/ratis-assembly-3.2.0-bin.tar.gz to ../ratis.repository/org/apache/ratis/ratis-assembly/3.2.0/ratis-assembly-3.2.0-bin.tar.gz
...
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:21 min

prepare-bin

Built from extracted source tarball, but did not overwrite artifacts in local repo.

[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  53.882 s

publish-mvn

Did not rebuild, only tried to publish existing jars. (But failed due to missing credentials, since I did not want to really publish it.)

[INFO] -----------------------< org.apache.ratis:ratis >-----------------------
[INFO] Building Apache Ratis 3.2.0                                       [1/19]
[INFO]   from pom.xml
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- deploy:2.8.2:deploy (default-cli) @ ratis ---
Uploading to apache.staging.https: https://repository.apache.org/service/local/staging/deploy/maven2/org/apache/ratis/ratis/3.2.0/ratis-3.2.0.pom
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Ratis 3.2.0:
[INFO] 
[INFO] Apache Ratis ....................................... FAILURE [  1.123 s]
...
[INFO] Total time:  1.707 s
...
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy (default-cli) on project ratis: Failed to deploy artifacts: Could not transfer artifact org.apache.ratis:ratis:pom:3.2.0 from/to apache.staging.https (https://repository.apache.org/service/local/staging/deploy/maven2): status code: 401, reason phrase: Unauthorized (401) -> [Help 1]

Copy link
Contributor

@szetszwo szetszwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 the change looks good.

Thanks a lot for the detailed description!

@adoroszlai adoroszlai merged commit 703189e into apache:master Dec 14, 2024
12 checks passed
@adoroszlai adoroszlai deleted the RATIS-2206 branch December 14, 2024 12:16
@adoroszlai
Copy link
Contributor Author

Thanks @szetszwo for the review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants