Skip to content

How to release

Jump to bottom
Brian Zhou edited this page Sep 29, 2022 · 13 revisions

If you are releasing a version of Pravega Flink Connector, then you must read and follow the instructions, preferably read ahead a few steps to avoid errors. The steps here are based on the experience we are building across releases, so if you see any point that requires changes or more detail, feel free to raise it or modify the document.

Before you Begin

Before you begin, take a moment to ensure that you have the following:

  1. A set of GPG signing key and password, which will be used for signing the artifacts. Follow these Github instructions for generating and configuring one.

GnuPG 2.1 switched to a new keyring format, for which the keys are stored in the pubring.kbx file. Since the release requires a secring.gpg file, you could create one like this:

# List the keys
gpg --list-secret-keys --keyid-format LONG

# Locate the key id, which is of the form `Dg4D421F...`

# Create the secring.gpg
gpg --export-secret-key <Your key-id> > ~/.gnupg/secring.gpg
  1. Sonatype username/password with access to the io.pravega group. See this guide for how to create an account.

Perform the release

Preparing the branch and release notes

First, please make any necessary changes to the branch you will be working on as part of releasing. There are two possible situations:

  • Bug-fix release: This is a minor release version over an existing release branch, there is no need to create a new branch.
  • Feature release or non-backward-compatible release: This is a major release based on master branch. This is a version change to either the first or the middle digit, and it requires a new release branch. The process is the following, assuming for the sake of example that the new release is 0.3.0:
# Tag the branch cut position with `branch-X.Y`
git checkout master
git tag branch-0.3
git push origin branch-0.3

# Create the release branch `rX.Y`
git checkout -b r0.3
git push origin r0.3

Finally raise an issue and corresponding pull request to update the master branch to the next major version, i.e. 0.4.0-SNAPSHOT. Note that we are bumping up the middle digit because, in our example, we are releasing 0.3.0. If we were releasing say 1.0.0, then we would change connectorVersion to 1.1.0-SNAPSHOT.

Now the release branch is ready to go.

Days before the release starts, please gather the information of the highlight features and important fixes in the release and prepare the release notes. See previous releases for an example of how to put together notes.

Making a release candidate

First identify the branch you'll be working on. It will be named rX.Y, e.g., r0.2. The steps are:

  1. Create a new temporary release branch on the basis of r0.2, named as rc0.2.1-rc0.
  2. Changing connectorVersion and pravegaVersion in gradle.properties from X.Y.Z-SNAPSHOT to X.Y.Z. For example, if the current value is connectorVersion=0.2.1-SNAPSHOT, then change it to connectorVersion=0.2.1.
  3. Push this commit
  4. Tag the commit with vX.Y.Z-rc0 and push this tag. For example, v0.2.1-rc0.
  5. Drop the artifacts in sonatype in the last release candidate if there is one.
  6. Publish the release artifacts in the staging maven repository for verification. (Note: 1. Do not click "Release" in this step! 2. For now, we need to publish the recent three Flink versions(on three branches), so there will be totally three artifacts to publish)
  7. Delete this temporary branch.

Publishing to sonatype maven repository

For step 5, please run the following steps:

  • Build with the following command:
# For Scala 2.12
./gradlew clean assemble publish -PdoSigning=true -Psigning.password=<signing-password> -PpublishUrl=mavenCentral -PpublishUsername=<sonatype-username> -PpublishPassword=<sonatype-password>
# For Scala 2.11
./gradlew clean assemble publish -PdoSigning=true -Psigning.password=<signing-password> -PpublishUrl=mavenCentral -PpublishUsername=<sonatype-username> -PpublishPassword=<sonatype-password> -PflinkScalaVersion=2.11
  • Login to Nexus Repository Manager using sonatype credentials with write access to io.pravega group.
  • Under Build Promotion -> Staging Repositories, locate the staging repository that was created for the latest publish (format iopravega-XXXX, for example iopravega-1004)
  • Select the repository and select the Close button in the top menu bar. This will perform validations to ensure that the contents meets the maven requirements (contains signatures, javadocs, sources, etc). This operation takes a short time to complete, press the Refresh button in the top menu bar occasionally until the operation completes.
  • Once the operation completes, locate the URL field in the Summary tab of the newly closed repository (it will be something like https://oss.sonatype.org/content/repositories/iopravega-XXXX where XXXX is the number of the staging repository). This should be tested to ensure that all artifacts are present and functions as expected.

Verifying and voting the release candidate

The community including the release manager will be responsible for verifying the release candidate. There are a few things to check including

  • All the unit and integration tests pass
  • Checksums of the artifacts and source code are correct
  • Verify that it can locate, build and run with the staging artifacts. For example, use the Pravega samples and checkout dev branch to verify. Better to run some Flink applications in it to make sure the connector works well as expected.

It is possible that a release candidate is problematic and we need to do a new release candidate. In this case, we need to repeat the steps above as many times as needed.

Once the community passes the vote, the formal release can go on.

Formal release

Step 1: Tagging

# Checkout to the latest successful-voted tag
git checkout v0.3.0-rc2
# Tag this commit as `v0.3.0` and push
git tag v0.3.0
git push origin v0.3.0

Step 2: Create a release on GitHub

On the GitHub repository page, go to "Releases" and create a new draft. On the draft, fill out the tag field and select the appropriate branch, and then:

Prepare the distribution artifacts

Run the following commands to build the distribution artifacts

git checkout r0.3
./gradlew clean publishToMavenLocal

The files resulting from the build will be under ~/.m2/repository/io/pravega/pravega-connectors-flink-<FLINK-VERSION>_2.12/<RELEASE-VERSION>. For each one of the .jar files in that directory, generate checksums (currently md5, sha1, and sha256). It is easy to do it with a simple bash script along the lines of:

#!/bin/bash

for file in ./*.jar ; do md5sum $file > $file.md5 ; done
for file in ./*.jar ; do shasum -a 1 $file > $file.sha1 ; done
for file in ./*.jar ; do shasum -a 256 $file > $file.sha256 ; done

Note: In the future, we might want to automate the generation of checksums via gradle.

Prepare the source code

Source code is also required in the release. For source code, we need to clean up the unnecessary files such as IDE settings, local temp files, logs and files in the submodule, and then package the folder into a .zip file and a .tar.gz file. For each file, we still need to generate checksums (currently md5, sha1, and sha256).

Upload the files to the pre-release draft

In the draft on GitHub, upload all the jar files and source code package (and its checksums).

Release notes

Create a release notes text file containing the following:

  1. The prepared introductory text, highlighting the important changes going in the release.
  2. A full list of commits that you can get with the following command:
   > git log <commit-id-of-last-release>..<current-commit-id>
  • <commit-id-of-last-release> depends on the kind of release you are doing. If it is bug-fix release, then we can use the tag of the last branch release. For new branches, we have been adding branch-X.Y tags at the branch point for convenience.
  • <current-commit-id> is the commit point of the release candidate you are working on.

Finally, click the "Publish release" button to publish git github release.

Step 3: Releasing the artifact to maven central

If release candidate phase is ommitted, please refer to the above section to publish the artifact first. Choose the latest release candidate in sonatype and click on the "Release" button in the top menu bar to release the artifact. Wait until it shows up in Maven Central, it takes some time.

After the release

Once the release is done, create an issue and corresponding pull request to change the connectorVersion in gradle.properties to X.Y.(Z+1)-SNAPSHOT for the release branch.

Appendix

Generate documentation files for publishing

For generating the documentation files for publishing to http://pravega.io site, run the following command from the flink-connector workspace location and it generate the docs under <CONNECTOR_HOME>/build/mkdocs/documentation/generated/ location.

./gradlew clean mkdocsBuild

Quick Links

Clone this wiki locally