Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation on testing toolchain changes is out of date #43

Open
cushon opened this issue Jan 26, 2021 · 10 comments
Open

Documentation on testing toolchain changes is out of date #43

cushon opened this issue Jan 26, 2021 · 10 comments
Assignees
Labels
P2 P2 We'll consider to work on this in future. (Assignee optional)

Comments

@cushon
Copy link

cushon commented Jan 26, 2021

@comius

This page is out of date: https://github.com/bazelbuild/java_tools/blob/master/docs/behind-the-release.md#manually-trying-out-java_tools-before-the-release-process

  • The //src:java_tools_javaX.zip target doesn't exist anymore (I think it's been replaced by e.g. //src:java_tools_zip).

  • It refers to --java_toolchain, which doesn't work with bazel at head due to the switch to toolchain resolution.

@cushon
Copy link
Author

cushon commented Apr 16, 2021

I think the current process is something like the following. @comius does that look right, or is there a better way to do this?

  • From the main Bazel repo, build //src:java_tools_zip and stash it somewhere:

    $ bazel build //src:java_tools_zip
    $ cp bazel-bin/src/java_tools.zip /tmp/
    
  • Patch distdir_deps.bzl to refer to the local java_tools_zip:

    --- a/distdir_deps.bzl
    +++ b/distdir_deps.bzl
    @@ -267,11 +267,10 @@ DIST_DEPS = {
                 "remote_java_tools_test",
                 "remote_java_tools_for_testing",
             ],
    -        "archive": "java_tools-v11.3.zip",
    -        "sha256": "52b66d8df456f2ce057d8e435904789463df8414af8110aa13af68ce32d8c4cc",
    +        "archive": "java_tools3.zip",
    +        "sha256": "6e59059c571571528d7e5e264d6ec6bba4ba570241f71fc6d31f17cbbcb30fc8",
             "urls": [
    -            "https://mirror.bazel.build/bazel_java_tools/releases/java/v11.3/java_tools-v11.3.zip",
    -            "https://github.com/bazelbuild/java_tools/releases/download/java_v11.3/java_tools-v11.3.zip",
    +            "file:///tmp/java_tools.zip",
             ],
             "used_in": [
                 "additional_distfiles",
  • Build and run the patched Bazel

@comius
Copy link
Collaborator

comius commented Apr 16, 2021

I didn't see this issue. I will update docs. Yes, a lot has changed, for example toolchain definitions are not anymore in java_tools.

or is there a better way to do this?

This is the best way to run all the tests.

You could also unzip java_tools and add --override_repository=java_tools=/tmp/java_tools to just use them in a build for arbitrary project.

@comius comius self-assigned this Apr 16, 2021
@cushon
Copy link
Author

cushon commented Sep 28, 2021

Does --override_repository=java_tools=/tmp/java_tools still work? I tried that and it doesn't seem to do anything.

The approach in #43 (comment) still works for me.

@cushon
Copy link
Author

cushon commented Jan 31, 2022

You could also unzip java_tools and add --override_repository=java_tools=/tmp/java_tools to just use them in a build for arbitrary project.

Does --override_repository=java_tools=/tmp/java_tools still work? I tried that and it doesn't seem to do anything.

@comius ping? It would be nice to have a way to test java_tools changes without editing distdir_deps.bzl and rebuilding bazel. Is --override_repository= still a supported way to do that? Maybe I'm just holding it wrong, but it doesn't work for me.

The following does not pick up any local modifications to java_tools:

bazel build //src:java_tools_zip
unzip -q -o -d /tmp bazel-bin/src/java_tools.zip
bazel build --override_repository=java_tools=/tmp/java_tools //src:bazel

@comius
Copy link
Collaborator

comius commented Jan 31, 2022

The mechanism should still work, except that you need to override both java_tools and java_tools_linux repositories.
You build zips with bazel build //src:java_tools_zip and bazel build //src:java_tools_prebuilt_zip.

@cushon
Copy link
Author

cushon commented Jan 31, 2022

Is this the right idea? I can't get it to work:

bazel build //src:java_tools_zip //src:java_tools_prebuilt_zip
unzip -q -o -d /tmp bazel-bin/src/java_tools.zip
unzip -q -o -d /tmp/java_tools_prebuilt bazel-bin/src/java_tools_prebuilt.zip
bazel build --override_repository=java_tools=/tmp/java_tools --override_repository=java_tools_prebuilt=/tmp/java_tools_prebuilt //src:bazel

I'm testing a repro for bazelbuild/bazel#14093, which causes JavaBuilder to unconditionally OOM, and the only way I can see the behaviour I'm introducing is by patching distdir_deps.bzl

@oliviernotteghem
Copy link

Facing the same issue. Unfortunately, your workaround @cushon doesn't seem to work anymore (we're on Bazel 5.2). Getting the following error. Does the workaround still work for you?

ERROR: /Users/oliviern/Uber/android/build/.bazelw_cache/91a99194e71852b0e089ca5cc1c48073/external/rules_java/toolchains/BUILD:116:14: no such target '@remote_java_tools_darwin//:ijar_prebuilt_binary': target 'ijar_prebuilt_binary' not declared in package '' defined by /Users/oliviern/Uber/android/build/.bazelw_cache/91a99194e71852b0e089ca5cc1c48073/external/remote_java_tools_darwin/BUILD and referenced by '@rules_java//toolchains:ijar_prebuilt_binary_darwin'

@oliviernotteghem
Copy link

oliviernotteghem commented Jul 27, 2022

OK, there was 2 issues :

  1. WORKSPACE file is missing from zip file generate from //src:java_tools_zip (you can copy over manually the default one)
  2. --override_repository=java_tools=/tmp/java_tools should be --override_repository=remote_java_tools=/tmp/java_tools

After that, I can verify the files located at /tmp/java_tools are copied over to /external/remote_java_tools/java_tools and are used.

Seeing this error now (which can be temporary worked around by removing ErrorPronePlugin manually) :

Exception in thread "main" java.lang.NoSuchMethodError: 'com.google.common.collect.ImmutableMap com.google.common.collect.ImmutableMap$Builder.buildOrThrow()'
	at com.google.errorprone.scanner.ScannerSupplier.defaultSeverities(ScannerSupplier.java:64)
	at com.google.errorprone.scanner.ScannerSupplier.fromBugCheckerInfos(ScannerSupplier.java:85)
	at com.google.errorprone.scanner.BuiltInCheckerSuppliers.allChecks(BuiltInCheckerSuppliers.java:566)
	at com.google.errorprone.scanner.BuiltInCheckerSuppliers.errorChecks(BuiltInCheckerSuppliers.java:583)
	at com.google.devtools.build.buildjar.javac.plugins.errorprone.ErrorPronePlugin.<init>(ErrorPronePlugin.java:56)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.parse(BazelJavaBuilder.java:126)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.parseAndBuild(BazelJavaBuilder.java:76)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:65)

@cushon
Copy link
Author

cushon commented Dec 12, 2022

For posterity, I think the complete steps for java_tools are something like:

bazel build //src:java_tools_zip
mkdir /tmp/java_tools/
unzip -q -o -d /tmp/java_tools/ bazel-bin/src/java_tools.zip
touch /tmp/java_tools/WORKSPACE
bazel build --override_repository=remote_java_tools=/tmp/java_tools ...

And for java_tools_linux:

bazel build //src:java_tools_prebuilt_zip
mkdir /tmp/java_tools_linux/
unzip -q -o -d /tmp/java_tools_linux/ bazel-bin/src/java_tools_prebuilt.zip
touch /tmp/java_tools_linux/WORKSPACE

--override_repository=rules_java~7.3.2~toolchains~remote_java_tools_linux=/tmp/java_tools_linux

I think bzlmod changes mean it's necessary to override version (e.g. the ~7.3.2~ in the name), maybe there's a cleaner alternative for that.

@cushon
Copy link
Author

cushon commented Aug 9, 2024

Trying to do this again for bazelbuild/bazel#23246, the incantation for java_tools (I didn't need to modify architecture specific tools this time) was

bazel build --override_repository=rules_java++toolchains+remote_java_tools=/tmp/java_tools ...

To test a local build of turbine I also installed turbine to the local maven repo

mvn install

and updated Bazel to refer to it:

$ git diff MODULE.bazel
diff --git a/MODULE.bazel b/MODULE.bazel
index afabcbe5ff..73b938896c 100644
--- a/MODULE.bazel
+++ b/MODULE.bazel
@@ -113,7 +113,7 @@ maven.install(
         "com.google.http-client:google-http-client:1.42.0",
         "com.google.http-client:google-http-client-gson:1.42.0",
         "com.google.j2objc:j2objc-annotations:1.3",
-        "com.google.turbine:turbine:0.6.0",
+        "com.google.turbine:turbine:HEAD-SNAPSHOT",
         "com.guardsquare:proguard-base:jar:7.5.0",
         "com.ryanharter.auto.value:auto-value-gson-extension:1.3.1",
         "com.ryanharter.auto.value:auto-value-gson-factory:1.3.1",
@@ -199,6 +199,7 @@ maven.install(
     fail_if_repin_required = True,
     lock_file = "//:maven_install.json",
     repositories = [
+        "m2local",
         "https://repo1.maven.org/maven2",
     ],
     strict_visibility = True,

Running REPIN=1 bazel run @maven//:pin failed because mvn install doesn't write checksum files anymore, so I manually added the missing checksums with bash.

Suggestions for a better way I could have accomplished that are welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 P2 We'll consider to work on this in future. (Assignee optional)
Projects
None yet
Development

No branches or pull requests

3 participants