Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add remote persistent worker support #787

Open
wants to merge 24 commits into
base: main
Choose a base branch
from

Commits on Oct 29, 2024

  1. Configuration menu
    Copy the full SHA
    a62ed89 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    003b58d View commit details
    Browse the repository at this point in the history
  3. Fix clippy error

    aherrmann committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    0d4f1ea View commit details
    Browse the repository at this point in the history
  4. fix run_display test

    aherrmann committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    872064c View commit details
    Browse the repository at this point in the history
  5. Use WorkerRunInfo.exe for non-worker mode

    `WorkerRunInfo` has two fields `worker` and `exe` for the persistent worker command or the
    non-worker mode command respectively. This commit changes the remote persistent worker example to
    use these appropriately to distinguish between worker and non-worker mode execution.
    aherrmann committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    386ee26 View commit details
    Browse the repository at this point in the history
  6. Use WorkerRunInfo.worker for remote persistent worker

    The Buck2 Rust implementation needs to use the worker command instead of the non-worker command for
    remote persistent worker execution mode.
    aherrmann committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    dbb8d6a View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    2e448e0 View commit details
    Browse the repository at this point in the history
  8. Create proto_python_library rule

    We want to use a Buck2 managed hermetic Python toolchain. However, Python binaries generated by such
    a toolchain are sensitive to `PWD` and don't work in other working directories than the repository
    root. Unfortunately, `genrule` changes directory and is hence incompatible with such Python
    binaries. This commit defines a dedicated rule to invoke protoc without changing the working
    directory.
    aherrmann committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    e427c7a View commit details
    Browse the repository at this point in the history
  9. Hermetic Python toolchain and Buck2 installed packages

    The previous setup used Nix to provide a Python toolchain and packages in a reproducible fashion.
    However, this requires dedicated remote worker images with the Nix store paths pre-populated which
    complicates the setup for testing on Buck2 CI.
    
    Using a hermetic Python toolchain avoids these issues and works on the standard remote execution
    image.
    aherrmann committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    5f98607 View commit details
    Browse the repository at this point in the history
  10. Use proto_python_library rule

    Remove the old genrule targets to generate the Python gRPC/protobuf bindings.
    aherrmann committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    9890623 View commit details
    Browse the repository at this point in the history
  11. Standard BuildBuddy worker image

    Without Nix it is no longer required to use a custom remote worker image.
    aherrmann committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    cc593fa View commit details
    Browse the repository at this point in the history
  12. Remove the Nix flake

    aherrmann committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    7a7d45d View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    e4e6667 View commit details
    Browse the repository at this point in the history
  14. Shrink number of test targets

    It's useful to test multiple builds in parallel to catch potential issues related parallel worker
    requests. However, we also don't need to be excessive in the number of tests to not unnecessarily
    waste resources.
    aherrmann committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    b4376e9 View commit details
    Browse the repository at this point in the history
  15. fix README instructions

    aherrmann committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    88a399a View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    3f88885 View commit details
    Browse the repository at this point in the history
  17. Test persistent worker example on CI

    Requires a repository secret to be set up for the BuildBuddy API key named `BUILDBUDDY_API_KEY`.
    aherrmann committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    0a7f496 View commit details
    Browse the repository at this point in the history
  18. fix typo

    aherrmann committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    5d9b7cb View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2024

  1. Configuration menu
    Copy the full SHA
    722d948 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e503285 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    3ea6a03 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    6cf252e View commit details
    Browse the repository at this point in the history
  5. Enable manual pipeline runs

    aherrmann committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    7181c3d View commit details
    Browse the repository at this point in the history
  6. Fix annotations file path

    aherrmann committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    62b3f8a View commit details
    Browse the repository at this point in the history