Merge branch 'iceberg-extra-props' into iceberg-extra-props

.github/DEVELOPMENT.md

@@ -1,48 +1,28 @@
 # Development
 
-Developers should read [the development section of the website](https://trino.io/development),
-which covers thing like development philosophy and contribution process.
+Learn about development for all Trino organization projects:
 
-More information about the writing and building the documentation can
-be found in the [docs module](../docs).
+* [Vision](https://trino.io/development/vision)
+* [Contribution process](https://trino.io/development/process#contribution-process)
+* [Pull request and commit guidelines](https://trino.io/development/process#pull-request-and-commit-guidelines-)
+* [Release note guidelines](https://trino.io/development/process#release-note-guidelines-)
+
+Further information in the [development section of the
+website](https://trino.io/development) includes different roles, like
+contributors, reviewers, and maintainers, related processes, and other aspects.
+
+See [the Trino developer guide](https://trino.io/docs/current/develop.html) for
+information about the SPI, implementing connectors and other plugins plugins,
+the client protocol, writing tests and other lower level details.
+
+More information about writing and building the documentation can be found in
+the [docs module](../docs).
 
-* [Commits and pull requests](#commits-and-pull-requests)
 * [Code style](#code-style)
 * [Additional IDE configuration](#additional-ide-configuration)
 * [Building the Web UI](#building-the-web-ui)
 * [CI pipeline](#ci-pipeline)
 
-## Commits and pull requests
-
-### Format Git commit messages
-
-When writing a Git commit message, follow these [guidelines](https://chris.beams.io/posts/git-commit/).
-
-### Git merge strategy
-
-Pull requests are usually merged into `master` using the  [`rebase and merge`](https://docs.github.com/en/github/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/about-pull-request-merges#rebase-and-merge-your-pull-request-commits) strategy.
-
-A typical pull request should strive to contain a single logical change (but not
-necessarily a single commit). Unrelated changes should generally be extracted
-into their own PRs.
-
-If a pull request does consist of multiple commits, it is expected that every
-prefix of it is correct. That is, there might be preparatory commits at the 
-bottom of the stack that don't bring any value by themselves, but none of the
-commits should introduce an error that is fixed by some future commit. Every
-commit should build and pass all tests.
-
-Commit messages and history are also important, as they are used by other
-developers to keep track of the motivation behind changes. Keep logical diffs
-grouped together in separate commits, and order commits in a way that explains
-the progress of the changes. Rewriting and reordering commits may be a necessary
-part of the PR review process as the code changes. Mechanical changes (like
-refactoring and renaming)should be separated from logical and functional
-changes. E.g. deduplicating code or extracting helper methods should happen in a
-separate commit from the commit where new features or behavior is introduced.
-This makes reviewing the code much easier and reduces the chance of introducing
-unintended changes in behavior.
-
 ## Code Style
 
 We recommend you use IntelliJ as your IDE. The code style template for the
@@ -179,6 +159,7 @@ Disable the following inspections:
 
 - ``Java | Performance | Call to 'Arrays.asList()' with too few arguments``,
 - ``Java | Abstraction issues | 'Optional' used as field or parameter type``,
+- ``Java | Code style issues | Local variable or parameter can be 'final'``,
 - ``Java | Data flow | Boolean method is always inverted``.
 
 Update the following inspections:
@@ -235,17 +216,17 @@ folder). You must have [Node.js](https://nodejs.org/en/download/) and
 [Yarn](https://yarnpkg.com/en/) installed to execute these commands. To update
 this folder after making changes, simply run:
 
-    yarn --cwd core/trino-main/src/main/resources/webapp/src install
+    yarn --cwd core/trino-web-ui/src/main/resources/webapp/src install
 
 If no Javascript dependencies have changed (i.e., no changes to `package.json`),
 it is faster to run:
 
-    yarn --cwd core/trino-main/src/main/resources/webapp/src run package
+    yarn --cwd core/trino-web-ui/src/main/resources/webapp/src run package
 
 To simplify iteration, you can also run in `watch` mode, which automatically
 re-compiles when changes to source files are detected:
 
-    yarn --cwd core/trino-main/src/main/resources/webapp/src run watch
+    yarn --cwd core/trino-web-ui/src/main/resources/webapp/src run watch
 
 To iterate quickly, simply re-build the project in IntelliJ after packaging is
 complete. Project resources will be hot-reloaded and changes are reflected on
@@ -257,30 +238,22 @@ Trino aims for frequent releases, generally once per week. This is a goal but
 not a guarantee, as critical bugs may lead to a release being pushed back or
 require an extra emergency release to patch the issue.
 
-At the start of each release cycle, a GitHub issue is filed and pinned to track
-all necessary release notes. For example, see [the issue for Trino 395](https://github.com/trinodb/trino/issues/13913).
-In addition, a release notes pull request is updated and maintained throughout
-the week, tracking all merged commits to ensure every change is properly
-documented and noted. This uses the [release note template](../docs/release-template.md),
-with changes in each section arranged to have new features first, performance
-improvements second, and bugfixes third. See [the release notes for 395](https://github.com/trinodb/trino/pull/13975)
-as an example.
-
-Once it is time to release, the release process is kicked off. A code freeze is
-announced on the Trino Slack in the #releases channel, and then a maintainer
-utilizes the [release scripts](https://github.com/trinodb/release-scripts) to
-update Trino to the next version.
-
-## CI pipeline
-
-### Maven offline mode
-
-The CI pipeline runs most Maven commands in offline mode to avoid dependency on Maven Central.
-
-Before calling Maven a job should execute the `./.github/actions/setup` compound action to
-download all the dependencies it will require. 
-
-It uses two complementary tools to detect and download Maven dependencies, 
-but some plugin dependencies are not expressed in the tree hierarchy.
-
-These dynamic dependencies must be listed as `<DynamicDependency>` entries in the root `pom.xml`.
+At the start of each release cycle, a release notes pull request (PR) is started
+and maintained throughout the week, tracking all merged PRs to ensure every
+change is properly documented and noted.
+
+The PR uses the [release note template](../docs/release-template.md) and follows
+the [release notes
+guidelines](https://trino.io/development/process#release-note) to use and
+improve the proposed release note entries from the merged PRs. When necessary,
+documentation and clarification for the release notes entries is requested from
+the merging maintainer and the contributor.
+
+See [the release notes for
+455](https://github.com/trinodb/trino/pull/23096) as an example.
+
+Once it is time to release, the release notes PR is merged and the process is
+kicked off. A code freeze is announced on the Trino Slack in the #releases
+channel, and then a maintainer utilizes the [release
+scripts](https://github.com/trinodb/release-scripts) to update Trino to the next
+version.

.github/actions/compile-commit/action.yml

@@ -15,16 +15,15 @@ runs:
         echo "tree-hash=${TREE_HASH}" >> $GITHUB_OUTPUT
     - name: Cache compile-commit action success
       id: check-compile-commit-success
-      uses: actions/cache@v3
+      uses: actions/cache@v4
       with:
         path: compile-commit-success.txt
         # use tree hashes instead of commit hashes to reuse cache when changes in commits are reorganized without changing the workspace
         key: compile-commit-success-${{ github.event.pull_request.head.repo.full_name }}-${{ steps.repo-hash.outputs.tree-hash }}
     - uses: ./.github/actions/setup
-      if: steps.check-compile-commit-success.outputs.cache-hit != 'true'
       with:
-        # Dependencies are downloaded explicitly below, after MAVEN has been set here. This allows modifications to the MAVEN setup with a PR.
-        download_dependencies: false
+        cleanup-node: true
+      if: steps.check-compile-commit-success.outputs.cache-hit != 'true'
     - name: Check if a specified commit compiles
       shell: bash
       if: steps.check-compile-commit-success.outputs.cache-hit != 'true'
@@ -35,18 +34,16 @@ runs:
         # It's important that these values are NOT passed as parameters, because then their values would always be taken from PR HEAD
         # -------
         # allow overriding Maven command
-        export MAVEN="./mvnw --offline"
-        # maven.wagon.rto is in millis, defaults to 30m
-        MAVEN_INSTALL_OPTS="-Xmx3G -XX:+ExitOnOutOfMemoryError -Dmaven.wagon.rto=60000"
+        export MAVEN="./mvnw"
+        MAVEN_INSTALL_OPTS="-Xmx3G -XX:+ExitOnOutOfMemoryError"
         MAVEN_COMPILE_COMMITS="-B --strict-checksums --quiet -T 1C -DskipTests -Dmaven.source.skip=true -Dair.check.skip-all=true -Dmaven.javadoc.skip=true --no-snapshot-updates --no-transfer-progress -pl !:trino-server-rpm"
         export MAVEN_GIB="-P gib -Dgib.referenceBranch=refs/remotes/origin/${{ inputs.base_ref }}"
         RETRY=.github/bin/retry
         # -------
 
         # For building with Maven we need MAVEN_OPTS to equal MAVEN_INSTALL_OPTS
         export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
-        ./.github/bin/download-maven-dependencies.sh
-        $MAVEN package \
+        $MAVEN install \
           ${MAVEN_COMPILE_COMMITS} `# defaults, kept in sync with ci.yml` \
           -Dair.check.skip-all=false -Dair.check.skip-basic=true -Dair.check.skip-extended=true -Dair.check.skip-checkstyle=false \
           ${MAVEN_GIB}

.github/actions/process-test-results/action.yml

@@ -0,0 +1,68 @@
+name: "process-test-results"
+description: "Upload test results and annotate CI jobs"
+inputs:
+  artifact-name:
+    description: "Artifact name, must be unique across all jobs"
+    required: false
+    default: ${{ github.job }}
+  has-failed-tests:
+    description: "If any test in the calling job has failed"
+    required: true
+  upload-heap-dump:
+    description: "Should any heap dump files (hprof) be uploaded"
+    required: false
+    default: false
+
+runs:
+  using: composite
+  steps:
+  - name: Upload test results
+    uses: actions/upload-artifact@v4
+    if: format('{0}', inputs.has-failed-tests) == 'true'
+    with:
+      name: results ${{ inputs.artifact-name }}
+      if-no-files-found: 'ignore'
+      path: |
+        **/target/surefire-reports
+        **/target/checkstyle-*
+        testing/trino-product-tests/target/*
+        logs/*
+      retention-days: 5
+  - name: Upload test report
+    uses: actions/upload-artifact@v4
+    # Always upload the test report for the upload-test-results.yml workflow,
+    # but only the single XML file to keep the artifact small
+    with:
+      # Name prefix is checked in the `Upload test results` workflow
+      name: test report ${{ inputs.artifact-name }}
+      if-no-files-found: 'ignore'
+      path: |
+        **/surefire-reports/TEST-*.xml
+        testing/trino-product-tests/target/reports/**/testng-results.xml
+      retention-days: 5
+  - name: Upload heap dump
+    uses: actions/upload-artifact@v4
+    if: format('{0}', inputs.upload-heap-dump) == 'true'
+    with:
+      name: heap dump ${{ inputs.artifact-name }}
+      if-no-files-found: 'ignore'
+      path: |
+        **/*.hprof
+      retention-days: 14
+  - name: Process unit and integration test reports
+    uses: trinodb/github-actions/action-surefire-report@b63800bedfbc7ab1ff2e5fe7eaecf5ab82ce6a70
+    if: format('{0}', inputs.has-failed-tests) == 'true'
+    with:
+      # this workflow should never fail, as it is not a quality gateway
+      fail_if_no_tests: false
+      # don't create or update checks, only annotate current job
+      skip_publishing: true
+  - name: Process Product Test reports
+    uses: starburstdata/action-testng-report@f245422953fb97ec5075d07782a1b596124b7cc4
+    if: format('{0}', inputs.has-failed-tests) == 'true'
+    with:
+      # this workflow should never fail, as it is not a quality gateway
+      fail_if_empty: false
+      github_token: ${{ github.token }}
+      # don't create or update checks, only annotate current job
+      skip_publishing: true

.github/actions/setup/action.yml

@@ -3,16 +3,13 @@ description: "Verify checked out commits and setup Java"
 inputs:
   java-version:
     description: "Java version to setup"
-    default: 17
+    default: 23
   cache:
     description: "Cache Maven repo (true/false/restore)"
-    default: true
+    default: restore
   cleanup-node:
     description: "Clean up node (true/false) to increase free disk space"
     default: false # Disabled by default as it adds ~4 minutes of test runtime. Should be enabled case by case.
-  download_dependencies:
-    description: "Download all Maven dependencies so Maven can work in offline mode"
-    default: true
 
 runs:
   using: composite
@@ -40,33 +37,60 @@ runs:
         if: ${{ format('{0}', inputs.cleanup-node) == 'true' }}
         shell: bash
         run: ./.github/bin/free-disk-space.sh
-      - uses: actions/setup-java@v3
+      - uses: actions/setup-java@v4
         if: ${{ inputs.java-version != '' }}
         with:
           distribution: 'temurin' # use same JDK distro as in Trino docker images
           java-version: ${{ inputs.java-version }}
+      - name: Get NodeJS version from parent pom
+        shell: bash
+        run: |
+          node_version=$(grep -m 1 '<dep.frontend-node.version>' "pom.xml" | sed -n 's/.*<dep.frontend-node.version>\(.*\)<\/dep.frontend-node.version>.*/\1/p')
+          if [ -n "$node_version" ]; then
+            echo "node version is $node_version"
+            echo "NODE_VERSION=$node_version" >>${GITHUB_ENV}
+            exit 0
+          fi
+      - name: Cache and Restore local NodeJS
+        id: cache-nodejs
+        if: ${{ format('{0}', inputs.cache) == 'true' && env.NODE_VERSION != '' }}
+        uses: actions/cache@v4
+        with:
+          # Note: must be same set of paths as for cache:restore mode
+          path: .node
+          key: ${{ runner.os }}-nodejs-${{ env.NODE_VERSION }}
+          restore-keys: |
+            ${{ runner.os }}-nodejs-
+      - name: Restore local NodeJS
+        id: cache_restore-nodejs
+        if: ${{ format('{0}', inputs.cache) == 'restore' && env.NODE_VERSION != '' }}
+        uses: actions/cache/restore@v4
+        with:
+          # Note: must be same set of paths as for cache:true mode
+          path: .node
+          key: ${{ runner.os }}-nodejs-${{ env.NODE_VERSION }}
+          restore-keys: |
+            ${{ runner.os }}-nodejs-
       - name: Cache and Restore local Maven repo
         id: cache
         if: ${{ format('{0}', inputs.cache) == 'true' }}
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
+          # Note: must be same set of paths as for cache:restore mode
           path: ~/.m2/repository
           key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
           restore-keys: |
             ${{ runner.os }}-maven-
       - name: Restore local Maven repo
         id: cache_restore
         if: ${{ format('{0}', inputs.cache) == 'restore' }}
-        uses: actions/cache/restore@v3
+        uses: actions/cache/restore@v4
         with:
+          # Note: must be same set of paths as for cache:true mode
           path: ~/.m2/repository
           key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
           restore-keys: |
             ${{ runner.os }}-maven-
-      - name: Fetch any missing dependencies
-        shell: bash
-        if: ${{ format('{0}', inputs.download_dependencies) == 'true' }}
-        run: ./.github/bin/download-maven-dependencies.sh
       - name: Configure Problem Matchers
         if: ${{ inputs.java-version != '' }}
         shell: bash

.github/actions/update-check/action.yml

@@ -23,7 +23,7 @@ inputs:
 runs:
   using: composite
   steps:
-    - uses: actions/github-script@v6
+    - uses: actions/github-script@v7
       id: update-check-run
       if: inputs.check_name != '' && inputs.conclusion != ''
       env:
@@ -61,7 +61,7 @@ runs:
             conclusion: process.env.conclusion
           });
           return result;
-    - uses: actions/github-script@v6
+    - uses: actions/github-script@v7
       id: comment
       if: always()
       env: