Skip to content

Commit

Permalink
Squashed commit of the following:
Browse files Browse the repository at this point in the history 
commit 647d979
Author: Ed Hennis <[email protected]>
Date:   Wed Jun 1 17:57:22 2022 -0400

    [FOLD] Restore the "cancel previous runs" action

commit 69130e9
Author: Shi Cheng <[email protected]>
Date:   Thu May 26 17:51:20 2022 +0800

    enable build for gcc8/clang8/macOS

commit 4d494f0
Author: Shi Cheng <[email protected]>
Date:   Thu May 5 15:11:34 2022 +0800

    change to manual trigger for macos pipeline

commit 78ae642
Author: Shi Cheng <[email protected]>
Date:   Thu May 5 14:42:14 2022 +0800

    run pipelines on self hosted runner

commit b414cd9
Author: Ed Hennis <[email protected]>
Date:   Wed Jun 1 18:59:34 2022 -0400

    [FOLD] Make NINJA_BUILD flag explicit on derived builds

commit eba6511
Author: Ed Hennis <[email protected]>
Date:   Wed Jun 1 17:38:47 2022 -0400

    [FOLD] Consolidate the special case linux jobs

commit f72b4e6
Author: Ed Hennis <[email protected]>
Date:   Wed Jun 1 15:19:00 2022 -0400

    [FOLD] Linux runner fixes:

    * Consolidate manual test runners
    * Continue gracefully if no cache in derived jobs
    * Retry unit tests up to 2 more times automatically

commit f3f922a
Author: Ed Hennis <[email protected]>
Date:   Tue May 31 20:19:30 2022 -0400

    [FOLD] Try to pull all the linux jobs into one workflow

    * Because self-hosted runners are more reliable, AND github actions now
      supports re-running only failed jobs

commit 17a6eb1
Author: Ed Hennis <[email protected]>
Date:   Tue May 31 19:32:52 2022 -0400

    [FOLD] Fix Roles test under IPv6

commit 47538ae
Author: Ed Hennis <[email protected]>
Date:   Fri May 27 21:04:42 2022 -0400

    [FOLD] Try to fix MacOS builds by ignoring warnings

commit b045a76
Author: Ed Hennis <[email protected]>
Date:   Thu May 26 15:29:00 2022 -0400

    [FOLD] Update "min cmake" test to use 3.16, which is now the min

commit 3e4a747
Author: Ed Hennis <[email protected]>
Date:   Thu May 26 15:13:51 2022 -0400

    [FOLD] Use updated executable name for reporting mode

commit 1f08975
Author: Ed Hennis <[email protected]>
Date:   Tue May 24 19:21:03 2022 -0400

    [FOLD] Fix unit test to use updated TER code

commit 7176bdd
Author: Ed Hennis <[email protected]>
Date:   Tue May 24 16:19:32 2022 -0400

    Revert "Add a test case demonstrating VS2019 constexpr char* problem:"

    This reverts commit 1a328ab.

commit e19e33a
Author: Edward Hennis <[email protected]>
Date:   Fri Oct 1 18:46:33 2021 -0400

    Revert "Add missing "deque" headers in a bunch of places"

    This reverts commit 46bbac2.

commit 80694f6
Author: Edward Hennis <[email protected]>
Date:   Thu Sep 30 11:23:07 2021 -0400

    fixup! [FOLD] Restructure MacOS caches

commit bc81cd9
Author: Edward Hennis <[email protected]>
Date:   Tue Sep 28 15:32:52 2021 -0400

    [FOLD] Restructure MacOS caches

commit 93337bf
Author: Edward Hennis <[email protected]>
Date:   Tue Sep 28 13:28:48 2021 -0400

    [FOLD] Update GH CI documentation

commit 95245bb
Author: Edward Hennis <[email protected]>
Date:   Fri Sep 10 12:06:59 2021 -0400

    [FOLD] Make the check tests manual on Github CI Windows

    * Don't restrict Windows test threads

commit 1dbe296
Author: Edward Hennis <[email protected]>
Date:   Wed Sep 1 15:09:02 2021 -0400

    [FOLD] Restructure the linux caches

    * Reuse the NIH src cache across configs
    * Store timestamp (*_stamp) and SQLite download folders in src cache
    * Break up docker image names to fix artifact naming

commit 9a4c23d
Author: Edward Hennis <[email protected]>
Date:   Wed Sep 1 13:06:29 2021 -0400

    [FOLD] Review feedback: Clarified a few points in the README

commit fd86f7a
Author: Edward Hennis <[email protected]>
Date:   Thu Jun 24 14:09:36 2021 -0400

    [FOLD] Github folder documentation

commit 0173363
Author: Edward Hennis <[email protected]>
Date:   Mon Jul 12 19:38:33 2021 -0400

    [FOLD] Move the build-action folder up a level

commit 8850a4e
Author: Edward Hennis <[email protected]>
Date:   Thu Jun 17 17:48:35 2021 -0400

    [FOLD] Improve Linux and Mac job naming

commit 7643040
Author: Edward Hennis <[email protected]>
Date:   Mon Jun 14 20:33:37 2021 -0400

    [FOLD] Pull files out of unity to reduce CI OOM issues

    * Also include a debugging step for failed unity builds

commit 5915782
Author: Edward Hennis <[email protected]>
Date:   Mon Jun 14 20:35:02 2021 -0400

    Revert "[FOLD] Reduce the unity file size for gcc-9 release builds"

    This reverts commit 7f8af40.

commit 5421b2f
Author: Edward Hennis <[email protected]>
Date:   Mon Jun 14 17:54:47 2021 -0400

    [FOLD] Run MacOS unit tests single thread

commit 5be6d44
Author: Edward Hennis <[email protected]>
Date:   Mon Jun 14 16:52:34 2021 -0400

    [FOLD] Remove inputs and outputs from build-action

    * All parameters must be passed in the env, like build-and-test.sh

commit 0441306
Author: Edward Hennis <[email protected]>
Date:   Mon Jun 14 13:35:37 2021 -0400

    [FOLD] (Tweak) Rename MacOS test runner job

commit 31befac
Author: Edward Hennis <[email protected]>
Date:   Mon Jun 14 13:30:14 2021 -0400

    [FOLD] Reduce the unity file size for gcc-9 release builds

commit e9c5dd6
Author: Edward Hennis <[email protected]>
Date:   Tue Jun 8 17:33:18 2021 -0400

    [FOLD] Review feedback: Cleanups:

    * Restructure job matrices to use env when there's one option
    * Remove image param from build-action
    * Limit Visual Studio test jobs to help make them more reliable

commit 4ed952a
Author: Edward Hennis <[email protected]>
Date:   Tue Jun 8 17:31:58 2021 -0400

    [FOLD] Change some "== false" comparisons to "!= true"

commit 98ed069
Author: Edward Hennis <[email protected]>
Date:   Tue Jun 8 17:04:38 2021 -0400

    [FOLD] Review feedback: change "post condition" to a "warning"

commit df47bc0
Author: Edward Hennis <[email protected]>
Date:   Fri Jun 4 16:17:07 2021 -0400

    [FOLD] Reenable the manual DatabaseShard test, fixed in 1.8.0-b3

commit 548e4f9
Author: Edward Hennis <[email protected]>
Date:   Fri May 28 18:17:29 2021 -0400

    [FOLD] review feedback (action):

    * Clean up cache space regardless of size
    * Improve clarity of some comments
    * Remove matrix.CMAKE_ADD where it's not used

commit 304fae6
Author: Edward Hennis <[email protected]>
Date:   Fri May 28 14:31:13 2021 -0400

    [FOLD] review feedback:

    * clean up cmake formatting
    * Remove redundant `install:` section in Travis config
    * Update cancel workflow action version and add a note about when it
      works

commit e12540b
Author: Edward Hennis <[email protected]>
Date:   Fri May 28 12:16:00 2021 -0400

    [FOLD] NOW what's going on with Windows builds:

    * Dump vc environment if parsing fails

commit b7b9826
Author: Edward Hennis <[email protected]>
Date:   Fri Nov 6 13:53:42 2020 -0500

    Build and test rippled in Github Actions:

    * VS19 debug builds DO NOT RUN TESTS. Known issue with those builds, but
      if it gets fixed, I want to be able to see those tests succeed.

commit 3836bcd
Author: Edward Hennis <[email protected]>
Date:   Mon May 17 17:51:22 2021 -0400

    Add a test case demonstrating VS2019 constexpr char* problem:

    * This intended to future-proof in two ways. 1) so we don't lose the
      institutional knowledge why we keep using VS2017. 2) if/when the issue
      is resolved, we'll know it.

commit bcafe63
Author: Edward Hennis <[email protected]>
Date:   Mon May 17 17:47:23 2021 -0400

    Env unit test RPC errors return a unique result:

    * telLOCAL_ERROR is not new, but it is not used anywhere in the
      transaction engine, unlike temINVALID. This will make those types of
      errors distinct and easier to diagnose.

commit fa29bee
Author: Edward Hennis <[email protected]>
Date:   Mon May 17 17:45:42 2021 -0400

    Add missing "deque" headers in a bunch of places

commit 3b9a6d9
Author: Edward Hennis <[email protected]>
Date:   Mon May 17 17:37:50 2021 -0400

    Get the list of manual tests from rippled:

    * Split that function into a separate file so it can be used in
      other contexts.

commit 533e459
Author: Edward Hennis <[email protected]>
Date:   Mon May 17 17:34:54 2021 -0400

    Improve Windows Visual Studio CMake build compatibility

commit 5377eaa
Author: Edward Hennis <[email protected]>
Date:   Mon May 10 13:27:12 2021 -0400

    Update version of vcpkg for Windows builds:

    * Include a warning message when a newer version is found
  • Loading branch information
@ximinez
ximinez committed Aug 9, 2024
1 parent c19a88f commit 5bfa08c
Show file tree
Hide file tree
Showing 22 changed files with 2,238 additions and 98 deletions.
239 changes: 239 additions & 0 deletions .github/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,239 @@
# Github configuration

This file provides a brief explanation of the organization of rippled's
Github configuration for actions, workflows, and templates.

## Templates

The `ISSUE_TEMPLATE` folder holds several files used to improve the
experience of creating an issue.
* `config.yml` configures external links to XRPL resources that are provided
when a user starts the process of creating a new issue. Currently, that
includes XRPL documentation and Ripple's bug bounty program.
* `bug_report.md` is the template used to create bug reports.
* `feature_request.md` is the template used to create feature requests.

Additionally, `pull_request_template.md` in this folder is the template
used when opening Pull Requests.

## Workflows

The `workflows` folder holds several files that describe Github Actions
workflows.

### Documentation

`doxygen.yml` builds and [publishes](http://ripple.github.io/rippled/) the
rippled developer documentation using `doxygen` whenever a new commit or
commits are published to the `develop` branch.

### Code formatting

`clang-format.yml` checks that source code is correctly formatted. It runs
all rippled source files through `clang-format`, and fails if any changes
are found. If it fails, it publishes a patch file as an artifact that the
developer can apply to their branch, as well as instructions on how to use
it.

### Levelization checking

`levelization.yml` checks for changes in source code levelization, and
raises an alert if any changes are found (good or bad). See [the levelization
documentation](../Builds/levelization/README.md) for more information.

### Continuous integration

For some simplicity the continuous
integration tasks were split into three different workflows: one job each for
Linux, MacOS, and Windows. One file describes one workflow.

The workflows are generally built using some or all of the following stages:

1. Build and/or download common dependencies, and store them in a cache
intended to be shared by later stages. For example, boost, vcpkg, or the
external projects required by our `cmake` configuration.
2. Build and test "base" configurations. A base configuration is one that
builds `rippled` using default settings other than compiler,
debug/release, and unity/non-unity.
3. Use the cache from the base configuration to build and usually test a
special case configuration (e.g. Coverage or Reporting mode).
4. Use the `rippled` "base" artifact to run a non-default test scenario (e.g.
manual tests, or IPv6 tests).

Every build and test job, whether base or special case, will store some of
the intermediate build-results (e.g. NIH cache, `ccache` folders, vcpkg
libraries) in a job-specific cache. Additionally, each build and test job
will upload two artifacts: The cmake logs, and the `rippled` executable built
by the job.

#### Linux

1. `linux-general.yml` stages:
1. Pre-downloads dependencies into "NIH" ("Not Invented Here") caches.
2. Builds and tests `rippled` using all of the combinations of the
following.
* gcc-8, gcc-9, clang-8, clang-9, and clang-10
* Debug and Release
* unity and non-unity
3. Special cases:
a. For clang-8, Debug, unity, uses the cache from the base stage to
build and test the following special configurations:
* Reporting
* Coverage
b. For clang-8, Release, unity, uses the cache from the base stage to
build and test the following special configurations:
* Address sanitizer (asan)
* Undefined behavior sanitizer (ubsan)
* A thread sanitizer (tsan) job is defined, but disabled, because it
currently fails to run.
c. For gcc-8, Debug unity, uses the cache from the base stage to build
and test the following special configurations:
* Coverage
* Non-static
* Non-static with shared libraries
* Makefile (instead of Ninja)
* Minimum cmake version
* The separate
[`validator-keys`](https://github.com/ripple/validator-keys-tool)
application
4. Manual tests:
a. For gcc-8, Debug unity, uses the `rippled` artifact from the base job
to run all of the manual
tests defined in `rippled` except a few that use too much memory to
run on Github's hosts.
b. For gcc-8, Release, unity, uses the `rippled` artifact from the base
job to run all of the manual
tests defined in `rippled` except a few that use too much memory to
run on Github's hosts.

#### MacOS

6. `macos.yml` stages:
1. Pre-downloads dependencies into an "NIH" ("Not Invented Here") cache,
**and** builds Boost into a separate cache if not already cached. Two
caches are used because Boost changes very rarely, and is not modified
by the `rippled` build process, so it can be used across many
different jobs without conflict, and saving space.
2. Builds and tests `rippled` using:
* Debug and Release
3. There is no stage 3 in this workflow.
4. Uses the Debug `rippled` artifact from the base job to run the IPv6
tests using the `--unittest-ipv6` command line parameter.

#### Windows

7. `windows.yml` stages:
1. Pre-downloads dependencies into an "NIH" ("Not Invented Here") cache,
**and** builds vcpkg library dependencies and boost into separate
caches if not already cached. Three caches are used because the
vcpkg and boost libraries change very rarely, and are not modified
by the `rippled` build process, so they can be used across many
different jobs without conflict, and avoid duplication across jobs.
2. Builds and tests `rippled` using the following configurations.
(Note that MSVC 2019 Debug builds do not run tests by
default due to unresolved issues with how it handles `static
constexpr char*` variables. The tests can be forced to run by
including "ci_run_win" in the git commit message.)
* Ninja generator, MSVC 2017, Debug, unity
* Ninja generator, MSVC 2017, Release, unity
* Ninja generator, MSVC 2019, Debug, unity (without tests)
* Ninja generator, MSVC 2019, Release, unity
* Visual Studio 2019 generator, MSVC 2017, Debug, non-unity
* Visual Studio 2019 generator, MSVC 2017, Release, non-unity
* Visual Studio 2019 generator, MSVC 2019, Release, non-unity
3. There is no stage 3 in this workflow.
4. There is no stage 4 in this workflow.

##### Cacheing

###### Strategy

[Github Actions caches](https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows)
are immutable once written, and use explicit key names for access. Caches can
be shared across workflows, and can match partial names by prefix. They can
also access caches created by base and default branches, and to the forked
from (parent) repository, but not across arbitrary forks, forked to (child)
repositories, and distinct branches. Finally, they have a relatively short
expiration time (7 days), and a relatively small size limit (5Gb).

The caching policies used by these workflows attempt to take advantage of
these properties to save as much time as possible when building, while
minimizing space when feasible. There is almost certainly room for
improvement.

Thus, for example, the `linux-general.yml` workflow downloads the "NIH"
dependencies into a single cache (per docker image). All of the source and
git timestamp files for these dependencies are stored in a single folder,
which is reused across jobs, and *not* duplicated across job-specific caches.

Each "base" job stores dependency build output along with `ccache` output.
This significantly speeds up subsequent builds of the same
base job.

Once the base job is done, any "special case" jobs (e.g. Coverage, address
sanitizer) use their base job's cache to initialize their specific caches.
This further helps reduce duplicated effort, which helps speed up those jobs.

Finally, most caches build their cache key using values from the job
configuration and some components based on the hashes of the `cmake`
configuration, the rippled source code, and the workflow config itself.

To pull it all together with an example, the base job in
`linux-clang8-release.yml` might have a cache key that looks like (hashes
abbreviated for readability and simplicity):
* `Linux-ecbd-clang-8-Release-ON-base-40b1-5fec-a88b`
Once that job finishes, the "asan" job's cache key would look like:
* `Linux-ecbd-clang-8-Release-ON-asan-40b1-5fec-a88b`
* It would be initialized using `Linux-ecbd-clang-8-Release-ON-base-40b1-5fec-a88b`.

Once the whole workflow finishes, the developer makes a change to source and
pushes a new commit. The new cache key might look like.
* `Linux-ecbd-clang-8-Release-ON-base-abcd-1234-a88b`
No cache with that key is found, so it looks for
* `Linux-ecbd-clang-8-Release-ON-base-abcd-1234`
* `Linux-ecbd-clang-8-Release-ON-base-abcd`
* `Linux-ecbd-clang-8-Release-ON-base`
That last prefix matches the cache from the previous run
(`Linux-ecbd-clang-8-Release-ON-base-40b1-5fec-a88b`), and initializes
with that. Chances are that most of that cache will be useful to that new
build, and will cut the build time significantly.

Once the base job finishes, the "asan" job's cache key would be:
* `Linux-ecbd-clang-8-Release-ON-asan-abcd-1234-a88b`
And would initialize from the just-finished
`Linux-ecbd-clang-8-Release-ON-base-abcd-1234-a88b`

The components are organized in the following order
* Operating system: Caches aren't useful to be shared across OSes
* Hashes of the `cmake` config: Any changes to the `cmake` config can have
significant changes on the way that code is organized, dependencies are
organized, dependency folders are organized, etc., which would render the
caches incompatible. So to be safe, caches with different `cmake` configs
can never be reused.
* Additionally, this hash includes the file
`.github/workflows/cache-invalidate.txt`. This file can be manually
changed to force new builds to start with fresh caches in case some
unforseen change causes the build to fail with a reused cache.
* Compiler
* Build type (Debug/Release)
* Unity flag
* Job name
* Hash of all the header files under `src`: Because changing one header file
is more likely to affect a bunch of different object files, when this
changes, it invalidates more.
* Hash of all the source files under `src`, including headers: Changing any
source file is going to generate a new build. But with the same header
hash, a build cache that is likely to be very similar can be reused.
* Hash of the workflow instructions (the yml file, and the
`build-action/action.yml` if appropriate). If the workflow is changed
without changing any of the source, a new cache may be needed, but it can
be seeded with a previous build of the same source code.


## Action

The `build-action` folder holds an `action.yml` that is used by all of the
Linux workflows to do the actual build without tons of duplication.
Unfortunately, not all types of commands can by used in an action, so there
is still some boilerplate required in each job: checkout, cache, artifacts,
plus any steps that get displayed in a separate section in the Github UI.
82 changes: 82 additions & 0 deletions .github/build-action/action.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
name: 'Build and Test'
description: 'Build and Test rippled or an alternate target'
# Pre-condition: The environment has all the necessary base
# build variables set
# WARNING: This cache cannot be used to build with a different compiler.
runs:
using: "composite"
steps:
- name: Dump-environment
shell: bash
run: |
env
set | grep =
- name: Install ccache
shell: bash
run: |
if ! type -a ccache
then
apt install ccache
fi
- name: Reset ccache stats
shell: bash
run: |
ccache -s
# Reset the stats so only this build counts
ccache -z
- name: Disk space
shell: bash
run: |
df -h
if [[ -d "${CACHE_DIR}" ]]
then
du -sh ${CACHE_DIR}
fi
if [[ -d "${CCACHE_DIR}" ]]
then
du -sh ${CCACHE_DIR}
fi
find ${NIH_SRC_PATH} ${NIH_CACHE_ROOT} -maxdepth 2 \
-type d -exec du -sh {} \; || true
- name: Build and test
shell: bash
run: |
export NUM_PROCESSORS=${NUM_PROCESSORS:-$( nproc )}
echo NUM_PROCESSORS is ${NUM_PROCESSORS}
if [ "${BUILD_TYPE}" == "Release" ]
then
export CMAKE_EXTRA_ARGS+=" -Dassert=ON"
fi
# Set and/or override some environment variables based on
# inputs to the action.
if [ "${CMAKE_ADD}" != "" ] ; then export CMAKE_EXTRA_ARGS+=" ${CMAKE_ADD}"; fi
# Do the build
bin/ci/ubuntu/build-and-test.sh
- name: Debug output
shell: bash
run: |
if [ -e build ] ; then find build \( -name CMakeOutput.log -o -name CMakeError.log \) -ls -exec cat {} \; ; fi
df -h
if [[ -d "${CACHE_DIR}" ]]
then
du -sh ${CACHE_DIR}
fi
if [[ -d "${CCACHE_DIR}" ]]
then
du -sh ${CCACHE_DIR}
fi
if [[ -d "${NIH_CACHE_ROOT}" ]]
then
find ${NIH_CACHE_ROOT} -maxdepth 3 \( -iname src -prune -o \
-type d -exec du -sh {} \; \)
fi
- name: Post-run ccache stats
shell: bash
run: ccache -s
12 changes: 12 additions & 0 deletions .github/workflows/cache-invalidate.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
This file is used to generate part of the cache key used in the
CI-related workflows.
The contents of this file are ABSOLUTELY MEANINGLESS.
What matters is that any time the caches need to be forced
to invalidate, changing this file will be sufficient.
For convenience, a number is provided at the bottom of the file,
which can be incremented to make things simple.
If there is ever a merge conflict, simply choose the larger number.
However, to reduce the risk of conflicts, avoid including changes
to this file in any Pull Requests, unless necessary

1
Loading

0 comments on commit 5bfa08c

Please sign in to comment.