Skip to content

Commit

Permalink
Add patched recipe for SOCI: (XRPLF#4510)
Browse files Browse the repository at this point in the history
SOCI is the C++ database access library. The SOCI recipe was updated in
Conan Center Index (CCI), and it breaks for our choice of options. This
breakage occurs when you build with a fresh Conan cache (e.g. when you
submit a PR, or delete `~/.conan/data`).

* Add a custom Conan recipe for SOCI v4.0.3
* Update dependency building to handle exporting and installing Snappy
  and SOCI
  * Fix workflows to use custom SOCI recipe
* Update BUILD.md to include instruction for exporting the SOCI Conan
  recipe:
  * `conan export external/soci soci/4.0.3@`

This solution has been verified on Ubuntu 20.04 and macOS.

Context:

* There is a compiler error that the `sqlite3.h` header is not available
  when building soci.
* When package B depends on package A, it finds the pieces it needs by
  importing the Package Configuration File (PCF) that Conan generates
  for package A.
  * Read the CMake written by package B to check that it is importing
    the PCF correctly and linking its exports correctly.
  * Since this can be difficult, it is often more efficient to check
    https://github.com/conan-io/conan-center-index/issues for package B
    to see if anyone else has seen a similar problem.
  * One of the issues points to a problem area in soci's CMake. To
    confirm the diagnosis, review soci's CMake (after any patches are
    applied) in the Conan build directory `build/$buildId/src/`.
  * Review the Conan-generated PCF in
    `build/$buildId/build/$buildType/generators/`.
  * In this case, the problem was likely (re)introduced by
    conan-io/conan-center-index#17026
* If there is a problem in the source or in the Conan recipe, the
  fastest fix is to copy the recipe and either:
  * Add a source patch to fix any problems in the source.
  * Change the recipe to fix any problems in the recipe.
* In this case, this can be done by finding soci's Conan recipe at
  https://github.com/conan-io/conan-center-index/tree/master/recipes/soci
  and then copying the `all` directory as `external/$packageName` in our
  project. Then, make any changes.
  * Test packages can be removed from the recipe folder as they are not
    needed.
  * If adding a patch in the `patches` directory, add a description for
    it to `conandata.yml`.
  * Since `conanfile.py` has no `version` property on the recipe class,
    builders need to pass a version on the command line (like they do
    for our `snappy` recipe).
* Add an example command to `BUILD.md`.

Future work: It may make sense to refer to recipes by revision, by
checking in a lockfile.
  • Loading branch information
thejohnfreeman authored and seelabs committed Apr 26, 2023
1 parent c3da03c commit ed9f669
Show file tree
Hide file tree
Showing 9 changed files with 879 additions and 0 deletions.
33 changes: 33 additions & 0 deletions .github/actions/build/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: build
inputs:
generator:
default: null
configuration:
required: true
cmake-args:
default: null
# An implicit input is the environment variable `build_dir`.
runs:
using: composite
steps:
- name: dependencies
uses: ./.github/actions/dependencies
with:
configuration: ${{ inputs.configuration }}
- name: configure
shell: bash
run: |
cd ${build_dir}
cmake \
${{ inputs.generator && format('-G {0}', inputs.generator) || '' }} \
-DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake \
-DCMAKE_BUILD_TYPE=${{ inputs.configuration }} \
${{ inputs.cmake-args }} \
..
- name: build
shell: bash
run: |
cmake \
--build ${build_dir} \
--config ${{ inputs.configuration }} \
--parallel ${NUM_PROCESSORS:-$(nproc)}
23 changes: 23 additions & 0 deletions .github/actions/dependencies/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: dependencies
inputs:
configuration:
required: true
# An implicit input is the environment variable `build_dir`.
runs:
using: composite
steps:
- name: export custom recipes
shell: bash
run: |
conan export external/snappy snappy/1.1.9@
conan export external/soci soci/4.0.3@
- name: install dependencies
shell: bash
run: |
mkdir ${build_dir}
cd ${build_dir}
conan install \
--output-folder . \
--build missing \
--settings build_type=${{ inputs.configuration }} \
..
111 changes: 111 additions & 0 deletions .github/workflows/nix.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
name: nix
on: [push, pull_request]

jobs:

dependencies:
strategy:
fail-fast: false
matrix:
platform:
- linux
compiler:
- gcc
- clang
configuration:
- Debug
- Release
include:
- compiler: gcc
profile:
version: 11
cc: /usr/bin/gcc
cxx: /usr/bin/g++
- compiler: clang
profile:
version: 14
cc: /usr/bin/clang-14
cxx: /usr/bin/clang++-14
runs-on: [self-hosted, heavy]
container: thejohnfreeman/rippled-build-ubuntu:12e19cd9034b
env:
build_dir: .build
steps:
- name: check environment
run: |
echo ${PATH} | tr ':' '\n'
conan --version
cmake --version
env
- name: configure Conan
run: |
conan profile new default --detect
conan profile update settings.compiler.cppstd=20 default
conan profile update settings.compiler=${{ matrix.compiler }} default
conan profile update settings.compiler.version=${{ matrix.profile.version }} default
conan profile update settings.compiler.libcxx=libstdc++11 default
conan profile update env.CC=${{ matrix.profile.cc }} default
conan profile update env.CXX=${{ matrix.profile.cxx }} default
conan profile update conf.tools.build:compiler_executables='{"c": "${{ matrix.profile.cc }}", "cpp": "${{ matrix.profile.cxx }}"}' default
- name: checkout
uses: actions/checkout@v3
- name: dependencies
uses: ./.github/actions/dependencies
with:
configuration: ${{ matrix.configuration }}
- name: archive cache
run: tar -czf conan.tar -C ~/.conan .
- name: upload cache
uses: actions/upload-artifact@v3
with:
name: ${{ matrix.platform }}-${{ matrix.compiler }}-${{ matrix.configuration }}
path: conan.tar


test:
strategy:
fail-fast: false
matrix:
platform:
- linux
compiler:
- gcc
- clang
configuration:
- Debug
- Release
cmake-args:
-
- "-Dunity=ON"
needs: dependencies
runs-on: [self-hosted, heavy]
container: thejohnfreeman/rippled-build-ubuntu:12e19cd9034b
env:
build_dir: .build
steps:
- name: download cache
uses: actions/download-artifact@v3
with:
name: ${{ matrix.platform }}-${{ matrix.compiler }}-${{ matrix.configuration }}
- name: extract cache
run: |
mkdir -p ~/.conan
tar -xzf conan.tar -C ~/.conan
- name: check environment
run: |
echo ${PATH} | tr ':' '\n'
conan --version
cmake --version
env
ls ~/.conan
- name: checkout
uses: actions/checkout@v3
- name: build
uses: ./.github/actions/build
with:
generator: Ninja
configuration: ${{ matrix.configuration }}
cmake-args: ${{ matrix.cmake-args }}
- name: test
run: |
${build_dir}/rippled --unittest --unittest-jobs $(nproc)
99 changes: 99 additions & 0 deletions .github/workflows/windows.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
name: windows
# We have disabled this workflow because it fails in our CI Windows
# environment, but we cannot replicate the failure in our personal Windows
# test environments, nor have we gone through the trouble of setting up an
# interactive CI Windows environment.
# We welcome contributions to diagnose or debug the problems on Windows. Until
# then, we leave this tombstone as a reminder that we have tried (but failed)
# to write a reliable test for Windows.
# on: [push, pull_request]
on:
workflow_dispatch:
push:
branches:
- 'action'
paths:
- '.github/workflow/windows.yml'

jobs:

test:
strategy:
fail-fast: false
matrix:
generator:
- Visual Studio 16 2019
configuration:
- Release
runs-on: windows-2019
env:
build_dir: .build
steps:
- name: checkout
uses: actions/checkout@v3
- name: choose Python
uses: actions/setup-python@v3
with:
python-version: 3.9
- name: learn Python cache directory
id: pip-cache
run: |
pip install --upgrade pip
echo "dir=$(pip cache dir)" >> $GITHUB_OUTPUT
- name: restore Python cache directory
uses: actions/cache@v2
with:
path: ${{ steps.pip-cache.outputs.dir }}
key: ${{ runner.os }}-${{ hashFiles('.github/workflows/windows.yml') }}
- name: install Conan
run: pip install wheel 'conan<2'
- name: check environment
run: |
$env:PATH -split ';'
python --version
conan --version
cmake --version
dir env:
- name: configure Conan
run: |
conan profile new default --detect
conan profile update settings.compiler.cppstd=20 default
conan profile update settings.compiler.runtime=MT default
conan profile update settings.compiler.toolset=v141 default
- name: learn Conan cache directory
id: conan-cache
run: |
echo "dir=$(conan config get storage.path)" >> $GITHUB_OUTPUT
- name: restore Conan cache directory
uses: actions/cache@v2
with:
path: ${{ steps.conan-cache.outputs.dir }}
key: ${{ hashFiles('~/.conan/profiles/default', 'conanfile.py', 'external/rocksdb/*', '.github/workflows/windows.yml') }}
- name: export custom recipes
run: |
conan export external/snappy snappy/1.1.9@
conan export external/soci soci/4.0.3@
- name: install dependencies
run: |
mkdir $env:build_dir
cd $env:build_dir
conan install .. --build missing --settings build_type=${{ matrix.configuration }}
- name: configure
run: |
$env:build_dir
cd $env:build_dir
pwd
ls
cmake `
-G "${{ matrix.generator }}" `
-DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake `
-Dassert=ON `
-Dreporting=OFF `
-Dunity=OFF `
..
- name: build
run: |
cmake --build $env:build_dir --target rippled --config ${{ matrix.configuration }} --parallel $env:NUMBER_OF_PROCESSORS
- name: test
run: |
& "$env:build_dir\${{ matrix.configuration }}\rippled.exe" --unittest
Loading

0 comments on commit ed9f669

Please sign in to comment.