-
Notifications
You must be signed in to change notification settings - Fork 379
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for Android versions 5-11. #1023
Conversation
Currently only tested on
This should also simplify migrating to newer Android NDK and SDK versions in the future. This will also need a wiki entry, I believe. I've also tested the Android NDK versions:
Also, the We also have added |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
I've restored using Python for the Android NDK install steps, since it's a bit more resilient than making all the symlinks outself manually. We could probably speed it up slightly making manual symlinks, but considering the major bottleneck is the Android build itself, and this adds no size to the resulting image, I don't see any reason to remove it. I've added |
bors try --target aarch64-linux-android |
tryBuild succeeded: |
8a888c5
to
782ecd1
Compare
f74d84b
to
17c074a
Compare
16dee77
to
75870b7
Compare
79fb560
to
97d747e
Compare
There might be other things I'm missing: I've tried to run through this numerous times, but at 4k lines of code, there may be something I've overlooked. I've re-run the tests for the entire suite of Android versions on |
tryBuild succeeded: |
since these scripts are very android specific and most of it would not be used in CI, would it make sense to move this to cross-toolchain? |
Sure, that could work. We can then also cherry pick pieces if we say, want to upgrade to Android 11. |
If we do migrate this to cross-toolchains, it would be nice if we could have some basic CI to test the additions, however: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks amazing, some minor comments
I'm not super familiar with the android build system so can't comment to much on the approach, but this seems great!
This might now be the most complete containerized build system for c/c++/rust projects in android available anywhere 😅
i'm fine with merging this into this repo
**Configuration** We provide the following arguments to configure the Android version: - `ANDROID_NDK` - `ANDROID_SDK` (note that this is dependent on `ANDROID_VERSION) - `ANDROID_VERSION` - `ANDROID_SYSTEM_COMPLETE`: do a complete Android build - `ANDROID_SYSTEM_NONE`: do not build the Android system, disables runner support **Version Support** We now support NDK versions r10e-r25b, SDK versions 21-33, and Android versions 5.0, 5.1, 6.0, 7.0, 8.0, 8.1, 9.0, 10.0, and 11.0. We also validate that the NDK, SDK, and Android versions are compatible. For Android 12+, we do support using the NDK with the proper API level, allowing cross-compilation for the desired target, however, running or testing the generated binary is currently not supported without a complete build. Note that support for non-complete builds with Android versions 12+ is unlikely to ever occur, due to issues with APEX in the build system. **Implementation Details** Next, we've improved the removal of unittests during the build process, to ensure fast builds while maintaining compatibility with various newer Android versions. To do this, we've implemented a Python library and a script. The Python library contains a complete parser (correctly parses all valid input) for Soong blueprint files, using an LALR grammar, and a rudimentary parser for Makefiles. The Soong parser removes any `gtest` dependencies, as well as any subdirectories or scope names containing `test`. For example: ```go cc_library { name: "lib", srcs: ["lib.cc",], } cc_test { name: "test", defaults: ["target"], srcs: ["test.cc"], } ``` Will become: ```go cc_library { name: "lib", srcs: ["lib.cc",], } ``` The Makefile parser first splits the file based on conditional directives (`ifeq`, `endif`, etc.) to ensure any subsequent processing doesn't lead to unbalanced directives. Next, we split the text within each directive based on comment sections used in the Android source tree. For example: ```Makefile test_tags := tests include $(call subdir,$(LOCAL_PATH)) c_flags := \ -g \ -Wall ``` We can therefore remove the `Benchmarks` and `Unit tests` sections without removing the `Other section`. The Python library is reasonably performant (it adds no noticeable overhead to the build process) and is compact (in total, < 60KB). Also, it is much more resilient than a series of `sed` scripts. Finally, extensive unittests have been added for the Python library, for both code linting (`flake8`) and unittests (via `tox`). Since we cannot assume the presence of Python on the host machine, the tests can be optionally enabled via the `--python` flag (or `PYTHON` environment variable, to hook into the git hooks), and custom overrides for the `flake8` and `tox` commands are provided (since the user may wish to specify a specific Python version, such as `python3.7 -m flake8`). **Linker** For Android 10+, since we use a minimal Android build, we only support the bootstrap and ASAN linkers unless using a complete Android build. Supporting the APEX linker requires a nearly complete Android runtime, requiring 60+GB image sizes, slow builds, and other prohibitive factors. **Complete Builds** Complete builds are currently untested, for Androids version 12+ they exceed the default storage capacity of WSL2 images (250GB). They should work and properly install both the `/system` and `/apex` directories, however, the builds are slow and untested.
c2ff7ac
to
25c7121
Compare
I've updated the NDK to |
tryBuild succeeded: |
bors r=Emilgardis |
1023: Add support for Android versions 5-11. r=Emilgardis a=Alexhuszagh **Configuration** We provide the following arguments to configure the Android version: - `ANDROID_NDK` - `ANDROID_SDK` (note that this is dependent on `ANDROID_VERSION) - `ANDROID_VERSION` - `ANDROID_SYSTEM_COMPLETE`: do a complete Android build - `ANDROID_SYSTEM_NONE`: do not build the Android system, disables runner support **Version Support** We now support NDK versions r10e-r25b, SDK versions 21-33, and Android versions 5.0, 5.1, 6.0, 7.0, 8.0, 8.1, 9.0, 10.0, and 11.0. We also validate that the NDK, SDK, and Android versions are compatible. For Android 12+, we do support using the NDK with the proper API level, allowing cross-compilation for the desired target, however, running or testing the generated binary is currently not supported without a complete build. Note that support for non-complete builds with Android versions 12+ is unlikely to ever occur, due to issues with APEX in the build system. **Implementation Details** Next, we've improved the removal of unittests during the build process, to ensure fast builds while maintaining compatibility with various newer Android versions. To do this, we've implemented a Python library and a script. The Python library contains a complete parser (correctly parses all valid input) for Soong blueprint files, using an LALR grammar, and a rudimentary parser for Makefiles. The Soong parser removes any `gtest` dependencies, as well as any subdirectories or scope names containing `test`. For example: ```go cc_library { name: "lib", srcs: ["lib.cc",], } cc_test { name: "test", defaults: ["target"], srcs: ["test.cc"], } ``` Will become: ```go cc_library { name: "lib", srcs: ["lib.cc",], } ``` The Makefile parser first splits the file based on conditional directives (`ifeq`, `endif`, etc.) to ensure any subsequent processing doesn't lead to unbalanced directives. Next, we split the text within each directive based on comment sections used in the Android source tree. For example: ```Makefile test_tags := tests include $(call subdir,$(LOCAL_PATH)) c_flags := \ -g \ -Wall ``` We can therefore remove the `Benchmarks` and `Unit tests` sections without removing the `Other section`. The Python library is reasonably performant (it adds no noticeable overhead to the build process) and is compact (in total, < 60KB). Also, it is much more resilient than a series of `sed` scripts. Finally, extensive unittests have been added for the Python library, for both code linting (`flake8`) and unittests (via `tox`). Since we cannot assume the presence of Python on the host machine, the tests can be optionally enabled via the `--python` flag (or `PYTHON` environment variable, to hook into the git hooks), and custom overrides for the `flake8` and `tox` commands are provided (since the user may wish to specify a specific Python version, such as `python3.7 -m flake8`). **Linker** For Android 10+, since we use a minimal Android build, we only support the bootstrap and ASAN linkers unless using a complete Android build. Supporting the APEX linker requires a nearly complete Android runtime, requiring 60+GB image sizes, slow builds, and other prohibitive factors. **Complete Builds** Complete builds are currently untested, for Androids version 12+ they exceed the default storage capacity of WSL2 images (250GB). They should work and properly install both the `/system` and `/apex` directories, however, the builds are slow and untested. Co-authored-by: Alex Huszagh <[email protected]>
Build failed: |
Currently waiting on the fix for |
bors r+ p=1 |
1023: Add support for Android versions 5-11. r=Emilgardis a=Alexhuszagh **Configuration** We provide the following arguments to configure the Android version: - `ANDROID_NDK` - `ANDROID_SDK` (note that this is dependent on `ANDROID_VERSION) - `ANDROID_VERSION` - `ANDROID_SYSTEM_COMPLETE`: do a complete Android build - `ANDROID_SYSTEM_NONE`: do not build the Android system, disables runner support **Version Support** We now support NDK versions r10e-r25b, SDK versions 21-33, and Android versions 5.0, 5.1, 6.0, 7.0, 8.0, 8.1, 9.0, 10.0, and 11.0. We also validate that the NDK, SDK, and Android versions are compatible. For Android 12+, we do support using the NDK with the proper API level, allowing cross-compilation for the desired target, however, running or testing the generated binary is currently not supported without a complete build. Note that support for non-complete builds with Android versions 12+ is unlikely to ever occur, due to issues with APEX in the build system. **Implementation Details** Next, we've improved the removal of unittests during the build process, to ensure fast builds while maintaining compatibility with various newer Android versions. To do this, we've implemented a Python library and a script. The Python library contains a complete parser (correctly parses all valid input) for Soong blueprint files, using an LALR grammar, and a rudimentary parser for Makefiles. The Soong parser removes any `gtest` dependencies, as well as any subdirectories or scope names containing `test`. For example: ```go cc_library { name: "lib", srcs: ["lib.cc",], } cc_test { name: "test", defaults: ["target"], srcs: ["test.cc"], } ``` Will become: ```go cc_library { name: "lib", srcs: ["lib.cc",], } ``` The Makefile parser first splits the file based on conditional directives (`ifeq`, `endif`, etc.) to ensure any subsequent processing doesn't lead to unbalanced directives. Next, we split the text within each directive based on comment sections used in the Android source tree. For example: ```Makefile test_tags := tests include $(call subdir,$(LOCAL_PATH)) c_flags := \ -g \ -Wall ``` We can therefore remove the `Benchmarks` and `Unit tests` sections without removing the `Other section`. The Python library is reasonably performant (it adds no noticeable overhead to the build process) and is compact (in total, < 60KB). Also, it is much more resilient than a series of `sed` scripts. Finally, extensive unittests have been added for the Python library, for both code linting (`flake8`) and unittests (via `tox`). Since we cannot assume the presence of Python on the host machine, the tests can be optionally enabled via the `--python` flag (or `PYTHON` environment variable, to hook into the git hooks), and custom overrides for the `flake8` and `tox` commands are provided (since the user may wish to specify a specific Python version, such as `python3.7 -m flake8`). **Linker** For Android 10+, since we use a minimal Android build, we only support the bootstrap and ASAN linkers unless using a complete Android build. Supporting the APEX linker requires a nearly complete Android runtime, requiring 60+GB image sizes, slow builds, and other prohibitive factors. **Complete Builds** Complete builds are currently untested, for Androids version 12+ they exceed the default storage capacity of WSL2 images (250GB). They should work and properly install both the `/system` and `/apex` directories, however, the builds are slow and untested. Co-authored-by: Alex Huszagh <[email protected]>
Build failed: |
bors retry |
bors r=Emilgardis |
Already running a review |
Build succeeded: |
Configuration
We provide the following arguments to configure the Android version:
ANDROID_NDK
ANDROID_SDK
(note that this is dependent on `ANDROID_VERSION)ANDROID_VERSION
ANDROID_SYSTEM_COMPLETE
: do a complete Android buildANDROID_SYSTEM_NONE
: do not build the Android system, disables runner supportVersion Support
We now support NDK versions r10e-r25b, SDK versions 21-33, and Android versions 5.0, 5.1, 6.0, 7.0, 8.0, 8.1, 9.0, 10.0, and 11.0. We also validate that the NDK, SDK, and Android versions are compatible.
For Android 12+, we do support using the NDK with the proper API level, allowing cross-compilation for the desired target, however, running or testing the generated binary is currently not supported without a complete build. Note that support for non-complete builds with Android versions 12+ is unlikely to ever occur, due to issues with APEX in the build system.
Implementation Details
Next, we've improved the removal of unittests during the build process, to ensure fast builds while maintaining compatibility with various newer Android versions. To do this, we've implemented a Python library and a script. The Python library contains a complete parser (correctly parses all valid input) for Soong blueprint files, using an LALR grammar, and a rudimentary parser for Makefiles.
The Soong parser removes any
gtest
dependencies, as well as any subdirectories or scope names containingtest
. For example:Will become:
The Makefile parser first splits the file based on conditional directives (
ifeq
,endif
, etc.) to ensure any subsequent processing doesn't lead to unbalanced directives. Next, we split the text within each directive based on comment sections used in the Android source tree. For example:We can therefore remove the
Benchmarks
andUnit tests
sections without removing theOther section
.The Python library is reasonably performant (it adds no noticeable overhead to the build process) and is compact (in total, < 60KB). Also, it is much more resilient than a series of
sed
scripts.Finally, extensive unittests have been added for the Python library, for both code linting (
flake8
) and unittests (viatox
). Since we cannot assume the presence of Python on the host machine, the tests can be optionally enabled via the--python
flag (orPYTHON
environment variable, to hook into the git hooks), and custom overrides for theflake8
andtox
commands are provided (since the user may wish to specify a specific Python version, such aspython3.7 -m flake8
).Linker
For Android 10+, since we use a minimal Android build, we only support the bootstrap and ASAN linkers unless using a complete Android build. Supporting the APEX linker requires a nearly complete Android runtime, requiring 60+GB image sizes, slow builds, and other prohibitive factors.
Complete Builds
Complete builds are currently untested, for Androids version 12+ they exceed the default storage capacity of WSL2 images (250GB). They should work and properly install both the
/system
and/apex
directories, however, the builds are slow and untested.