Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup Linux CI for Qt5.15.8+KDAB using minimum glibc version compatibility #9350

Closed
alexjba opened this issue Jan 30, 2023 · 63 comments
Closed

Comments

@alexjba
Copy link
Contributor

alexjba commented Jan 30, 2023

Description

Prebuilt binaries of Qt5.15.8 are compiled using glibc 2.28 and migrating to this version will breaking compatibility with older linux versions.

The purpose of this task is to setup the Linux CI to use latest Qt5.15.8 + KDAB fixes and keep the glibc dependency to a lower version (Ideally 2.23 or lower). Also, we will have this issue when moving to Qt6 and solving this will also help long term with newer qt versions.

There are multiple options that could be considered:

  1. Finding an out of the box setup - precompiled Qt binaries compatible with older linux versions
  2. Compiling Qt5.15.8 with KDAB patches + Status and link with glibc 2.23 (probably the easiest way to do it is on a linux distribution that uses this version)
  3. Try this hack https://github.com/wheybags/glibc_version_header and enable Nix environment on CI
  4. https://github.com/phusion/holy-build-box

On the CI side there are two approaches that were considered when migrating to Qt5.15: Docker and Nix. Nix is preferred, but it has the same limitation - will break compatibility with older linux dostros. We're using Docker for now.

See: #9222, #9232

@anastasiyaig
Copy link
Contributor

@alexjba @jakubgs we need that for all platforms (windows and mac inclusive). windows now is 5.15.2 and arm mac is 5.15.8 when intel is 5.15.5

@jakubgs
Copy link
Member

jakubgs commented Aug 1, 2023

Actually, we could combine using Nix together with use of https://github.com/wheybags/glibc_version_header.
And that might be the best option for us, since it would provide locked Qt for both Linux and MacOS, and possibly even Windows if we managed to use WSL for CI. Though that might require extra work.

@iurimatias iurimatias modified the milestones: 0.15, 0.16 Sep 18, 2023
@jakubgs jakubgs assigned yakimant and unassigned jakubgs Oct 12, 2023
@yakimant
Copy link
Member

Hi guys, I started to research on this issue.

Here is a first finding - there is no binary for Qt 5.15.3+ available.

❯ aqt list-qt linux desktop
5.9.0 5.9.1 5.9.2 5.9.3 5.9.4 5.9.5 5.9.6 5.9.7 5.9.8 5.9.9
5.10.0 5.10.1
5.11.0 5.11.1 5.11.2 5.11.3
5.12.0 5.12.1 5.12.2 5.12.3 5.12.4 5.12.5 5.12.6 5.12.7 5.12.8 5.12.9 5.12.10 5.12.11 5.12.12
5.13.0 5.13.1 5.13.2
5.14.0 5.14.1 5.14.2
5.15.0 5.15.1 5.15.2
6.0.0 6.0.1 6.0.2 6.0.3 6.0.4
6.1.0 6.1.1 6.1.2 6.1.3
6.2.0 6.2.1 6.2.2 6.2.3 6.2.4
6.3.0 6.3.1 6.3.2
6.4.0 6.4.1 6.4.2 6.4.3
6.5.0 6.5.1 6.5.2 6.5.3
6.6.0

which taked binaries from https://download.qt.io/online/qtsdkrepository/linux_x64/desktop/

See miurahr/aqtinstall#572

Do I miss smth here and there are other "Prebuilt binaries of Qt5.15.8" for Linux?

@yakimant
Copy link
Member

@yakimant
Copy link
Member

@caybro
Copy link
Member

caybro commented Oct 13, 2023

23.04 apparently has 5.15.8

@alexjba
Copy link
Contributor Author

alexjba commented Oct 13, 2023

Homebrew maintained binaries: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/versions?filters%5Bversion_type%5D=tagged

For example 5.15.8_3-1: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/89485747?tag=5.15.8_3-1 (includes Linux one)

Yes, at that time we've found the binaries in Homebrew, nix packages and the Ubuntu distro @caybro is using.

@caybro Can you please share the url for the Ubuntu distro you're using? IDK if it helps, but for the record 😄

@caybro
Copy link
Member

caybro commented Oct 13, 2023

Homebrew maintained binaries: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/versions?filters%5Bversion_type%5D=tagged
For example 5.15.8_3-1: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/89485747?tag=5.15.8_3-1 (includes Linux one)

Yes, at that time we've found the binaries in Homebrew, nix packages and the Ubuntu distro @caybro is using.

@caybro Can you please share the url for the Ubuntu distro you're using? IDK if it helps, but for the record 😄

Not using Ubuntu but Fedora but here you go 😆
https://packages.fedoraproject.org/search?query=qt5-*

@alexjba
Copy link
Contributor Author

alexjba commented Oct 13, 2023

@yakimant The downside of using Homebrew packages is that the Docker image will increase in size. Around 8gb if I remember correctly. Probably it doesn't really matter if we're downloading the image only once per machine and this doesn't affect the CI jobs duration.

@yakimant
Copy link
Member

@caybro
Copy link
Member

caybro commented Oct 13, 2023

That's interesting, nix uses KDE patched sources for Qt5: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/qt-5/5.15/srcs-generated.json

But original for Qt6: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/qt-6/srcs.nix

Because for Qt 6, there's no difference between publicly available OSS and commercial versions

@yakimant
Copy link
Member

Thanks for you comments!

How can I reproduce the issue?

No issues for Windows and macOS, right?
(Only finding/building the binaries)

@alexjba
Copy link
Contributor Author

alexjba commented Oct 13, 2023

To reproduce this you'll need to download Qt5.15.8+ binaries and use them to compile status desktop on the Docker image we're currently using.

@yakimant
Copy link
Member

Cool! Is it a build time or runtime issue?

@yakimant
Copy link
Member

Also, can you please share, which 5.15.8+ binaries did you use, that are compiled with glibc 2.28?
And why it is a problem, if Ubuntu 20.04 has glibc 2.31, which is newer:
https://packages.ubuntu.com/search?suite=all&section=all&arch=any&keywords=libc-dev-bin&searchon=names

@yakimant
Copy link
Member

yakimant commented Feb 13, 2024

@micieslak helped to disable Qt WebEngine, which we expected to cause a crash: #13540

The app is running and login is succesfull, no crash anymore.

@stefandunca, can you please help to understand, how much do we need / use it?
@anastasiyaig, can you please continue smoke testing?

Thank you guys, great help with the project!

@stefandunca
Copy link
Contributor

...
@stefandunca, can you please help to understand, how much do we need / use it? @anastasiyaig, can you please continue smoke testing?
...

As discussed privately, we need this for Wallet Connect. There is a workaround using a WebView, but it would require more work to re-implement.
Also, the outdated web browser feature for dapp integration, which is now disabled, requires it.

@alexjba
Copy link
Contributor Author

alexjba commented Feb 16, 2024

...
@stefandunca, can you please help to understand, how much do we need / use it? @anastasiyaig, can you please continue smoke testing?
...

As discussed privately, we need this for Wallet Connect. There is a workaround using a WebView, but it would require more work to re-implement. Also, the outdated web browser feature for dapp integration, which is now disabled, requires it.

Not sure it would help. It probably depends on what is the root cause of this crash.
WebView still depends on WebEngine on linux.
https://doc.qt.io/qt-5/qtwebview-index.html

I guess the first step would be to pinpoint the root cause.

@anastasiyaig
Copy link
Contributor

@yakimant i tested this build with my autotests and all looks good. The only thing i noticed, is when i run application in test mode (adding env var STATUS_RUNTIME_TEST_MODE = True) - then my tests cant attach to the nix build. All good in this mode for old build, no errors

Nix build with test mode enabled:
https://ci.status.im/job/status-desktop/job/e2e/job/manual/1466/allure/#categories/c4e6cb3e88347080bb258e8e1856aa48/74c37dd22099f53/
imeoutError: Unable to establish TCP connection with localhost:62966.

Normal nightly with test mode enabled:
https://ci.status.im/job/status-desktop/job/e2e/job/manual/1467/allure/

Wonder what that could be , because my big hope was to upgrade to 5.15.8 and run my tests with this env var

@anastasiyaig
Copy link
Contributor

I also found this today (just normal run of my tests, no test mode enabled):

aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.32' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.35' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.32' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.35' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.32' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.35' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.32' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
aut/StatusIm-Desktop-240216-121542-6be832-x86_64.AppImage: /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)

as result, squish cant connect and all tests failed

06:57:16:772 Debug: Using server configuration file '/home/jenkins/workspace/status-desktop/e2e/manual@3/squish.ini'
06:57:16:772 Info: Starting server of Squish version 7.2.1.
06:57:16:773 Debug: Read configuration file '/opt/squish-runner-7.2.1/etc/squishserverrc'
06:57:16:773 Info: Listening on 127.0.0.1:5886
06:57:16:773 Info: Listening on ws://127.0.0.1:45517?tGIe2BW4n3oLux1OFV0W
06:57:16:773 Info: Announcing server presence via SSDP 
06:57:17:776 Warning: Channel Connection closed during handshake:43026 rejected: 127.0.0.1
06:57:17:776 Warning: Channel Connection closed during handshake:43040 rejected: 127.0.0.1
07:21:17:280 Debug: Shutting down server

Logs are available here https://ci.status.im/job/status-desktop/job/e2e/job/manual/1564/

@yakimant do you have any isight what that could be? The host selected was Linux 03, maybe it is not yet properly set up?

@iurimatias iurimatias modified the milestones: 2.29.0 Beta, 2.30.0 Beta Mar 5, 2024
@yakimant
Copy link
Member

yakimant commented Apr 9, 2024

  • The second issue is about system libstdc++ used, which needs newer version of glibc.

Possible solutions:

  • bundle libstdc++

What's interesting libc from Nix Store /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6 is searched, not the /lib/... one. Maybe we need to cleanup the /nix/store/... paths from libs.

@yakimant
Copy link
Member

yakimant commented May 8, 2024

We have a progress on qtwebengine/wallet issue after Athens offsite.

The build loads post-login screen:
https://ci.status.im/job/status-desktop/job/systems/job/linux/job/x86_64/job/package_nix/38/

@anastasiyaig, can you please have a look?
We also nee to sync on the issues you mentioned above.

Meanwhile I will rebase PR from master.

@yakimant
Copy link
Member

Doesn't run on 22.04 in my case, just stucks on loading after entering password.

Logs arent different from 20.04 run, but just for the reference, these are the last lines:

Critical: AppMain: Unknown section type (qrc:/app/mainui/AppMain.qml:1169, expression for currentIndex)
Warning: qrc:/app/mainui/AppMain.qml:1139:21: Unable to assign [undefined] to int (qrc:/app/mainui/AppMain.qml:1139, )
Warning: qrc:/app/AppLayouts/Onboarding/OnboardingLayout.qml:151:9: Unable to assign [undefined] to QObject* (qrc:/app/AppLayouts/Onboarding/OnboardingLayout.qml:151, )
Warning: qrc:/main.qml:173:9: Unable to assign [undefined] to QObject* (qrc:/main.qml:173, )
WRN 2024-05-10 12:22:43.493+02:00 Error decoding signal                      topics="signals-manager" tid=12035 file=signals_manager.nim:49 err="Unknown signal received: mediaserver.started"
ERR 2024-05-10 12:22:43.996+02:00 getChatContentModule: unexisting chat key  topics="chat-section-module" tid=12035 file=module.nim:543 chatId= methodName=getChatContentModule
ERR 2024-05-10 12:22:44.061+02:00 getChatContentModule: unexisting chat key  topics="chat-section-module" tid=12035 file=module.nim:543 chatId= methodName=getChatContentModule
ERR 2024-05-10 12:22:44.075+02:00 getChatContentModule: unexisting chat key  topics="chat-section-module" tid=12035 file=module.nim:543 chatId= methodName=getChatContentModule
ERR 2024-05-10 12:22:44.333+02:00 getChatContentModule: unexisting chat key  topics="chat-section-module" tid=12035 file=module.nim:543 chatId= methodName=getChatContentModule
ERR 2024-05-10 12:22:44.336+02:00 getChatContentModule: unexisting chat key  topics="chat-section-module" tid=12035 file=module.nim:543 chatId= methodName=getChatContentModule
ERR 2024-05-10 12:22:44.879+02:00 getChatContentModule: unexisting chat key  topics="chat-section-module" tid=12035 file=module.nim:543 chatId= methodName=getChatContentModule

@yakimant
Copy link
Member

Update on current issues:

  • Ubuntu 22 - most probably VM or local setup issue. Test ran on the app on CI agen with Ubuntu 22. We need someone to check this on local machine.
  • Test mode - I see it's commented out in Jeninsfile, so low priority
  • Issue /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6: version GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)` - solved. I striped RPATH from the AppImage.

So I see no blockers. @anastasiyaig, can you confirm, when you are back.
Especially the Ubuntu 22.

@anastasiyaig
Copy link
Contributor

sure @yakimant , thanks for your time and effort, i will follow up on that when I am back

@yakimant
Copy link
Member

@alexjba, @caybro, if you are interested, here is the build:
https://ci.status.im/job/status-desktop/job/systems/job/linux/job/x86_64/job/package_nix/41/

If we don't find blockers - I will polish the code and merge.

The next steps (if everything goes well) would be:

  • Build will run in parallel and I can switch later
  • Update Qt version to 5.15.12
  • Try nix-appimage as a better approach.

@yakimant
Copy link
Member

yakimant commented Jun 5, 2024

According to @anastasiyaig, smoke tests are passing.
So I will do some cleanup in PR and merge, build will run in parallel.

@anastasiyaig
Copy link
Contributor

yup right we can do more tests afterwards

@yakimant
Copy link
Member

yakimant commented Jun 26, 2024

Leftovers from #9232:

  • jenkins-lib: refactor linux.bundle to have named arguments
  • qt: workarounds to be able to build on nixpkgs 20.04
  • go: switch to build from binary
  • upgrade appimagetool & linuxdeployqt (add runtime jdependency to 1st)
  • check shell inputs are not redundant, we might added more than needed
  • remote nix store cleanup
  • qa: test go upgrade didn't break anything, test translations, run full e2e suit

Next:

  • Try nix-appimage
  • status-desktop build derivation
  • macOS and Windows support
  • Qt latest updgrade

This list will be updated

@yakimant
Copy link
Member

The latest opensourse version is 5.15.14:

Comercial is 5.15.17:

They delay source code by 1 year, so it will appear like that:

  • 5.15.15 - 2024 Aug 31
  • 5.15.16 - 2024 Nov 17
  • 5.15.17 - 2025 May 22

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

No branches or pull requests

10 participants