Segment fault with Ubuntu 24.04 20250120.5.0

[RadxaYuntian]

Description

We have a scheduled job running every Sunday, which failed today, with no code change in last 2 weeks.

After checking the build log, it always failed at a dkms package installation. Once the workflow file is changed to print the dkms log, the error is always gcc segment fault.

Changing running environment to ubuntu22-04 fixed the segment fault. Action still failed but that's because the change we made to investigate this issue.

What may be unusual for us is that we are using binfmt to run aarch64 gcc in a devcontainer, because the final output is an aarch64 system image. So this is not some normal gcc failing.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022
• Windows Server 2025

Image version and build link

20250120.5.0

Is it regression?

20250105.1.0: https://github.com/RadxaOS-SDK/rsdk/actions/runs/12848906725

Expected behavior

DKMS install successfully without gcc segfault.

Actual behavior

gcc segfault:

   2025-01-26 07:47:36,252 bdebstrap ERROR: mmdebstrap failed with exit code 25. See above for details.
  
  /workspaces/rsdk
  
  DKMS make.log for radxa-overlays-0.1.20 for kernel 6.1.68-2-stable (aarch64)
  Sun Jan 26 07:47:16 UTC 2025
  make: Entering directory '/usr/src/linux-headers-6.1.68-2-stable'
  Segmentation fault (core dumped)
  warning: the compiler differs from the one used to build the kernel
    The kernel was built by: aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110
    You are using:           gcc (Debian 12.2.0-14) 12.2.0
    CC [M]  /var/lib/dkms/radxa-overlays/0.1.20/build/radxa-overlays.o
    DTC     /var/lib/dkms/radxa-overlays/0.1.20/build/arch/arm64/boot/dts/amlogic/overlays/meson-g12-disable-gpu.dtbo
    DTC     /var/lib/dkms/radxa-overlays/0.1.20/build/arch/arm64/boot/dts/amlogic/overlays/meson-g12-disable-hdmi.dtbo
    DTC     /var/lib/dkms/radxa-overlays/0.1.20/build/arch/arm64/boot/dts/rockchip/overlays/radxa-s0-ext-antenna.dtbo
  gcc: internal compiler error: Segmentation fault signal terminated program cc1
  Please submit a full bug report, with preprocessed source (by using -freport-bug).
  See <file:///usr/share/doc/gcc-12/README.Bugs> for instructions.
  make[2]: *** [scripts/Makefile.lib:409: /var/lib/dkms/radxa-overlays/0.1.20/build/arch/arm64/boot/dts/rockchip/overlays/radxa-s0-ext-antenna.dtbo] Error 4
  make[1]: *** [scripts/Makefile.build:500: /var/lib/dkms/radxa-overlays/0.1.20/build/arch/arm64/boot/dts/rockchip/overlays] Error 2
  make[1]: *** Waiting for unfinished jobs....

Repro steps

1. Clone https://github.com/RadxaOS-SDK/rsdk
2. Cherry pick RadxaYuntian/rsdk@090908a to view dkms log
3. Trigger workflow_dispatch for build.yaml

[RadxaYuntian]

The gcc version Debian 12.2.0-14 was released on 2023/01/08, so the last successful run (2025/01/19) and today's failed run are both using the same version in the devcontainer.

[deviantintegral]

I can confirm this as well at https://github.com/pbkhrv/rtl_433-hass-addons/actions/runs/12972957498/job/36181006667. That job is compiling aarch64 in Docker under QEMU (I know, proper cross compiling would be better, but this is what the official Home Assistant builder action does so 🤷 ).

Is there a way to specify the runner image version to a previous 24.04 release to confirm the regression?

[woblerr]

The same problem for buildx for linux/arm64 via QEMU: https://github.com/woblerr/docker-pgbackrest/actions/runs/12965488407/job/36165276019#step:7:2658

Rollback to the ubuntu-22.04 runner solved the problem.

[MyreMylar]

Chiming in to say that we are seeing segfaults on our test runners for pygame-ce in the ppc64le architecture build since getting version 20250120.5.0. and, perhaps related, it also reporting that it can no longer detect the GNU compiler type for our S390x architecture build.

As @deviantintegral says it would be nice to have a way to roll back to a previous runner image to isolate the problem.

[RaviAkshintala]

Hi @RadxaYuntian Thank you for bringing this issue to our attention. We will look into this issue and will update you after investigating.

[BrianPugh]

I'm also having very similar issues in tamp when using cibuildwheel to build python wheels for ppc64le and aarch64 targets.

[rtobar]

Same issue here with gcc segfault, but in my case I saw it both with ubuntu-latest and ubuntu-20.04. Updating/downgrading to ubuntu-22.04 solved it as mentioned by other people.

[RadxaYuntian]

We are building a Debian 12 system image, which contains a DKMS package, and DKMS itself was called by the kernel postinst hooks. So it is not like some intermediate environment that we are free to install any software: we need to clean up as well. Also there is no gcc-10 packages in Debian 12 repositories, so we will have to build gcc-10 from source with shipped gcc-12, and try to get rid of gcc-12 without breaking the package dependencies to install this one as the default compiler, then clean up gcc-10, and reinstall gcc-12.

I think this is too much effort for a temporary workaround.

We will roll back to 22.04 for now.

[baszoetekouw]

I believe this is because of mismatch between gcc and kernel version(Kernel Version: 6.8.0-1020-azure) . this kernel version only support with gcc 10 . i request you if possible try to install gcc 10 and re try the test or you can proceed with the ubuntu-22 image until new version released. thank you !

I'm also seeing these issues with gcc-10 (e.g, https://github.com/OpenConext/OpenConext-BaseContainers/actions/runs/13181115842/job/36791848196), so I don't think the compiler version is the problem.

[baszoetekouw]

To add to the discussion: the failures seem to be a bit random, in the sense that builds reliably fail, but the exact spot where the failure occurs changes from build to build. I've been trying to work around this issue by using different compilers (multiple versions of gcc and clang for different parts of the build), only to see parts of the build fail that were fine in previous runs.

[baszoetekouw]

Another (and much nicer) workaround I've found, is to use the new ubuntu-24.04-arm runner to build arm64 stuff. Added bonus is that those are quite a but faster dan Qemu.

[kishorekumar-anchala]

hI @RadxaYuntian - Kindly raise the issue in repo . thanks, closing the issue.

[deviantintegral]

Is this fixed or not? This issue isn't with partner images, it's with the official github ones. If it's fixed, what runner image release version will contain the fix?

[baszoetekouw]

@kishorekumar-anchala why are you closing this? This issue is about the official github ubuntu-24.04 runners, not about the partner-provided arm runners. I was only pointing to those to use as a workaround for the problems with the github-provided runners

[RadxaYuntian]

I retriggered the failed workflow and the issue is not resolved. @RaviAkshintala please reopen this issue.

[woblerr]

hI @RadxaYuntian - Kindly raise the issue in repo . thanks, closing the issue.

This issue should stay here as it relates to the official image on GitHub. If this is not the case, then the restrictions should be specified in the image documentation.