MSSQL containers crashing after recent update

[HofmeisterAn]

Description

Hi, it looks like the recent update is causing issues with running MSSQL containers. Some Testcontainers users mentioned they're having trouble with the MSSQL module. I ran into similar issues before with Ubuntu 24.04 (testcontainers/testcontainers-dotnet#1248 (comment)). I just tried it now with 22.04, and the container crashes as well 😕. You can reproduce it by running the following command on the agent:

docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=YourStrong!Passw0rd' mcr.microsoft.com/mssql/server:2019-CU18-ubuntu-20.04

This program has encountered a fatal error and cannot continue running at Thu Sep 19 11:33:13 2024
The following diagnostic information is available:

         Reason: 0x00000001
         Signal: SIGABRT - Aborted (6)
          Stack:
                 IP               Function
                 ---------------- --------------------------------------
                 0000561e37a26bec <unknown>
                 0000561e37a26632 <unknown>
                 0000561e37a25c41 <unknown>
                 00007fa9f816a090 killpg+0x40
                 00007fa9f816a00b gsignal+0xcb
                 00007fa9f8149859 abort+0x12b
                 0000561e379af2a2 <unknown>
        Process: 9 - sqlservr
         Thread: 128 (application thread 0x1f8)
    Instance Id: e686a0a0-eb06-4753-ae51-d73296433979
       Crash Id: 5e19a47d-b473-48c7-a86b-68f8e5b84fc1
    Build stamp: f708684a2cfcb51177273c54f975ed8c62029cbc89aa962bf7d6b956f01a0c27
   Distribution: Ubuntu 20.04.5 LTS
     Processors: 4
   Total Memory: 16766771200 bytes
      Timestamp: Thu Sep 19 11:33:13 2024
     Last errno: 2
Last errno text: No such file or directory
/bin/cat: /proc/9/maps: Permission denied
/bin/cat: /proc/9/environ: Permission denied
/usr/bin/find: '/proc/9/task/9/fdinfo': Permission denied
/usr/bin/find: '/proc/9/task/11/fdinfo': Permission denied
/usr/bin/find: '/proc/9/task/12/fdinfo': Permission denied
/usr/bin/find: '/proc/9/task/13/fdinfo': Permission denied
/usr/bin/find: '/proc/9/task/14/fdinfo': Permission denied
/usr/bin/find: '/proc/9/task/15/fdinfo': Permission denied

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

20240915.1.0

Is it regression?

Yes, 20240908.1.0

Expected behavior

The container starts successfully.

Actual behavior

The container crashes; please see the attached logs above.

Repro steps

runs-on: ubuntu-22.04
steps:
  - name: Debug MSSQL Container Start
    run: docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=YourStrong!Passw0rd' mcr.microsoft.com/mssql/server:2019-CU18-ubuntu-20.04

[RaviAkshintala]

Hi @HofmeisterAn Thank you for bringing this issue to us. We are looking into this issue and will update you on this issue after investigating.

[HofmeisterAn]

I would like to add that I am also seeing other containers failing to start. Many of the container images are (seem) outdated, but they worked previously. Developers might encounter breaking changes.

[RaviAkshintala]

Hi @HofmeisterAn Could you please add runs-on: ubuntu-20.04 to your workflow file and try it?

[HofmeisterAn]

It is running on Ubuntu-20.04, but the kernel has not changed here.

[kirill-kosh]

Hi,
Can you please advise, if it possible to refer specific image for runner? It is working on recently version 20240908.1.0 as stated in the issue description. Is it possible to use it forcibly? It might be temporary workaround until issue is resolved.
// I'm not familiar with GitHub runners good enough to understand if it is impossible. It seems not possible after quick documentation review. But maybe I miss something.

[RaviAkshintala]

Hi @HofmeisterAn Could you please try with the ubuntu-latest. I hope the issue will be resolve,Thanks.

[HofmeisterAn]

Hi @HofmeisterAn Could you please try with the ubuntu-latest. I hope the issue will be resolve,Thanks.

It fails for all versions except ubuntu-20.04: https://github.com/testcontainers/testcontainers-dotnet/actions/runs/10971169435.

[RaviAkshintala]

Hi @HofmeisterAn Could you please try with the below command to start the MSSQL containers.
https://github.com/RaviAkshintala/testcontainers-dotnet/actions/runs/10993413941/workflow#L38

[omkar-foss]

I'm seeing this issue with another community-maintained SQL Server image (https://github.com/mcmoe/mssqldocker), failing workflow here. This had been working fine until few days back when this workflow failed.

I would like to highlight that this image mcmoe/mssqldocker hasn't been updated since like 3 years now, so recent image changes can't be an issue. Have also tried with several combinations of the official Microsoft SQL Server image, you can find the trials (4 commits) in this PR, all have failed including this one.

Since this is happening with two completely isolated images of SQL Server, could this be caused by something upstream, possibly a config change or some kernel-level change at the GitHub runner end?

[ecki]

BTW Microsoft’s 2022-latest image does not have an issue, probably because it’s Ubuntu 24.04 based.

[omkar-foss]

Thanks, 2022-latest image is working well with ubuntu-latest as well as ubuntu-20.04, trials here.

Side note: If using sqlcmd for health checks (using health-cmd), we have to use options -N (encrypt client-server connection) and -C (implicitly trust server cert) to make it work.

Update: In case someone is looking for a fully-functional SQL Server 2022 + Ubuntu Latest GitHub actions sample, I've created a very basic one in a repo here. Hope it helps.

[Blue101black]

Been running ubuntu-22.04 for past year, no changes, all of a sudden broken. Haven't ran it since september 6th, so don't know when exactly it broke.

Downgrading to 20.04 has fixed it

Task         : Bash
Description  : Run a Bash script on macOS, Linux, or Windows
Version      : 3.245.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
Script contents:
sqlcmd -Slocalhost,9998 -Usa -PMssql1234 -Q "RESTORE DATABASE ExampleProject FROM DISK = '/var/opt/mssqlbackups/sqlserver_known_good_database_v3_21_32_ExampleProject.bak' WITH MOVE 'ExampleProject' TO '/var/opt/mssql/ExampleProject.mdf', MOVE 'ExampleProject_log' TO '/var/opt/mssql/ExampleProject_log.ldf'"
========================== Starting Command Output ===========================
/usr/bin/bash /home/vsts/work/_temp/88394921-117e-45cd-a2f7-04a36e262c3d.sh
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login timeout expired.
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : TCP Provider: Error code 0x2749.
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online..

[mmrazik]

Since this is happening with two completely isolated images of SQL Server, could this be caused by something upstream, possibly a config change or some kernel-level change at the GitHub runner end?

This can be easily reproduced on 24.04 system outside of actions and outside of azure devops agents just by running the following (the below is from my physical box running vanilla ubuntu 24.04):

mmrazik@nibbler:~$ docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=YourStrong!Passw0rd' -p 1433:1433 --name sqlserver2017 mcr.microsoft.com/mssql/server:2017-CU13-ubuntu
This program has encountered a fatal error and cannot continue running.
The following diagnostic information is available:

       Reason: 0x00000003
      Message: mappedBase == address
   Stacktrace: 00005df4150abd55 00005df4150ac975 00005df4150ac580
               00005df415062c36
      Process: 7 - sqlservr
       Thread: 77 (application thread 0x1108)
  Instance Id: e52f9f0e-ac2f-466c-a0d6-cbbe6b7981eb
     Crash Id: 27132938-6938-4f95-8a7d-38842c02efe1
  Build stamp: 7d599fe53e35b5a1b0c8a5e4185d8b7334e01a8c5fa77540415502a85f37ef27

Capturing core dump and information...
dmesg: read kernel buffer failed: Operation not permitted
No journal files were found.
No journal files were found.
Attempting to capture a dump with paldumper
mmrazik@nibbler:~$

If something has changed it has changed in the upstream ubuntu.

Note that this is 2017 (or 2019) mssql (i.e. very old). The 2017 mssql image is based on ubuntu 16.04 which is long unsupported. If you need to run software this old then I would strongly recommend NOT to use docker which shares kernel with the host OS (and in this case it is a very new one compared to the base image) but rather rely on full virtualization and running mssql in operating system for which mssql was built (in this case it seems like 16.04). I would not expect that mssql (or any other software) from 2017 will be running on latest operating systems. Sooner or later it will break.

[HofmeisterAn]

Hi @HofmeisterAn Could you please try with the below command to start the MSSQL containers. https://github.com/RaviAkshintala/testcontainers-dotnet/actions/runs/10993413941/workflow#L38

It’s working with 2019-latest. According to this comment the lowest working version is 2019-CU26-ubuntu-20.04. As @mmrazik mentioned, the issue is reproducible, not only for hosted agents and runners. I’ve seen it fail on other environments recently as well. It’s most likely related to some kernel changes. I don’t think we can provide backward compatibility for the older versions (CUs) again, but we can keep this issue as a reference for developers who run into it. Having an overview of which CUs work on which agent or runner would be helpful.

[RaviAkshintala]

Hi @HofmeisterAn Could you please try using the new image version 20241006.1 It might resolve your issue.

[HofmeisterAn]

I can confirm that the image runs again with 20241006.1. Oddly, it even works with ubuntu-latest, but not with ubuntu-24.04, which refers to the latest version, IIRC. Great that you figured out the issue 😎 May I ask what it was?

[RaviAkshintala]

Hi @HofmeisterAn

In the previous Ubuntu 22 version, the kernel was downgraded to version 6.5.* We are closing this issue. Thanks