[Kubernetes] Cannot get the server running or probe is failing

[clempat]

I have difficulties to have my server running. The Kubernetes probe is failing:

Startup probe failed: dial tcp 10.52.190.75:2456: connect: connection refused

I am running it with Kubernetes and the pod is starting on a node with the following kernel/arch:

Kernel Version:
6.2.8-060208-generic
OS/Arch:
linux/amd64

  Resource            Requests         Limits
  --------            --------         ------
  cpu                 1905m (47%)      400m (10%)
  memory              9299232Ki (28%)  22306944Ki (68%)
  ephemeral-storage   0 (0%)           0 (0%)
  hugepages-1Gi       0 (0%)           0 (0%)
  hugepages-2Mi       0 (0%)           0 (0%)
  gpu.intel.com/i915  1                1

My configuration (changing a lot as I am trying to debug): https://github.com/clempat/home-ops/blob/main/kubernetes/games/valheim/values.yaml

Some logs

[Valheim][root] : ###########################################################################
[Valheim][root] : Valheim Server - Mon Apr 10 07:33:51 PM Europe 2023
[Valheim][root] : Initializing your container...
[Valheim][root] : You must be in development. Good luck!
[Valheim][root] : ###########################################################################
[Valheim][root] : Switching UID and GID
[Valheim][root] : usermod: no changes
[Valheim][root] :
Configuring Preset Env
Preset Env Configured
[Valheim][root] : Setting up file systems
[Valheim][root] : Navigating to steam home...
[Valheim][root] : Launching as steam...
[Valheim][steam]: ###########################################################################
[Valheim][steam]: Valheim Server - Mon Apr 10 07:33:51 PM Europe 2023
[Valheim][steam]: STEAM_UID 1000 - STEAM_GUID 1000
[Valheim][steam]: Installing Valheim via odin 1.5.3...
[Valheim][steam]: ###########################################################################
[Valheim][steam]: Variables loaded.....
[Valheim][steam]: Port: 2456
[Valheim][steam]: Name: (REDACTED)
[Valheim][steam]: World: (REDACTED)
[Valheim][steam]: Public: false
[Valheim][steam]: With Crossplay:
[Valheim][steam]: Password: (REDACTED)
[Valheim][steam]: Running Install...
[Valheim][steam]: Attempting to update before launching the server!
[Valheim][steam]: Installing Updates...
[ODIN][INFO]  - Installing 896660 to /home/steam/valheim
[ODIN][INFO]  - steamcmd found in path
ln: failed to create symbolic link '/home/steam/.steam/root': No such file or directory
ln: failed to create symbolic link '/home/steam/.steam/steam': No such file or directory
Redirecting stderr to '/home/steam/Steam/logs/stderr.txt'
ILocalize::AddFile() failed to load file "public/steambootstrapper_english.txt".
[  0%] Checking for available update...
[----] Downloading update (0 of 57,748 KB)...
[  0%] Downloading update (0 of 57,748 KB)...
[  0%] Downloading update (0 of 57,748 KB)...
[  0%] Downloading update (2,683 of 57,748 KB)...
[  4%] Downloading update (6,062 of 57,748 KB)...
[ 10%] Downloading update (8,331 of 57,748 KB)...
[ 14%] Downloading update (10,745 of 57,748 KB)...
[ 18%] Downloading update (12,878 of 57,748 KB)...
[ 22%] Downloading update (15,041 of 57,748 KB)...
[ 26%] Downloading update (17,474 of 57,748 KB)...
[ 30%] Downloading update (19,628 of 57,748 KB)...
[ 33%] Downloading update (21,623 of 57,748 KB)...
[ 37%] Downloading update (23,633 of 57,748 KB)...
[ 40%] Downloading update (25,834 of 57,748 KB)...
[ 44%] Downloading update (27,893 of 57,748 KB)...
[ 48%] Downloading update (29,976 of 57,748 KB)...
[ 51%] Downloading update (32,332 of 57,748 KB)...
[ 55%] Downloading update (34,462 of 57,748 KB)...
[ 59%] Downloading update (36,504 of 57,748 KB)...
[ 63%] Downloading update (38,763 of 57,748 KB)...
[ 67%] Downloading update (40,355 of 57,748 KB)...
[ 69%] Downloading update (41,557 of 57,748 KB)...
[ 71%] Downloading update (42,533 of 57,748 KB)...
[ 73%] Downloading update (43,569 of 57,748 KB)...
[ 75%] Downloading update (44,657 of 57,748 KB)...
[ 77%] Downloading update (45,723 of 57,748 KB)...
[ 79%] Downloading update (46,797 of 57,748 KB)...
[ 81%] Downloading update (47,911 of 57,748 KB)...
[ 82%] Downloading update (48,883 of 57,748 KB)...
[ 84%] Downloading update (49,837 of 57,748 KB)...
[ 86%] Downloading update (51,031 of 57,748 KB)...
[ 88%] Downloading update (52,078 of 57,748 KB)...
[ 90%] Downloading update (53,146 of 57,748 KB)...
[ 92%] Downloading update (54,430 of 57,748 KB)...
[ 94%] Downloading update (55,598 of 57,748 KB)...
[ 96%] Downloading update (56,764 of 57,748 KB)...
[ 98%] Downloading update (57,748 of 57,748 KB)...
[100%] Downloading update (57,748 of 57,748 KB)...
[100%] Download Complete.
[----] Applying update...
[----] Extracting package...
[----] Extracting package...
[----] Extracting package...
[----] Extracting package...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Cleaning up...
[----] Update complete, launching...
Redirecting stderr to '/home/steam/Steam/logs/stderr.txt'
/tmp/dumps is not owned by us - delete and recreate
Unable to delete /tmp/dumps. Continuing anyway.
[  0%] Checking for available updates...
[----] Verifying installation...
[  0%] Downloading update...
[  0%] Checking for available updates...
[----] Download complete.
[----] Extracting package...
[----] Extracting package...
[----] Extracting package...
[----] Extracting package...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Cleaning up...
[----] Update complete, launching Steamcmd...
Redirecting stderr to '/home/steam/Steam/logs/stderr.txt'
/tmp/dumps is not owned by us - delete and recreate
Unable to delete /tmp/dumps. Continuing anyway.
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation - version 1679680174
-- type 'quit' to exit --
Loading Steam API...OK

Connecting anonymously to Steam Public...OK
Waiting for client config...OK
Waiting for user info...OK
Success! App '896660' already up to date.
[ODIN][INFO]  - Successfully installed Valheim!
[Valheim][steam]: Initializing Variables....
[ODIN][INFO]  - Successfully written /home/steam/valheim/config.json
[Valheim][steam]: Checking for TYPE flag
[Valheim][steam]: Found Type Vanilla
[Valheim][steam]: Running with Vanilla Valheim <3
[Valheim][steam]: Starting server...
[ODIN][INFO]  - Setting up start scripts...
[ODIN][INFO]  - Looking for burial mounds...
[ODIN][INFO]  - Server has been started and Daemonized. It should be online shortly!
[ODIN][INFO]  - Keep an eye out for 'Game server connected' in the log!
[ODIN][INFO]  - (this indicates its online without any errors.)
[Valheim][steam]: Herding Graydwarfs...
==> /home/steam/valheim/logs/valheim_server.log <==
WARNING: Shader If subshaders removal was intentional, you may have forgotten turning Fallback off?
ERROR: Shader TextMeshPro/Distance Field (Surface) shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader TextMeshPro/Mobile/Distance Field shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
WARNING: Shader Unsupported: 'TextMeshPro/Distance Field' - All subshaders removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
WARNING: Shader If subshaders removal was intentional, you may have forgotten turning Fallback off?
ERROR: Shader TextMeshPro/Distance Field shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
WARNING: Shader Unsupported: 'TextMeshPro/Distance Field' - All subshaders removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
WARNING: Shader If subshaders removal was intentional, you may have forgotten turning Fallback off?

==> /home/steam/valheim/logs/valheim_server.err <==

==> /home/steam/valheim/logs/output.log <==

==> /home/steam/valheim/logs/auto-update.out <==

==> /home/steam/valheim/logs/auto-backup.out <==

==> /home/steam/valheim/logs/valheim_server.log <==
Unloading 6 Unused Serialized files (Serialized files now loaded: 0)
UnloadTime: 6.875070 ms
Only custom filters can be played. Please add a custom filter or an audioclip to the audiosource (Amb_MainMenu).
HDR Render Texture not supported, disabling HDR on reflection probe.
04/10/2023 19:34:18: Setting -savedir to: /home/steam/.config/unity3d/IronGate/Valheim
04/10/2023 19:34:18: Get create world berlin
04/10/2023 19:34:18: Using environment steamid 892970
04/10/2023 19:34:18: Using steam APPID:892970

==> /home/steam/valheim/logs/valheim_server.err <==
[S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.
CAppInfoCacheReadFromDiskThread took 2 milliseconds to initialize
Setting breakpad minidump AppID = 892970
[S_API FAIL] Tried to access Steam interface SteamNetworkingUtils003 before SteamAPI_Init succeeded.

==> /home/steam/valheim/logs/valheim_server.log <==
04/10/2023 19:34:18: Server ID 90071992547409920
04/10/2023 19:34:18: Authentication:k_ESteamNetworkingAvailability_Waiting
04/10/2023 19:34:18: Steam game server initialized
04/10/2023 19:34:18: Using default prefs
04/10/2023 19:34:18: Valheim version:0.214.305
04/10/2023 19:34:18: Worldgenerator version setup:2
ERROR: Shader TextMeshPro/Sprite shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
04/10/2023 19:34:18: Render threading mode:SingleThreaded
04/10/2023 19:34:18: Missing audio clip in music respawn
04/10/2023 19:34:18: Builder started
04/10/2023 19:34:18: Loaded localization file #0 - 'localization' language: 'English'
04/10/2023 19:34:18: Loaded localization file #1 - 'localization_extra' language: 'English'
04/10/2023 19:34:18: Sending PlayFab login request (attempt 1)
04/10/2023 19:34:18: Login postponed until ID has been set.
The shader Hidden/Dof/DepthOfFieldHdr (UnityEngine.Shader) on effect Main Camera (UnityStandardAssets.ImageEffects.DepthOfField) is not supported on this platform!
The image effect Main Camera (UnityStandardAssets.ImageEffects.DepthOfField) has been disabled as it's not supported on the current platform.
The shader Hidden/SunShaftsComposite (UnityEngine.Shader) on effect Main Camera (UnityStandardAssets.ImageEffects.SunShafts) is not supported on this platform!
The shader Hidden/SimpleClear (UnityEngine.Shader) on effect Main Camera (UnityStandardAssets.ImageEffects.SunShafts) is not supported on this platform!
The image effect Main Camera (UnityStandardAssets.ImageEffects.SunShafts) has been disabled as it's not supported on the current platform.
04/10/2023 19:34:18: Starting music menu
04/10/2023 19:34:19: UI Group status changed Menu = True

Unloading 17 unused Assets to reduce memory usage. Loaded Objects now: 137740.
Total: 103.353329 ms (FindLiveObjects: 5.555673 ms CreateObjectMapping: 17.603561 ms MarkObjects: 79.900318 ms  DeleteObjects: 0.293279 ms)

ERROR: Shader TextMeshPro/Mobile/Distance Field shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/RadialCutoutShader shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
WARNING: Shader Unsupported: 'TextMeshPro/Distance Field' - All subshaders removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
WARNING: Shader If subshaders removal was intentional, you may have forgotten turning Fallback off?
ERROR: Shader TextMeshPro/Distance Field shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
WARNING: Shader Unsupported: 'TextMeshPro/Distance Field' - All subshaders removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
WARNING: Shader If subshaders removal was intentional, you may have forgotten turning Fallback off?
Unloading 6 Unused Serialized files (Serialized files now loaded: 1)
UnloadTime: 12.560851 ms
HDR Render Texture not supported, disabling HDR on reflection probe.
HDR Render Texture not supported, disabling HDR on reflection probe.
04/10/2023 19:34:19: Using default prefs
04/10/2023 19:34:19: isModded: False
04/10/2023 19:34:19: Zonesystem Awake 407
04/10/2023 19:34:19: DungeonDB Awake 407
04/10/2023 19:34:19: Registering lobby
04/10/2023 19:34:19: Worldgenerator version setup:2
04/10/2023 19:34:21: Audioman already exist, destroying self

Unloading 91 unused Assets to reduce memory usage. Loaded Objects now: 142432.
Total: 124.369183 ms (FindLiveObjects: 5.567625 ms CreateObjectMapping: 18.589300 ms MarkObjects: 99.405513 ms  DeleteObjects: 0.806293 ms)

04/10/2023 19:34:26: Zonesystem Start 407
04/10/2023 19:34:27: Added 3 locations, 0 vegetations, 0 environments, 0 biome env-setups, 0 clutter  from locations_cp1
04/10/2023 19:34:27: Added 1 locations, 0 vegetations, 1 environments, 0 biome env-setups, 0 clutter  from locations_mountaincaves
04/10/2023 19:34:27: Added 27 locations, 25 vegetations, 5 environments, 1 biome env-setups, 3 clutter  from locations_mistlands
04/10/2023 19:34:27: DungeonDB Start 407
04/10/2023 19:34:27: ZRpc timeout set to 30s
04/10/2023 19:34:27: Load world: berlin (berlin)
04/10/2023 19:34:27: Loading 81 zdos , my id 2917135798 data version:29
04/10/2023 19:34:27: Loaded 0 dead zdos
04/10/2023 19:34:27: Removed 0 OLD generated ZDOS
04/10/2023 19:34:27: Loaded 10039 locations
04/10/2023 19:34:27: Unloading unused assets
Unloading 10 Unused Serialized files (Serialized files now loaded: 1)
The shader Hidden/Dof/DepthOfFieldHdr (UnityEngine.Shader) on effect Main Camera (UnityStandardAssets.ImageEffects.DepthOfField) is not supported on this platform!
The image effect Main Camera (UnityStandardAssets.ImageEffects.DepthOfField) has been disabled as it's not supported on the current platform.
The shader Hidden/SunShaftsComposite (UnityEngine.Shader) on effect Main Camera (UnityStandardAssets.ImageEffects.SunShafts) is not supported on this platform!
The shader Hidden/SimpleClear (UnityEngine.Shader) on effect Main Camera (UnityStandardAssets.ImageEffects.SunShafts) is not supported on this platform!
The image effect Main Camera (UnityStandardAssets.ImageEffects.SunShafts) has been disabled as it's not supported on the current platform.
04/10/2023 19:34:27: Game server connected

Unloading 10 unused Assets to reduce memory usage. Loaded Objects now: 1661229.
Total: 599.029311 ms (FindLiveObjects: 97.864339 ms CreateObjectMapping: 41.163613 ms MarkObjects: 458.968857 ms  DeleteObjects: 1.032130 ms)

[Valheim][steam]: Halting server! Received interrupt!
[ODIN][INFO]  - Stopping server /home/steam/valheim
[ODIN][INFO]  - Scanning for Valheim process
[ODIN][INFO]  - Found Process with pid 146! Sending Interrupt!
[ODIN][INFO]  - Process signal interrupt sent successfully!
Executing Unity Signal Handler

04/10/2023 19:36:20: Game - OnApplicationQuit
04/10/2023 19:36:20: Shuting down
04/10/2023 19:36:20: ZNet Shutdown
04/10/2023 19:36:20: clone 1
04/10/2023 19:36:20: World save writing starting
04/10/2023 19:36:20: World save writing started
04/10/2023 19:36:20: Saved 81 zdos
04/10/2023 19:36:20: World save writing finishing
04/10/2023 19:36:20: World save writing finished
04/10/2023 19:36:20: Considering autobackup. World time: 111.18, short time: 7200, long time: 43200, backup count: 4
04/10/2023 19:36:20: Skipping backup. World session not long enough.
04/10/2023 19:36:20: World saved ( 63.261ms )
04/10/2023 19:36:20: Considering autobackup. World time: 111.18, short time: 7200, long time: 43200, backup count: 4
04/10/2023 19:36:20: Skipping backup. World session not long enough.
04/10/2023 19:36:20: Unloading unused assets
Unloading 1 Unused Serialized files (Serialized files now loaded: 1)
04/10/2023 19:36:20: Sending disconnect msg
04/10/2023 19:36:20: Disposing socket
04/10/2023 19:36:20: Stopping listening socket
04/10/2023 19:36:20: Last socket, unregistering callback
04/10/2023 19:36:20: ZSteamSocket  UnregisterGlobalCallbacks, existing sockets:0

Unloading 1 unused Assets to reduce memory usage. Loaded Objects now: 1662040.
Total: 593.318959 ms (FindLiveObjects: 111.948535 ms CreateObjectMapping: 42.581294 ms MarkObjects: 437.893846 ms  DeleteObjects: 0.894154 ms)

Setting up 2 worker threads for Enlighten.
  Thread -> id: 7fd6affff640 -> priority: 1
  Thread -> id: 7fd751500640 -> priority: 1
04/10/2023 19:36:23: Steam manager on destroy
04/10/2023 19:36:24: ZNet OnDestroy
04/10/2023 19:36:24: Net scene destroyed

==> /home/steam/Steam/logs/stderr.txt <==
Installing breakpad exception handler for appid(steamcmd)/version(1679680174)/tid(142)
Installing breakpad exception handler for appid(steamcmd)/version(1679680174)/tid(143)
Installing breakpad exception handler for appid(steamcmd)/version(1679680174)/tid(144)
Installing breakpad exception handler for appid(steamcmd)/version(1679680174)/tid(145)
Installing breakpad exception handler for appid(steamcmd)/version(1679680174)/tid(146)
Installing breakpad exception handler for appid(steamcmd)/version(1679680174)/tid(147)
Installing breakpad exception handler for appid(steamcmd)/version(1679680174)/tid(148)
Installing breakpad exception handler for appid(steamcmd)/version(1679680174)/tid(150)
Installing breakpad exception handler for appid(steamcmd)/version(1679680174)/tid(151)
Installing breakpad exception handler for appid(steamcmd)/version(1679680174)/tid(152)

[clempat]

Ok, as far I could go, the probes are the issue and killing the container. I will still investigate to help anyone to run this container with Kubernetes.

[clempat]

Still not the solution for now I activated the light HTTP server with HTTP_PORT and do the probe against it so the container does not get destroy.

I imagine I have some network stuff going on. The HTTP server is showing me offline, but I checked I can join it using my public ip.

[clempat]

For this last point, I know that the issue is the following:

Another note: your server MUST be public (eg. PUBLIC=1) in order for Odin+Huginn to collect and report statistics

[mbround18]

@clempat is this resolved?