network-online.target is inactive breaking many dependent services

[hpaluch]

Describe the bug

While trying sample WordPress project from https://github.com/major/quadlets-wordpress I found that systemctl --user ... tasks were stuck because there is service podman-user-wait-network-online.service waiting for systemctl is-active network-online.target (see containers/podman#24796 (comment)).

But on FCOS network-online.target never becomes online - which breaks all dependent services and is bug.

Reproduction steps

1. install latest stable FCOS, tested fedora-coreos-41.20241122.3.0-live.x86_64.iso, inside is OSTREE_VERSION='41.20241122.3.0' using any kind of Ignition - trivial one with just SSH key is OK
2. Run systemctl is-active network-online.target
3. It will produce: inactive

Expected behavior

Command systemctl is-active network-online.target should produce active

Actual behavior

But systemctl is-active network-online.target produces inactive instead.

System details

• tested both fedora-coreos-41.20241122.3.0-live.x86_64.iso using small part from https://www.portainer.io/blog/from-zero-to-production-with-fedora-coreos-portainer-and-wordpress-in-7-easy-steps (python webserver to provide Ignition file to booted ISO)
• also tested fedora-coreos-41.20241122.3.0-qemu.x86_64.qcow2 using https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-libvirt/

Here are more details:

rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Fri 2024-12-27 07:28:28 UTC)
Deployments:
● fedora:fedora/x86_64/coreos/stable
                  Version: 41.20241122.3.0 (2024-12-16T21:04:43Z)
                   Commit: c81a94e71f7bbb4a7eaf1aa2162183d1c63fda4e5b7fb8a3054f42ff9607d8a1
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1

  fedora:fedora/x86_64/coreos/stable
                  Version: 41.20241122.3.0 (2024-12-16T21:04:43Z)
                   Commit: c81a94e71f7bbb4a7eaf1aa2162183d1c63fda4e5b7fb8a3054f42ff9607d8a1
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1

Butane or Ignition config

variant: fcos
version: 1.5.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - 'ssh-ed25519 ...'

Additional information

Workaround: enable iscsi service:

$ sudo systemctl enable --now iscsi
$ systemctl is-active network-online.target

active

I found failing service using (from containers/podman#24796 (comment)):

$ systemctl list-dependencies --reverse  network-online.target

network-online.target
○ ├─docker.service
○ ├─fwupd-refresh.service
○ └─iscsi.service

And evaluating systemctl status ... for each service in tree.

So there are 2 possible solutions:

1. Enable iscsi.service as default
2. or remove hard-dependency on iscsi.service from network-online.target

[hpaluch]

Looking at related issues I'm not sure if it is podman bug or network-online.target bug. User level service
is

$ systemctl --user status
● localhost.localdomain
    State: starting
    Units: 137 loaded (incl. loaded aliases)
     Jobs: 8 queued
   Failed: 0 units
    Since: Fri 2024-12-27 08:50:33 UTC; 1min 1s ago
  systemd: 256.8-1.fc41
  Tainted: unmerged-bin
   CGroup: /user.slice/user-1000.slice/[email protected]
           ├─app.slice
           │ └─podman-user-wait-network-online.service
           │   ├─1512 sh -c "until systemctl is-active network-online.target; do sleep 0.5; done"
           │   └─1888 sleep 0.5
           └─init.scope
             ├─1475 /usr/lib/systemd/systemd --user
             └─1484 "(sd-pam)"

And file/package is:

$ rpm -qf /usr/lib/systemd/user/podman-user-wait-network-online.service 
podman-5.3.0-1.fc41.x86_64

[jbtrystram]

That's weird, I have been ordering my quadlets on network-online.target for a while on my FCOS host without issues.
But not user-level ones.
Can you try with a dummy service (without podman being involved) ?

[hpaluch]

That's weird, I have been ordering my quadlets on network-online.target for a while on my FCOS host without issues. But not user-level ones. Can you try with a dummy service (without podman being involved) ?

This problem affects non-root containers only - because faulty /usr/lib/systemd/user/podman-user-wait-network-online.service is invoked for non-root containers only.

Root (privileged) containers are not affected by this bug.

[jlebon]

In general, network-online.target is not enabled by default. It must be pulled in by whatever unit wants it, and there are no units in FCOS that are enabled by default that do pull it in. So this sounds more like a podman bug as described in the issue you linked.