scx_utils: Add retryable errors

[hodgesds]

Automatically (try to) create the socket dir if it doesn't exist. Sometimes the server will fail with:

$ sudo ./target/release/scx_layered --run-example --stats 5
thread '<unnamed>' panicked at src/main.rs:1960:74:
called `Result::unwrap()` on an `Err` value: No such file or directory (os error 2)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
21:52:54 [INFO] CPUs: online/possible=16/16 nr_cores=8
21:52:54 [INFO] configuring node 0, LLCs 2
21:52:54 [INFO] configuring llc 0 for node 0
21:52:54 [INFO] configuring llc 1 for node 0
21:52:55 [INFO] Layered Scheduler Attached. Run `scx_layered --monitor` for metrics.
^CEXIT: unregistered from user space

[htejun]

So, in launch(), there's this:

        if let Some(dir) = path.parent() {
            std::fs::create_dir_all(dir).with_context(|| format!("creating {:?}", dir))?;
        }

Shouldn't this be enough? It's doing create_dir_all() on the parent. If you can reproduce the problem, can you find out why that's not working?

[hodgesds]

I think I see the problem in scx_layered, during startup the monitor is called before the launch() is called during the scheduler init. To reproduce I run rm -rf /var/run/scx && scx_layered --run-example --stats 1.

[hodgesds]

Updated by adding more retryable errors. In the case where the server hadn't started the path wouldn't be found so the stats server connection would never happen.

[htejun]

With const array like:

    const RETRYABLE_ERRORS: [std::io::ErrorKind; 2] = [
        std::io::ErrorKind::NotFound,
        std::io::ErrorKind::ConnectionRefused,
    ];

this doesn't have to be global and moved into monitor_stats().

[htejun]

Ah, that makes sense. Thanks for debugging it.