Support affinity/pinning of parallel flows to different CPUs

[marcosfsch]

Enhancement Request

• Current behavior
  iperf3 supports setting affinity to a single CPU, which made sense when it was single threaded

• Desired behavior
  Support setting a rage/list of CPUs to affinity of multiple flows: Ex: "iperf3 -c localhost -P 4 -A 1-4,1-4"

• Implementation notes
  There are at least two way this can be implementated and I'd suggest the second approach:

1. Replicate numactl behavior, where "numactl -C 1-4" would bind each flow to the CPU range "1-4" and Linux would be responsible for the scheduling, allowing for dinamically rebalancing the flows but generating noise and perfromance drops.
2. Statically schedule each flow to a different CPU, i.e. in a round-robin fashion, so that when you have a maching set of parallel threads and CPUs you'd have 1 flow per CPU, minimizing reschedules during the transfer and optimizing performance

Another note, is that ideally you should be able to explicitly define a CPU list, which normally uses comma as a separator, i.e. "1,3,5,7". But this would impact either in I) changing the current client/server CPU separator ("iperf3 -c localhost -P 4 -A 1,3,5,7/2,4,6,8") or to use a different delimiter character for defining the list.

[bmah888]

You raise a good point in that the current -A behavior doesn't work very well in a multi-threaded iperf3. So far our standard practice within ESnet is just to do numactl, as you suggested in the first approach. How bad are the downsides you mentioned...have you or others observed these problems? (I don't think we have, but multithreaded application performance analysis is not my forte.)

In considering these different implementations, we also want to keep in mind other OS platforms that support -A, such as FreeBSD (a supported platform), and Windows (while not officially supported, an environment that I'd like to avoid gratuitously breaking).

[davidBar-On]

Submitted PR #1778 with a suggested multi-CPUs Affinity support.