Not all communicators subscribe to info keys

[devreal]

While reviewing #12189 I found what seems to be a discrepancy in the info key handling between communicators created with different functions. Some communicators subscribe to the info keys, some only copy them. When subscribed, the communicator will get notified of updates when the user calls MPI_Comm_set_info. Otherwise, no such update happens and we don't even parse the info keys. Some time ago we fixed the handling of mpi_assert_allow_overtake so that it can be set once after the communicator has been created (#9843) so that is a valid use-case.

Here is the list of functions that support changing info keys through MPI_Comm_set_info:

• Anything created through ompi_comm_dup_with_info (https://github.com/open-mpi/ompi/blob/main/ompi/communicator/comm.c#L1348) and ompi_comm_split_type_core (incl. ompi_comm_split_with_info, ompi_comm_split_unguided) (https://github.com/open-mpi/ompi/blob/main/ompi/communicator/comm.c#L995): MPI_Comm_dup_with_info, MPI_Comm_dup, MPI_Comm_split_type`, and some han, acoll, and rdma communicator.

The list of functions that do not support it:

• Anything that is created through ompi_comm_create_w_info, which only copies the info objects (https://github.com/open-mpi/ompi/blob/main/ompi/communicator/comm.c#L446): MPI_Comm_create
• Anything that is created through ompi_comm_split_with_info, which only copies the info objects (https://github.com/open-mpi/ompi/blob/main/ompi/communicator/comm.c#L703): MPI_Comm_split (through ompi_comm_split) and internal HAN and acoll communicators.

We should probably harmonize that in some way, i.e., always subscribe to the info keys. For internal communicators that will never have other info keys set we could pass a flag to suppress. However, it's strange that setting mpi_assert_allow_overtake on a communicator created with MPI_Comm_dup would work while it wouldn't work on a communicator created with MPI_Comm_create.

[edgargabriel]

@devreal what about the dynamic communicator creation functions? Spawn, Join and Connect/Accept?

[devreal]

I don't see the subscriber mechanism used in https://github.com/open-mpi/ompi/blob/main/ompi/dpm/dpm.c

[edgargabriel]

Shouldn't it be used however? Comm_spawn and friends take an info object as an argument as well

[devreal]

Yes, it is being used. The info object is queried for various info keys (starting at https://github.com/open-mpi/ompi/blob/main/ompi/dpm/dpm.c#L929) but as far as I can tell they are never added to the communicator so asking for them will not return anything. And of course things like allow-overtake seem to be dropped entirely.

I believe we have to make these keys available for query later so the application can see if we understood them.

[bosilca]

1. We are inconsistent across communicator creation functions. However, all functions that take the info key as an argument are consistent.
2. The current behavior is consistent with the MPI standard requirements. We are free to ignore info keys in which case they should not be returned to the user.
3. The info key for dynamic processing are not related to the communicator. MPI 4.1 Section 11.8 is very clear about this:

a set of key-value pairs telling the runtime system where and how to start the processes (handle, significant only at root)

[edgargabriel]

@bosilca regarding Comm_spawn, MPI4.1 does make the explicit statement that the memory_alloc_kind info object can be passed to COmm_spawn as well, so we might need to add that ability:

In the World Model, an info hint passed to an MPI startup mechanism requests support for memory allocation kinds for all objects derived from the World Model. This info hint can also be supplied to MPI_COMM_SPAWN or MPI_COMM_SPAWN_MULTIPLE in the World Model.

[bosilca]

I have a different understanding of that statement. Yes, the memkind info key applies to the newly created world but not to the intercomm resulting from the spawn call. If it gets attached somewhere, it shall be to the MPI_COMM_WORLD of the spawnees.

[devreal]

The current behavior is consistent with the MPI standard requirements. We are free to ignore info keys in which case they should not be returned to the user.

Yes. We can ignore them everywhere. Or flip a coin and pretend we don't know any of the keys. Neither is what our users expect.

[bosilca]

we're not disagreeing there. I just pointed out that this is not a bug, or a divergence from the MPI standard requirements.

[edgargabriel]

I have a different understanding of that statement. Yes, the memkind info key applies to the newly created world but not to the intercomm resulting from the spawn call. If it gets attached somewhere, it shall be to the MPI_COMM_WORLD of the spawnees.

We should probably bring this up in the hybrid working group to clarify the expected behavior. I am fine either way to be honest, its not like Comm_spawn is used thaat heavily.