UCT/API: introduce sys_dev field for mem_alloc #10448
Conversation
Akshay-Venkatesh
requested review from
yosefe,
brminich,
rakhmets and
tvegas1
| UCT_MEM_ALLOC_PARAM_FIELD_NAME = UCS_BIT(4) | ||
| UCT_MEM_ALLOC_PARAM_FIELD_NAME = UCS_BIT(4), | ||
|
|
||
| /** Enables @ref uct_mem_alloc_params_t::sys_device */ |
There was a problem hiding this comment.
it is called sys_dev in uct_mem_alloc_params_t. Pls align one of them with the other.
also UCT_MEM_ALLOC_PARAM_FIELD_SYS_DEVICE
There was a problem hiding this comment.
UCT_MEM_ALLOC_PARAM_FIELD_SYS_DEV
| @@ -101,12 +102,15 @@ ucs_status_t uct_mem_alloc(size_t length, const uct_alloc_method_t *methods, | |||
| params->name : "anonymous-uct_mem_alloc"; | |||
| mem_type = (params->field_mask & UCT_MEM_ALLOC_PARAM_FIELD_MEM_TYPE) ? | |||
| params->mem_type : UCS_MEMORY_TYPE_HOST; | |||
| sys_dev = (params->field_mask & UCT_MEM_ALLOC_PARAM_FIELD_SYS_DEVICE) ? | |||
| params->sys_dev : UCS_SYS_DEVICE_ID_UNKNOWN; | |||
There was a problem hiding this comment.
whould we return INVALID_PARAM if sys_dev is not UCS_SYS_DEVICE_ID_UNKNOWN and mem_type is host?
There was a problem hiding this comment.
@brminich This assumes that GPU MDs cannot allocate host memory which is true today but maybe it is better for UCT MDs to return that error than catch it here?
There was a problem hiding this comment.
if you think it can be a valid case in the future, then ok
src/uct/api/uct.h
Outdated
| * Index of the system device on which memory is to be allocated. | ||
| * Eg: UCS_SYS_DEVICE_ID_UNKNOWN to allocate on host memory, or a specific | ||
| * index to allocate memory on GPU device. |
There was a problem hiding this comment.
Just
"System device on which memory is to be allocated, or UCS_SYS_DEVICE_ID_UNKNOWN to allow allocating on any device".
The host memory comment is misleading because mem_type field is used to select host memory
What? Why?
As we move towards supporting multi-device configurations and relaxing GPU context setting requirements, UCP and other UCT users need to be able to indicate on which system device the allocation is being requested on without having to rely on inferring the same from context set against calling thread. This patch extends allocation params struct to include system device. In this initial implementation all UCT allocators ignore the system device argument but eventually different transports can convert the given system device into a bus_id and allocate against that specific device.