mgmt/mcumgr/lib: Parasitic use of CONFIG_HEAP_MEM_POOL_SIZE in image management

[de-nordic]

Describe the bug
The mcumgr image management needs to allocate context flash_img_context that holds several pieces of information, including quite significant in size flash image buffer, controller by other Kconfig option CONFIG_IMG_BLOCK_BUF_SIZE.
This context can either be allocated as static variable at compile time or from heap.
The problem is that a user does not have control over where the buffer is actually allocated, and when CONFIG_HEAP_MEM_POOL_SIZE is set to non-0 the mcumgr will automatically switch to using the HEAP.
Because this is not mentioned anywhere in Kconfig and the relation is non obvious it may be surprising to the user that perfectly working application stops working when mcumgr is enabled within it or opposite, the working smp_svr no longer wants to work when added to an application.

To Reproduce
Steps to reproduce the behavior:

1. Build smp_svr with CONFIG_HEAP_MEM_POOL_SIZE smaller or equal to CONFIG_IMG_BLOCK_BUF_SIZE for the smp_svr image management to no longer work.

Attempt to upload image will finish with error 1 (UNKNOWN) being returned to the mcumgr client and uploads will no longer work.

Expected behavior
The connection between enabling CONFIG_HEAP_MEM_POOL_SIZE and mcumgr image management switching to use the HEAP should be both clearly indicated in Kconfig and documentations, and decision upon switching to HEAP should not be based on CONFIG_HEAP_MEM_POOL_SIZE >0 but, due to the significant size of the context, should be clearly left to the user with addition of a Kconfig option to control such switch.

Impact
Annoyance: increased pressure on heap at runtime leads to failed image upload attempts, changes to the Zephyr that impact heap allocation make applications that use to work completely suddenly not being able to perform image uploads; merging of an application that does use heap with smp_svr that does not seem to be ding so, by the project configuration, suddenly makes either stop working.

Logs and console output

Environment (please complete the following information):
Zephyr: 104a668

Additional context
Yey, it triggers #44219

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.