BLE throughput of DFU by Mcumgr is too slow

[icephyr]

Is your enhancement proposal related to a problem? Please describe.
Hi guys, I met a problem about the DFU of BLE Chip, hope anyone can help on this issue, thanks.

I tried to upgrade the firmware of nRF52832 running zephyr project, with Mcumgr, the tool under linux. The Mcumgr communicates with BLE chip by BLE connection. The firmware can be successfully transferred via BLE, as shown below:

However，the transmission speed is really too slow, only 2.4 KiB/s as shown in the picture above, and it will cost 1mintue and 3 seconds to transfer the whole firmware file, which will be a disaster when I need to upgrade many many BLE devices.
So I wonder if any methods can help to speed up the throughput of BLE transmission in DFU process. By the way, the zephyr project version I am using is V1.13, not the LTS version yet. Has the problem been resolved in the lastest newest branch ? I will have a try if so.

Thanks.

[aescolar]

@carlescufi Can you help @icephyr ?

[teburd]

@icephyr There are a few Kconfig tweaks you can try to improve BLE transfers in general, not sure if you tried increasing the number and size of the BLE RX bufs and setting the MTU higher but those can change things drastically. It still has to write to flash though, there's no avoiding that.

[carlescufi]

@icephyr there are a lot of factors that can influence that speed. Among them there will also be your Linux host settings, as well as the Kconfig options that were mentioned by @BFrog. There is a throughput sample that you can try as a starting point for configuring the BLE stack for throughput in here.

[icephyr]

Thanks you guys.
As @BFrog and @carlescufi mentioned, I have not tried increasing the number and size of the BLE RX bufs and setting the MTU higher yet, I will have a try first.
So increase the number and size of the BLE RX bufs just on the slave side, while as for setting the MTU higher, the MTU need to be set on both slave side and master side, if I did't misunderstand what you means, right ?

[icephyr]

@icephyr there are a lot of factors that can influence that speed. Among them there will also be your Linux host settings, as well as the Kconfig options that were mentioned by @BFrog. There is a throughput sample that you can try as a starting point for configuring the BLE stack for throughput in here.

Hi @carlescufi , I add the config as you mentioned, but it didn't seem to work, the throughput is about 2.5KB/s.

the configs I added are :

CONFIG_HEAP_MEM_POOL_SIZE=2048
CONFIG_BT_RX_BUF_LEN=258
CONFIG_BT_GATT_CLIENT=y
CONFIG_BT_ATT_TX_MAX=10
CONFIG_BT_ATT_PREPARE_COUNT=2
CONFIG_BT_CONN_TX_MAX=10
CONFIG_BT_L2CAP_TX_BUF_COUNT=10
CONFIG_BT_L2CAP_TX_MTU=247
CONFIG_BT_L2CAP_DYNAMIC_CHANNEL=y
CONFIG_BT_CTLR_PHY=y
CONFIG_BT_CTLR_PHY_2M=y
CONFIG_BT_CTLR_RX_BUFFERS=2
CONFIG_BT_CTLR_TX_BUFFERS=10
CONFIG_BT_CTLR_TX_BUFFER_SIZE=251
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251
CONFIG_BT_CTLR_ADVANCED_FEATURES=y
CONFIG_BT_CTLR_XTAL_THRESHOLD=500000

[icephyr]

Update what I have tried.

I add the configs you mentioned above, on zephyr V1.13.0 version, but it seems not work. The througput is still about 2.5KB/s. I'm not sure if it is normal.
Then I realized that maybe stopping the applications(mainly scanning) before the DFU starts will help on speed up the througput. So I stopped the scanning behaviour of BLE chip before the file transportation by mcumgr, then the throughput reached 5.6 KB/s.
I need more test on the throughput optimization issue and will continue to update the results, thanks you guys.

[carlescufi]

@nwsetec could you please add your observations here, from the email thread?

[nwsetec]

Link to Zephyr mailing lists thread: https://lists.zephyrproject.org/g/users/topic/34540601#1741

After turning off scanning, reducing the BLE connection interval, increasing receive buffer sizes and enabling dynamic length extensions (DLE) I had a similar upper DFU speed of roughly 6kB/s.

I did some packet sniffing logs and shared with @cvinayak and he confirmed that BLE connection was not the bottleneck. @cvinayak also shared a video of Zephyr to Zephyr throughput of ~1.3Mbit/s (162kB/s)

I also spoke with @bgiori (mcumgr Android and iOS libraries) and he confirmed that it's the protocol's round trip time that is likely the current limit on the speed of the DFU upload, i.e. new data is only sent after the DFU sender receives the latest offset response from the DFU receiver.

Also notable is the nRF devices halt the CPU during a flash write which stops BLE use during this time, so DFU's to SoC/internal flash will have somewhat reduced BLE throughput but DFU throughput achievable should still likely be much higher than 6kB/s.

[icephyr]

Thanks you guys, I agree that the upper limit of DFU throughput should still likely be much higher than 6kB/s. I think it's a big issue, since such rate will take very long, especially for massive firmware upgrade, like BLE Mesh OTA. Hope this issue can be solved finally. I will study on the mcumgr round trip mechanism first and will share my results later.

[vikrant8052]

Hi @nwsetec,

After turning off scanning, reducing the BLE connection interval, increasing receive buffer sizes and enabling dynamic length extensions (DLE) I had a similar upper DFU speed of roughly 6kB/s.

Could you please guide me what exact configuration Parameters we have to set in context of samples/boards/nrf52/mesh/onoff_level_lighting_vnd_app to achieve 6KB+ throughput ?
(This app supports out of the box DFU if build with prj_smp_svr.conf)

[nwsetec]

Hi @Vikrant8051,

Apologies for the late reply.

I basically checked my project configuration against this Nordic project sample:
https://github.com/NordicPlayground/fw-nrfconnect-nrf/blob/master/samples/bluetooth/throughput/prj.conf#L28

In addition this is with BLE scanning turned off. If your mesh device is configured as a repeater I'm not sure how you will turn off the listening/scanning for mesh packets (if DFU speed optimisation is more important). I also used the smallest connection interval of 7.5ms but note that this (unfortunately) only worked for some connecting centrals. From memory another connecting v4.0 central device achieved a similar result with only a 15ms connection interval.

[Flo3561]

Hi there,

Are there any new findings? e.g. with new releases 2.0.0 or 2.1.0?

[lukasbrchl]

Hi guys, does anyone have new findings on this issue?

[cvinayak]

I am not an expert on the DFU protocol, but the BLE controller is capable of doing upto ~1.3Mbps of GATT Write commands. With a 1Mbps UART, and features provided in this PR #28089, a throughput of ~690 Kbps should be achievable.

[DMotocus]

@carlescufi Can you help @icephyr ?

⚰️

[carlescufi]

@carlescufi Can you help @icephyr ?

⚰️

There's not that much I can add to the existing thread @DMotocus. It ultimately all boils down to connection interval, GATT MTU and whether DLE and 2Mbit/s PHY is enabled or not. Since we are using a 3rd-party stack to send the image there are too many variables out of our control.

[lukasbrchl]

We were able to get only 1kB/s throughput from common Android smartphones and mcumgr-android library (or nRF Device Manager if you prefer). If anyone finds out how to make it faster, we would be glad to implement it in our solution.

[bgiori]

Hi all, we recently started an effort to improve the image upload speeds by updating the mcumgr client side to support sending multiple concurrent requests. Allowing even 2 concurrent requests has significantly improved upload speeds. Unfortunately I don't have anything concrete about the max upload speeds this new client can achieve, but it should be better than the current send-and-wait regardless.

The original improvement to the newtmgr (mcumgr) cli tool is here: apache/mynewt-newtmgr#162

I am currently working on an implementation for mcumgr-android and eventually mcumgr-ios.

[bgiori]

My initial implementation on android is getting ~24kB/s without any connection parameter tuning besides setting the att mtu to 515

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.

[nobodyguy]

It's definitely the protocol's round trip, I'm getting very similar results (around 2 kB/s) either via serial or BLE.

[philips77]

@bgiori any update on this? A specially iOS side?

[bgiori]

@philips77

Android is done, tag 0.12.0-beta4

iOS still needs an implementation. This is something I don't really have bandwidth for at the moment.

[nobodyguy]

Can anyone please share any current tips and tricks how to get some decent speed?
I was getting around 2kB/s speed last year in my app, I updated Zephyr to 2.7 and tried newtmgr 1.9.0 with apache/mynewt-newtmgr#162 optimization and I'm getting 350 B/s. With the current nRF connect mobile app I'm getting 1.7 kB/s right now.

[de-nordic]

@nobodyguy

You are talking about Ble or UART?
In case of Ble, can you test this: #41306

[nobodyguy]

@nobodyguy

You are talking about Ble or UART? In case of Ble, can you test this: #41306

BLE, I forgot to mention it.
Thank you, I'll give it a try.

[nobodyguy]

@de-nordic

You are talking about Ble or UART? In case of Ble, can you test this: #41306

Tested this on 2.7 branch with nRF Connect Android app and I was still getting 1.7 kB/s. I also tried this together with cherry-picked CONFIG_MCUMGR_SMP_BT_LATENCY_CONTROL support and still no improvement (nrf52832).

[philips77]

nRF Connect for Android doesn't get support latest features. We're working on bringing them first to "nRF Connect Device Manager" app, and will release version 1.1 soon. With SMP reassembly or pipelining enabled we're achieving ~25-50 kB/s. This is with MTU (ATT_MTU) set to 498 and DLE enabled (L2CAP_MTU) and set to 251. Without the new features, but just with those parameters it gives ~10kB/s.

[nobodyguy]

@philips77 That's great to hear, thank you for the clarification.

[nobodyguy]

This is with MTU (ATT_MTU) set to 498 and DLE enabled (L2CAP_MTU) and set to 251. Without the new features, but just with those parameters it gives ~10kB/s.

Can you also share those config options please? Just to verify I didn't miss any.

[DMotocus]

I've been trying to find out what is the best way to confirm the probability of character and the extremely unlikely king's taster theory. I'm speaking of PPE associated with work and the belligerent fraud of network service provider data previously not a issue. My low end hotspot is capable of 400mbps but lowered to 0.5 mbps about the speed of dial up. The big money is in digital download games technically without overhead cost and quality of life issues in contrast to character. It's like someone crimping the hose and wanting correct rates for the correct water pressure. A interesting idea but go ahead and water the lawn yourself. If you don't roll up the hose there will be hell to pay. It's pretty standard.

[zhamaliddin-kalzhan]

@philips77 Can you please share your config file where you achieved 25-50 kB/s?
Did you modify CONFIG_MCUMGR_BUF_SIZE and CONFIG_MCUMGR_BUF_COUNT? In my experience, those 2 parameters also greatly affect image upload speed.

[philips77]

Have a look at this config: https://github.com/nrfconnect/sdk-nrf/blob/2424a2dcef6e91301621dc0d67a3931beeec33b8/samples/bluetooth/peripheral_lbs/boards/thingy53_nrf5340_cpuapp.conf#L21 for Thingy:53
of

zephyr/samples/subsys/mgmt/mcumgr/smp_svr/overlay-bt.conf

Line 18 in 0cd311c

CONFIG_MCUMGR_SMP_REASSEMBLY_BT=y

for SMP Server sample for other boards.

CONFIG_MCUMGR_SMP_REASSEMBLY_BT=y
CONFIG_MCUMGR_BUF_SIZE=2475
CONFIG_OS_MGMT_MCUMGR_PARAMS=y
CONFIG_MCUMGR_BUF_COUNT=4 (default)

CONFIG_BT_L2CAP_TX_MTU=498
CONFIG_BT_BUF_ACL_RX_SIZE=502
CONFIG_BT_BUF_ACL_TX_SIZE=502
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251

[optical-o]

Have a look at this config: https://github.com/nrfconnect/sdk-nrf/blob/2424a2dcef6e91301621dc0d67a3931beeec33b8/samples/bluetooth/peripheral_lbs/boards/thingy53_nrf5340_cpuapp.conf#L21 for Thingy:53 of

zephyr/samples/subsys/mgmt/mcumgr/smp_svr/overlay-bt.conf

Line 18 in 0cd311c

CONFIG_MCUMGR_SMP_REASSEMBLY_BT=y

for SMP Server sample for other boards.

CONFIG_MCUMGR_SMP_REASSEMBLY_BT=y
CONFIG_MCUMGR_BUF_SIZE=2475
CONFIG_OS_MGMT_MCUMGR_PARAMS=y
CONFIG_MCUMGR_BUF_COUNT=4 (default)

CONFIG_BT_L2CAP_TX_MTU=498
CONFIG_BT_BUF_ACL_RX_SIZE=502
CONFIG_BT_BUF_ACL_TX_SIZE=502
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251

Currently these recommended settings will break mcumgr_fs layer due to FS_MGMT_DL_CHUNK_SIZE being to large to transfer. There should be some work on testing the SMP_REASSEMBLY with the mgmt_fs layer. @de-nordic