Add post for Waku v1 vs Waku v2 bandwidth comparison

[jm-clius]

I've converted the bandwidth comparison between Waku v1 and Waku v2 into a research post. Th original post can still be viewed here and in the Vac Book.

[D4nte]

Sorry won't have time to review.
The original post was great so happy to see it being pushed to the blog.

[staheri14]

Nice job! Looks good to me!
I have left a few questions and suggestions.

[staheri14]

Average bandwidth usage (for the same message rate) has remained roughly the same for Waku v2

Do you mean in comparison with the 30 nodes scenario?

[jm-clius]

Yes! Thanks, should be clarified.

[jm-clius]

Done in 21749a4

[staheri14]

Curious to know why the bandwidth consumption of waku v2 has decreased from >2000 kbps in 50 nodes case to almost 2000 kbps in 80 nodes?

[jm-clius]

Yeah, the quirks of real world experiments. :) Expectation is that it should be similar (barring normal, slight variations between different runs) to the smaller network size. In this instance my local system struggled to maintain throughput and my guess is that effective throughput also fell for Waku v2. That said, I significantly reconfigured my experimental system in order to achieve the 150 nodes scenario below. I think it's a good idea to rerun this test and see if I get more reasonable (expected) results. Thanks!

[staheri14]

I think it is useful to also see the non-normalized comparison as well, especially, to better understand waku v2 behavior for different network sizes

[staheri14]

In order to better formalize the bit complexity per second against the network size, I'd suggest fitting a polynomial into the data points of waku v2 (and waku v1). we can also use the resultant polynomial to approximately predict the bandwidth usage of larger network sizes (even without simulation).

[jm-clius]

I think it is useful to also see the non-normalized comparison as well, especially, to better understand waku v2 behavior for different network sizes

True. I do think though that the experimental system (blackbox, local system, external bandwidth monitoring tool with limited capability) is not a good characterisation of Waku v2 (or v1) bandwidth, but rather a fair comparison of the two. To accurately characterise Waku v2 bandwidth for different networks I'd like to add internal measurements, also to understand what the bandwidth is being used by (control messages, discovery, message publishing, connection setup, etc.). Watch this space! :)

[staheri14]

I do think though that the experimental system (blackbox, local system, external bandwidth monitoring tool with limited capability) is not a good characterisation of Waku v2 (or v1) bandwidth, but rather a fair comparison of the two. To accurately characterise Waku v2 bandwidth for different networks I'd like to add internal measurements, also to understand what the bandwidth is being used by (control messages, discovery, message publishing, connection setup, etc.). Watch this space! :)

Makes sense! looking forward to that :)

[staheri14]

is this average per message? or per node? or both?

[jm-clius]

Meant as in per node, but it would hold for per message bandwidth too.

[staheri14]

Might be good to clarify this in the text as well, that the reported value is the average bandwidth each node consumes per message.

[staheri14]

BTW, I just revisited the figures and I think the average might be per node but not per message. The reason is that the bandwidth usage has been dropped from >2000kpbs in 85 nodes case to below 1000kpbs in a network of 150 nodes (where the messaging rate has also dropped from 50 to 35). Normally, I would expect to see the same bandwidth usage (based on the trend in prior figures), but it seems there is dependency between bandwidth and the messaging rate, what do you think?

[jm-clius]

Ah, I understand your question a bit better now. Yes, if we were to normalise the bandwidth to per message we would not see much variation if we also tweaked the number of nodes generating traffic or the number of content topics. However, the actual scenario results are certainly per node, which is heavily dependent on the message rate - the higher the message rate, the higher the average bandwidth per node.

[staheri14]

As a user of waku, I would be also interested to see the bandwidth overhead per node and per message i.e, I would ask how much bandwidth each node has to spend for the transmission of one message to the entire network. In another set of simulations, the impact of message rate on bandwidth can be examined for scenarios with identical setup but varying messaging rates.

In the current PR, since the objective is to provide a comparative analysis of waku v1 with v2, I think the average per node is fine. But for the follow-up simulations that you mentioned (to characterize waku v2), I'd suggest having per-message results as well.

[jm-clius]

Agree! I also think to answer this question we'll need to add better internal metrics to validate what the external tool returns.

[staheri14]

does the elapsed time indicate the total time required for one round to finish? one round is completed when all the 10 nodes finish publishing 50 messages per second

[jm-clius]

No, the nodes maintain a constant rate of 50 messages per second for the full elapsed time. The fact that both networks (v1 and v2) shows more or less constant bandwidth usage for this constant propagation rate is an indication that both protocols can maintain stable meshes.

[staheri14]

I see, Thanks for the clarification!

[oskarth]

Nice one! Some comments

[oskarth]

Some suggestions re intro and outro to make it more like a dynamic article, otherwise LGTM

[oskarth]

Looks great!