fix(test) switch zebrad to a non-blocking tracing logger

[arya2]

Motivation

We want to switch zebrad to a non-blocking tracing logger to use a separate thread when writing tracing logs to stdout so that our tests won't deadlock.

Fixes #4834.

Solution

• Provide tracing FmtSubscriber (or fmt::Layer when tokio-console feature is enabled) with a lossy non-blocking writer to use instead of stdout

Review

Anyone can review this PR

Reviewer Checklist

• Tests for Expected Behaviour

[codecov]

Codecov Report

Merging #5032 (3e203d7) into main (ceff590) will increase coverage by 0.06%.
The diff coverage is 41.93%.

@@            Coverage Diff             @@
##             main    #5032      +/-   ##
==========================================
+ Coverage   79.27%   79.34%   +0.06%     
==========================================
  Files         310      310              
  Lines       38907    39108     +201     
==========================================
+ Hits        30843    31029     +186     
- Misses       8064     8079      +15     

[teor2345]

This looks great!

I just want to double-check a few edge cases. Let me know if you have any questions, or you'd like me to help with the code changes.

[teor2345]

I noticed that some tests were failing, so I pushed a commit that shows the error they are failing with.

[teor2345]

It looks like two other tests are failing with these changes, but only on macOS:

• restart_stop_at_height
• sync_one_checkpoint_mainnet

These are the tests that exit() abruptly, and don't drop the worker guard.

Here are some things to try:

• make the test-only state code sleep for a few seconds before calling exit()
• make the test-only state code panic instead of exiting (this could be complicated, I'm not sure if we propagate panics correctly from that far inside the state)

[teor2345]

Here are some things to try:

If we can't fix it quickly, it's ok to disable these tests on macOS. It's a tier 2 platform, so we don't guarantee test coverage.

[teor2345]

Thanks for fixing those tests, let's go!