Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Block production summary in 100 node #1876

Merged
merged 55 commits into from
Feb 24, 2020
Merged

Conversation

ailisp
Copy link
Member

@ailisp ailisp commented Dec 18, 2019

image

Output is roughly as above. Each line in each txt above is in format:
block_hash block_height approvals is_chunk_included_all_in_current_height chunk_included_height

@codecov
Copy link

codecov bot commented Dec 18, 2019

Codecov Report

❗ No coverage uploaded for pull request base (staging@c94c555). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##             staging    #1876   +/-   ##
==========================================
  Coverage           ?   86.76%           
==========================================
  Files              ?      176           
  Lines              ?    34153           
  Branches           ?        0           
==========================================
  Hits               ?    29634           
  Misses             ?     4519           
  Partials           ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c94c555...1c15651. Read the comment docs.

@ailisp
Copy link
Member Author

ailisp commented Dec 22, 2019

Test result so far:

block production time 10s, fork after about 4 hours
when check after 6 hours, all node agree validator is ['node17']

block production time 100s: not fork after 12 hours. cpu 100% though

block production time 55s:
doesn't fork after 8 hours, also 100% cpu

block production time 30s:
fork after 6 hours :(

block production time 40s:
fork after 8 hours (probably i should retest 50s or 55s or 60s one, giving it a longer time to see if it forks)

@SkidanovAlex

@bowenwang1996
Copy link
Collaborator

Maybe we are doomed. Which one do you refer to when you talk about block production time? Is it min_block_production_delay, max_block_production_delay, or max_block_wait_delay?

@ailisp
Copy link
Member Author

ailisp commented Dec 22, 2019 via email

@bowenwang1996
Copy link
Collaborator

So min_block_production_delay is 10s, max_block_production_delay is 20s and max_block_wait_delay is 60s?

@ailisp
Copy link
Member Author

ailisp commented Dec 22, 2019 via email

@bowenwang1996
Copy link
Collaborator

That's unfortunate. Is memory usage normal?

@ailisp
Copy link
Member Author

ailisp commented Dec 22, 2019 via email

@ailisp
Copy link
Member Author

ailisp commented Dec 23, 2019 via email

@bowenwang1996
Copy link
Collaborator

yeah there seems to be some new memory leak issue. Will investigate separately.

@ailisp ailisp requested a review from bowenwang1996 as a code owner January 9, 2020 07:35
@ailisp
Copy link
Member Author

ailisp commented Feb 22, 2020

@bowenwang1996 @mfornet
I removed all rust logging part and plan to merge this PR so we can run newer tests from new branches from staging.

* docs: Improve README for one hundred nodes

* test: Enable diagnostic in one hundred nodes test

* doc: Add more instructions for the 100 nodes test
Copy link
Collaborator

@bowenwang1996 bowenwang1996 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should clean up the analysis scripts a bit.

node_blocks = [0] * 100
block_by = {}
for i in range(100):
with open(f'pytest-node-{i}.txt') as f:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we clean up this a bit? It looks outdated.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. The problem is I don't exactly remember which was outdated 😂 but I guess you and Marcelo has already keep one or two useful, up to date scripts, So I can simply delete all analyze scripts here?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's also fine I guess

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleted. If you feel some of your analyze script help, please add to this branch

pytest/tests/stress/hundred_nodes/collect_logs.py Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants