Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ensure initial propmpt is long for perf metrics tests #1471

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
+3 −2
Draft

ensure initial propmpt is long for perf metrics tests #1471

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions tests/python_tests/test_llm_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -586,7 +586,8 @@ def run_perf_metrics_collection(model_descr, generation_config: Dict, prompt: st
def test_perf_metrics(model_descr, generation_config, prompt):
import time
start_time = time.perf_counter()
perf_metrics = run_perf_metrics_collection(read_model(model_descr), generation_config, prompt)
# To ensure the prefill stage takes much more time make initial prompt long.
perf_metrics = run_perf_metrics_collection(read_model(model_descr), generation_config, prompt * 200)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose it's more clear to extend prompt where it's initially created:

    (dict(max_new_tokens=20), 'table is made of' * 200),

total_time = (time.perf_counter() - start_time) * 1000
Copy link
Contributor

@ilya-lavrenov ilya-lavrenov Jan 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in general, assumptions about perf metrics like assert load_time < 1000.0 should not be in tests, because depending on machine / network stability load_time may significantly vary

fixed it here #1478 (comment)


# Check that load time is adequate.
Expand All @@ -608,7 +609,7 @@ def test_perf_metrics(model_descr, generation_config, prompt):
durations = np.array(raw_metrics.m_durations) / 1000
# Check that prefill is not included in durations for TPOT calculation.
# For the very long prompt prefill is slow and TTFT is much larger than any other token generation duration.
assert np.all(mean_ttft > durations * 2)
assert np.all(mean_ttft > durations * 10)

mean_tpot, std_tpot = perf_metrics.get_tpot()
assert (mean_tpot, std_tpot) == (perf_metrics.get_tpot().mean, perf_metrics.get_tpot().std)
Expand Down
Loading