add --save_all_results_to_dataframe to benchmark_models cli
#233
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… all results to a csv file
kxtran
reviewed
Jul 29, 2024
cli_docs.md
Outdated
| @@ -147,7 +147,9 @@ original_request: | |||
| ╰─────────────────────────┴───────────────────────────────────────────────────────┴──────────────────────────────────┴───────────────╯ | |||
| ``` | |||
|
|
|||
| You can also filter the completions by tags and generate a report in markdown file using `--file` or `-f`. And run our prompt analyzer (auto-prompt) using `--analyze_prompt`. | |||
| You can also filter the completions by tags and generate a report in markdown file using `--file` or `-f`. | |||
| If you want to save the all results for post-processing, use `--save_all_results_to_dataframe` to dump it into a csv file. | |||
There was a problem hiding this comment.
would you mind to add the file name(_all_result_dataframe_dump.csv) which the data will be stored in? I'm also thinking to include that the --file flag is required for this command.
log10/cli/completions.py
Outdated
| @@ -267,7 +267,22 @@ def download_completions(limit, offset, timeout, tags, from_date, to_date, compa | |||
| @click.option("--top_p", default=1.0, help="Top p") | |||
| @click.option("--analyze_prompt", is_flag=True, help="Run prompt analyzer on the messages.") | |||
| @click.option("--file", "-f", help="Specify the filename for the report in markdown format.") | |||
There was a problem hiding this comment.
would you mind to update the help to include the save results to csv file?
| pivot_df["Prompt Messages"] = all_df.groupby("Completion ID")["Prompt Messages"].first() | ||
|
|
||
| # save the dataframe to a csv file | ||
| if save_all_results_to_dataframe: |
There was a problem hiding this comment.
I think that we should validate that file flag is provided when save_all_results_to_dataframe command is used. So if save_all_results_to_dataframe is used, but file isn't provided, we can print out warning message to the users.
wenzhe-log10
force-pushed
the
wenzhe/eng-974-logger-cli-benchmark-dump-the-comparison-data-to-file
branch
from
7d235de to
33e1316
Compare
wenzhe-log10
commented
Jul 29, 2024
Comment on lines
+310
to
+311
| if save_all_results_to_dataframe and not file: | ||
| raise click.UsageError("--save_all_results_to_dataframe must be used with --file") |
There was a problem hiding this comment.
@kxtran
check --save_all_results_to_dataframe and --file here.
wenzhe-log10
deleted the
wenzhe/eng-974-logger-cli-benchmark-dump-the-comparison-data-to-file
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
to save all results table to a csv file for post-processing.
updated the doc.