Add new evaluation metrics

[jainapurva]

Added new tests to llama/eval.sh for more extensive testing on larger metrics

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/934

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit d0743e1 with merge base 72d2518 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[jerryzh168]

if not sure if all eval metrics are relevant for llama quant, cc @HDCharles to take a look

[HDCharles]

we can already specify any number of tasks with the --tasks argument, normally if I wanted to make it easier to run a small set of hard coded sets of experiments i would write an sh file that specified these things, not modify the lower level eval runner to have multiple ways to specify the tasks.

the current solution is specifying all tests explicitly in benchmarks.sh and we made evals.sh to do the same though I haven't added all the tests there yet. If you want to make it easier to run those sets i'd maybe add them there?

Is this a larger suggestion that the benchmarks we list in the README should be changed as well?

[HDCharles]

see comment

[jainapurva]

the current solution is specifying all tests explicitly in benchmarks.sh and we made evals.sh to do the same though I haven't added all the tests there yet. If you want to make it easier to run those sets i'd maybe add them there?

I think we need more eval metrics to test the different techniques on llama. Though that could be done by updating the eval.sh instead. Also, we can add more benchmarks to the readme file in future.

[drisspg]

@HDCharles just to confirm this is what you have in mind right?

[HDCharles]

this looks better