add run_zeroshot.py; add functionality to data.get_templated_dataset() (formerly add_templated_examples())

[danielkorat]

This PR simplifies running zero shot with SetFit.
The goal is to add a run_zeroshot.py script and remove the function data.get_augmented_samples() since its functionality is included in the generic data.get_templated_dataset().

[danielkorat]

Still work in progress

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[tomaarsen]

@danielkorat
This section:

setfit/tests/test_data.py

Lines 177 to 189 in 5e78e40

	@pytest.mark.parametrize(
	"dataset",
	[
	"emotion",
	"ag_news",
	"amazon_counterfactual_en",
	"SentEval-CR",
	"sst5",
	"enron_spam",
	"tweet_eval_stance_abortion",
	"ade_corpus_v2_classification",
	],
	)

is a leftover decorator from the now-removed test_get_augmented_samples. The decorator should also be removed if you want to remove test_get_augmented_samples.

[danielkorat]

@danielkorat This section:

setfit/tests/test_data.py

Lines 177 to 189 in 5e78e40

	@pytest.mark.parametrize(
	"dataset",
	[
	"emotion",
	"ag_news",
	"amazon_counterfactual_en",
	"SentEval-CR",
	"sst5",
	"enron_spam",
	"tweet_eval_stance_abortion",
	"ade_corpus_v2_classification",
	],
	)

is a leftover decorator from the now-removed test_get_augmented_samples. The decorator should also be removed if you want to remove test_get_augmented_samples.

@tomaarsen
I re-implemented these tests now.

[tomaarsen]

I have some small nitpicks, could you have a look at them?

[tomaarsen]

Looks good to me! get_templated_dataset feels pretty intuitive to use.
I'll let @lewtun decide if we need to keep add_templated_examples and add_augmented_samples, which this PR removes, but fully replace the function bodies with a deprecation warning instead. These warnings can point to the new get_templated_dataset. I suspect that the code of some notebooks or tutorials out there might break without explanation otherwise.

[danielkorat]

Looks good to me! get_templated_dataset feels pretty intuitive to use. I'll let @lewtun decide if we need to keep add_templated_examples and add_augmented_samples, which this PR removes, but fully replace the function bodies with a deprecation warning instead. These warnings can point to the new get_templated_dataset. I suspect that the code of some notebooks or tutorials out there might break without explanation otherwise.

get_templated_dataset is backwards compatible with add_templated_examples. So it's just a name change. add_templated_examples was only used in tests and in the zero-shot-classification.ipynb notebook. I refactored these files accordingly, so there is no mention of add_templated_examples anymore. We can add a line like:

add_templated_examples = get_templated_dataset

with a deprecation warning. What do you think @tomaarsen ?

[tomaarsen]

A deprecation warning would be great. Could you include a suggestion of the new function to use as well as that the old function will be removed in v1.0.0?

[tomaarsen]

I have some tiny last nitpicks

[danielkorat]

@tomaarsen please review

[tomaarsen]

Thank you @danielkorat for sticking with me through my many suggestions. The script is very useful, as is the new and improved get_templated_dataset(). I'm very impressed at the results of zeroshot as well: it is very consistent in its results compared to 8-shot, and zeroshot seems to outperform 8-shot in various benchmarks (e.g. SetFit/emotion).