Fix efficient ad

[abc-125]

📝 Description

• 🛠️ Fixes 📉 Investigate the low performance of EfficientAD #1943

Several changes to fix the performance:

• Replaced teacher weights with the ones from Nelson1425, thanks to @alexriedel1 for advice. Renamed layers to match anomalib implementation, new weights can be downloaded here.
• Removed Imagenet normalization from transforms for this model, because it is in the forward for every submodel already. Added checkup to detect if it is added by the user.
• The train_batch_size should be 1, as described in the paper. It seems hard to hardcode because it is used in the AnomalibDataModule, which has no access to the model's name, so I have just added a checkup to raise an error if the value is not 1.
• Removed parameter batch_size from efficient_ad.yaml, because it was affecting only Imagenet batch size, which should always be 1 according to the paper.
• Changed max epochs in efficient_ad.yaml to 1000, because if you don't have many images per class (as in MVTec AD), the model needs to run more epochs to finish 70000 steps, which are needed to reproduce results from the paper.

✨ Changes

Select what type of change your PR is:

• 🐞 Bug fix (non-breaking change which fixes an issue)
• 🔨 Refactor (non-breaking change which refactors the code base)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📚 Documentation update
• 🔒 Security update

✅ Checklist

Before you submit your pull request, please make sure you have completed the following steps:

• 📋 I have summarized my changes in the CHANGELOG and followed the guidelines for my type of change (skip for minor changes, documentation updates, and test enhancements).
• 📚 I have made the necessary updates to the documentation (if applicable).
• 🧪 I have written tests that support my changes and prove that my fix is effective or my feature works (if applicable).

For more information about code review checklists, see the Code Review Checklist.

[alexriedel1]

Do you see any performance increases? If so can you provide some numbers?

[abc-125]

Do you see any performance increases? If so can you provide some numbers?

Sure, here are the results for image-level AUROC, seed 42, different classes of MVTec dataset, and model sizes (S or M):

• After setting train_batch_size to 1 and max_epochs to 1000 to fit the paper settings, capsules S: 87.8
• After replacing teacher weights, capsules S: 98.8, screw S: 81.2
• After removing the second Imagenet Normalization, screw S: 95.9, cable S: 94.8, transistor S: 99.8, pill M: 98.1

The last row shows results similar to the results from the paper.

[alexriedel1]

Do you see any performance increases? If so can you provide some numbers?

Sure, here are the results for image-level AUROC, seed 42, different classes of MVTec dataset, and model sizes (S or M):

• After setting train_batch_size to 1 and max_epochs to 1000 to fit the paper settings, capsules S: 87.8
• After replacing teacher weights, capsules S: 98.8, screw S: 81.2
• After removing the second Imagenet Normalization, screw S: 95.9, cable S: 94.8, transistor S: 99.8, pill M: 98.1

The last row shows results similar to the results from the paper.

Looks good to me!
The AUC-ROC EfficientAD paper results for comparison are:
EfficientAD S:
Screw: 0.97315
Cable: 0.92537
Transistor: 0.998

EfficientAD M:
Pill: 0.98608

Anomalib Patchcore WideResnet_50:
Screw: 0.924
Cable: 0.990
Transistor: 1.000
Pill: 0.924

[samet-akcay]

Thanks a lot for the effort, @abc-125. The changes look good to me. If @alexriedel1 also agrees, we could merge.

@alexriedel1 can you let me know what you think? Thanks

[alexriedel1]

Thanks a lot for the effort, @abc-125. The changes look good to me. If @alexriedel1 also agrees, we could merge.

@alexriedel1 can you let me know what you think? Thanks

Yes go for it!
Be sure to include the new model weights in the releases and then the weight download can be updated accordingly

anomalib/src/anomalib/models/image/efficient_ad/lightning_model.py

Lines 35 to 39 in 2d59cb1

	WEIGHTS_DOWNLOAD_INFO = DownloadInfo(
	name="efficientad_pretrained_weights.zip",
	url="https://github.com/openvinotoolkit/anomalib/releases/download/efficientad_pretrained_weights/efficientad_pretrained_weights.zip",
	hashsum="c09aeaa2b33f244b3261a5efdaeae8f8284a949470a4c5a526c61275fe62684a",
	)

[ashwinvaidya17]

Thanks for the fix!

[blaz-r]

Great job 😄. This also partially fixes #1813

[samet-akcay]

@ashwinvaidya17, the tests are failing. I guess it's related to the weights?

[ashwinvaidya17]

I think the tests need to be updated

anomalib/tests/integration/model/test_models.py

Line 207 in f396dd4

train_batch_size=2,

[samet-akcay]

Ah yes, of course!

[abc-125]

I think the tests need to be updated

Thank you, @ashwinvaidya17! I have updated the train_batch_size for this model in test_models.py.

[samet-akcay]

I think the tests need to be updated

Thank you, @ashwinvaidya17! I have updated the train_batch_size for this model in test_models.py.

Thanks @abc-125, merging now!