How to run Torch Inferencer?

[laogonggong847]

Describe the bug

Hello @ashwinvaidya17 and @samet-akcay, I noticed that ./tools/inference/torch_inference.py appears changed in the two versions as shown.

When configuring the parser[ parser=ArgumentParser() ], the version of @ashwinvaidya17 is passed into config, while @samet-akcay, the latest version (and the one I'm using now), is not passed into config。

And there is a difference in the corresponding function load_model that loads the model

Because I am using the latest version of anomalib (samet-akcay)，When I use pytorch inference, it comes up with an error. The specific place where the error occurred I have located to be

model = torch.load(path, map_location=self.device)["model"]

Its specific error message is:
KeyError: 'model'

So I used the following code to debug：

I found that this is indeed a mistake, there is no "model"

---

So I have the following questions:

1: Is this KeyError: 'model' problem a bug that does exist, as I see it specifically mentioned in the update file (#600, #601). How should I go about fixing it.

2: I'm actually encapsulating my own program to use OnnxRuntime to implement the reasoning. I noticed that openvino's confidence score is calculated after getting a value from predictions=predictions[self.output_blob] . Do you have any suggestions on how I should get this similar result in OnnxRuntime by its inference?

I am looking forward to your reply, thank you very much!

Dataset

Folder

Model

PatchCore

Steps to reproduce the behavior

torch_inference.py

OS information

pytorch_1.12.1
CUDA_11.3

Expected behavior

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

No response

Configuration YAML

-

Logs

-

Code of Conduct

• I agree to follow this project's Code of Conduct

[glucasol]

Hi everyone, @ashwinvaidya17, @samet-akcay
Any update here? I am getting the same errors here: KeyError: 'model'.
Now the model is being loaded like this:

model = torch.load(path, map_location=self.device)["model"]

Instead of loading the 'state_dict' key, it's loading 'model' key which does not exist.

Thanks!

[nguyenanhtuan1008]

@samet-akcay sorry for disturb but any plan for fixing this?

[ironllamagirl]

I am facing the same issue. And I can't find a workaround.

I need this because I am trying to get the pred_mask to do my own processing on it. Anyone aware of another way to get the predicted masks? Thanks

[samet-akcay]

Hey everyone, I'll have a look at the issue

[samet-akcay]

I have looked into this. This is actually not a bug. In order to use the torch_inference.py one would need to export the model to torch in the config file. For example, the corresponding section in a config file, would look something like this:

logging:
  ...

optimization:
  export_mode: torch # options: torch, onnx, openvino

# PL Trainer Args. Don't add extra parameter here.
trainer:
  ...

So if you set the export_mode to torch, the current version should work. Let me know if you still have issues.

Depending on your reply, I will convert this issue to a Q&A in Discussions.

[d3tk]

If I have already exported my model without having torch in my config is there anyway to get it to work with the torch_inferencer.py without re-training?

[laogonggong847]

hello @samet-akcay @blaz-r @alexriedel1 ,

• Replicating model training using pytorch

---

I'm sorry to bother you all, but I have two questions for you.

1:anomalib is trained all using pytorch lighting, if I want to utilize pytorch for training, do I just go to the torch_model.py corresponding to all models?

2:patchcore doesn't have a loss function, if I want to use pytorch to build the corresponding training method, how should I do it, are there any existing tutorials or do you guys have any good suggestions?

Thank you very much for your reply.

[samet-akcay]

@laogonggong847, can you clarify what exactly you want to do? Do you want to write your own training script?

Note that patchcore does not have a loss function because it does not do a standard deep learning training. It uses a CNN just for feature extraction.

[laogonggong847]

hello @blaz-r , hello @samet-akcay , hello @ashwinvaidya17
Thank you very much for your reply, I apologize for the confusion I caused you in my previous statement, and I will re-describe my question here.

I have very little exposure to pytorch lighting, which makes it difficult for me to understand the code related to pytorch lighting. The training code of anomalib seems to utilize the Trainer implementation of pytorch lighting, which makes it difficult for me to have a clear understanding of it even if I have looked at the corresponding source code.

For these reasons, and in order to better learn and understand anomalib, I would like to rewrite the anomalib training code using pytorch.

My core goal is essentially to use pytorch to rewrite ". /tools/train.py". I noticed that there are two py files for all the models ("lighting_model.py" and "torch_model.py").

My question is, if I just use pytorch to rewrite the training code. How do I go about the three most basic parts of the model, the loss function, and the optimizer, and please and allow me to formulate these questions more specifically:

---

1: Problems with anomalib models for pytorch training:

Take patchcore as an example, we know that when pytorch training we all design a model of our own as needed, and for anomalib, can I just use class PatchcoreModel under torch_model.py corresponding to ./models/patchcore.
Specifically when I my_model = PatchcoreModel(). Is this my_model the model I need to train with pytorch? If not what else should I do?

---

2:Loss function problem during pytorch training:

2.1: I know that similar to patchcore and padim does not have loss function. So when rewriting the training code using pytorch, how should I design for this kind of method which does not have a loss function, I haven't come across a similar situation in my previous studies.

2.2: For cfa, which has a loss function, do I just need to utilize the loss.py in the corresponding folder to implement its loss function. Take cfa for example, ". /models/cfa/loss.py" in which you designed class CfaLoss. Do I "my_loss = CfaLoss()" to get the loss function when I train with pytorch.

---

3:Optimizer problem during pytorch training:

3.1: When designing the training code for anomalib using pytorch, the optimizer also involves the problem of two different models represented by patchcore and cfa, how should I deal with it.

3.2: In the "config.yaml" file, 'trainer' corresponds to many parameters in pytorch lighting. Do I need to keep these training parameters consistent when I train with pytorch? If not, will this cause a big difference in performance and other aspects between the model I train and the model I train with pytorch lighting.

---

Very much looking forward to your reply, thank you again！

[laogonggong847]

hello @samet-akcay @blaz-r
When training on a custom dataset, the structure of my data format is

--root_dir
    -- Normal_dir
    --Abnormal_dir

I scrutinized the source code when the dataset was divided , and I found that the images in Abnormal_dir are not used for dataset division during training. The training set, test set, and validation set don't seem to apply the images in Abnormal_dir. I would like to ask how are these abnormal samples being used in practice, what is their real role and where is this visible in the code?

[blaz-r]

Hello @laogonggong847. Abnormal images are only used in testing stage. The models are usually trained only on good data, due to nature of anomaly detection where we usually only have good samples at time of training.

[laogonggong847]

hello @blaz-r :)
Thank you very much for your reply, I rewrote the pytorch lighting training script using pytorch. But I found a problem that there are differences in the first and last layers of the model compared to the model trained by my own script and the model trained by anomalib. I observed that these differences are caused by "min", "max", "pixel_threshold" and "image_threshold". threshold".

---

I have two questions:
1:Taking the last layers as an example, I found that the middle layers are exactly the same, while the ones I trained myself have less "image_threshold" and "image_threshold". Since I am not familiar with pytorch lighting, can you tell me where these four parameters are calculated in pytorch lighting and how I should write them into the model?
2:What is the reason that my memory_bank is 0, I checked the structure and parameters of the middle layer are exactly the same, and the model I used is patchcore.
Looking forward to your reply again, thank you very much.

[alexriedel1]

@laogonggong847 Patchcore has no optimizable parameters, it just collects features extracted by the chosen backbone model. It saves these features in the memory_bank. I guess you didn't correctly rewrite the training script and thats whats causing these issues

[laogonggong847]

hello @alexriedel1
Yes I realize that my training script is indeed incomplete, because currently I only rewrite the "torch_model.py" and "lighting_model.py" counterparts of the patchcore in anomalib using pytorch. I haven't rewritten the "anomaly_map.py" part yet.

I think my rewrite of the training section should be error free for now, as the model I trained is identical except for the small portion shown in the image and the inconsistency at the beginning, which I think may be a result of my incomplete code.
Regarding the four thresholds, are they calculated during the training or validation phase of the model？ and how would it be appropriate for me to rewrite them using pytorch？
It would be great if you could help me locate the code that calculates the calculation of these thresholds!
Thanks for your reply, much appreciated!

[alexriedel1]

The min and max values are calculated during validation. In anomalib you find the process for min-max-normalization here https://github.com/openvinotoolkit/anomalib/blob/main/src/anomalib/utils/callbacks/min_max_normalization.py and here https://github.com/openvinotoolkit/anomalib/blob/main/src/anomalib/post_processing/normalization/min_max.py

Also check the base model on how to get the the thresholds: https://github.com/openvinotoolkit/anomalib/blob/main/src/anomalib/models/components/base/anomaly_module.py#L158

However you don't necessarily need the min max and thresholds for a working model..

[laogonggong847]

Thank you very much @alexriedel1
Thank you so much for your reply, it was so helpful.
Also I'm still confused about the two questions I showed earlier with the pictures.
Specifically these two questions:
1: memonry_bank The threshold calculated using anomalib is 16500X1536 and I have 0. Is it convenient for you to help me locate the code corresponding to the patchcore in anomalib that saves the result to memory_bank as well?

2: In the actual Patchcore pytorch model code, I don't see any relevant results for the four thresholds, are the TAs simply extra information that is inserted. Should I convert this ckpt model to onnx to check it further to check if they have the same structure.

Thanks again for your patience in helping and explaining

[alexriedel1]

1
The memory bank is filled in this function https://github.com/openvinotoolkit/anomalib/blob/93239858c51de63dba064bfc6b1e66522d2586ae/src/anomalib/models/patchcore/torch_model.py#L132C1-L144C1

It is the coreset of all features extracted during training

2
Be Aware that the Patchcore Lightning Module inherits from the AnomalyModule https://github.com/openvinotoolkit/anomalib/blob/93239858c51de63dba064bfc6b1e66522d2586ae/src/anomalib/models/patchcore/lightning_model.py#L24C32-L24C32 where the information about min max and thresholding is coming from..

[laogonggong847]

thank you very much @alexriedel1 ,

Thank you very much for your patience and professional help, your relevant information has made my understanding of anomalib's code structure much clearer.

Yes, I noticed the subsample_embedding before, but I noticed that in the ligthing_model.py of patchcore, the subsample_embedding is called in the validation of pytorch lighting as shown here:

Unfamiliarity with pytorch lighting caused me to overlook this. Thanks again for your help!

And in my own rewritten training code, I haven't added the validation part yet, so could this be the reason why the models end up with inconsistent memory_bank. Thank you very much for your help!

[laogonggong847]

hello @alexriedel1 ,
I have completed the pytorch rewrite of the training model script after your patient guidance last week, thank you again.
Also I have a new problem, for anomalib's pre-trained model, it always loads from the path ".. //torch//hub//checkpoints//xxxxxx.pth". If the corresponding pre-training model does not exist under this path, then it is downloaded from the web. I see that the processing for pre-trained models mainly utilizes timm, but I am not familiar with it. I now have a question, if I want to specify that the model under the path "D://My_model//xxxxxxx.pth" should be loaded, what should I do?
Or to be more specific, what I want to achieve is to check the folder "D://My_model" and if there is a "xxxxxxxx.pth" in this folder then apply it directly, if not then download it. I don't want the program to have to go to a certain path "... //torch//hub//checkpoints". This is a very simple question for you I'm sure, but my unfamiliarity with timm makes it difficult for me to deal with. I don't expect that modifying the source code related to pytorch hub will make it possible.
Thanks again for your help, and you can still use patchcore as a sample program.

[alexriedel1]

You can basically use this function to get the feature extractor https://github.com/openvinotoolkit/anomalib/blob/93239858c51de63dba064bfc6b1e66522d2586ae/src/anomalib/models/components/feature_extractors/timm.py#L52C1-L58C10

[laogonggong847]

Thank you very much for your professional and quick reply, yes I have located it here. But I still don't understand how I should set the parameter here to make sure that it loads the pre-training model at the path I specified!
After I set the path using os.environ["TORCH_HOME"], the program is still checking on the path I set + "//torch//hub//checkpoints//xxxxxx.pth".
This is not the real path I specified, I hope my formulation does not confuse you, I look forward to your reply, thank you very much!

[alexriedel1]

The timm.create_model method has the attribute checkpoint_path which you can set to a custom path I guess https://github.com/huggingface/pytorch-image-models/blob/3b8ef3f32fc059b4c4df03b7cb4a8e8685520562/timm/models/_factory.py#L38-L47

[laogonggong847]

Thank you very much for your professional and patient help, it's a bit late today, I'll test it tomorrow, thanks again for your continued help!