[Bug]: The results of running an ONNX model with ONNX Runtime and OpenVINO are significantly different.

[surprise335]

Describe the bug

When I use a general ONNX model, the preprocessing part is the same for both ONNX Runtime and OpenVINO, but the final outputs from the models are significantly different. The discrepancies between the image data and the pred_score are particularly large.

Dataset

MVTec

Model

PatchCore

Steps to reproduce the behavior

I first extract the necessary preprocessing part from openvino_inferencer.py, then the image is preprocessed and passed into the ONNX inference.

OS information

OS information:

• OS: [e.g. Ubuntu 20.04]
• Python version: [e.g. 3.10.0]
• Anomalib version: [e.g. 0.3.6]
• PyTorch version: [e.g. 1.9.0]
• CUDA/cuDNN version: [e.g. 11.7]
• GPU models and configuration: [e.g. 2x GeForce RTX 3090]
• Any other relevant information: [e.g. I'm using a MVTec dataset]

Expected behavior

111

Screenshots

No response

Pip/GitHub

GitHub

What version/branch did you use?

No response

Configuration YAML

no

Logs

this is a print of ONNXRUNTIME:
[array([[[[0.00080926, 0.00082284, 0.00086395, ..., 0.00610855,
          0.00570343, 0.00557138],
         [0.00081174, 0.00082533, 0.00086647, ..., 0.00636054,
          0.00595521, 0.00582308],
         [0.00081954, 0.00083316, 0.00087442, ..., 0.0071249 ,
          0.00671896, 0.00658659],
         ...,
         [0.00846975, 0.00861945, 0.00907553, ..., 0.07886717,
          0.07883535, 0.07882532],
         [0.00739349, 0.00755267, 0.00803747, ..., 0.07874256,
          0.07872541, 0.07872023],
         [0.00703752, 0.00719985, 0.00769415, ..., 0.0787018 ,
          0.07868947, 0.07868591]]]], dtype=float32), array([0.45953268], dtype=float32)]


this is a print of OPENVINO

{<ConstOutput: names[output] shape[1,1,224,224] type: f32>: array([[[[0.00174331, 0.00173091, 0.00169346, ..., 0.0038444 ,
          0.0036402 , 0.00357483],
         [0.00173121, 0.00171927, 0.00168322, ..., 0.00393786,
          0.00373232, 0.00366649],
         [0.00169494, 0.0016844 , 0.00165263, ..., 0.00422329,
          0.00401373, 0.00394652],
         ...,
         [0.00905577, 0.00934169, 0.01021195, ..., 0.08136584,
          0.08144516, 0.08147193],
         [0.00811003, 0.00840341, 0.00929627, ..., 0.08150722,
          0.08160616, 0.0816394 ],
         [0.00779786, 0.00809371, 0.00899405, ..., 0.08155414,
          0.08165953, 0.08169492]]]], dtype=float32), <ConstOutput: names[659] shape[1] type: f32>: array([0.29684675], dtype=float32)}

Code of Conduct

• I agree to follow this project's Code of Conduct

[blaz-r]

Hi, could you please share the exact code used to produce the results and also provide the version of anomalib that you used. Thank you.

[surprise335]

Hi, could you please share the exact code used to produce the results and also provide the version of anomalib that you used. Thank you.

My anomalib repository version is the latest, and the corresponding Python package version is 2.0.0.dev0. Below is my ONNX inference code:

import onnx
import onnxruntime as ort
import numpy as np
import cv2
import os 
import time 
import numpy as np
import json
from omegaconf import DictConfig, OmegaConf
from typing import Any, cast
import torch
from torchvision.transforms.v2.functional import to_dtype, to_image
from PIL import Image

def preprocess(img):
    if img.dtype != np.float32:
        img = img.astype(np.float32)
    if img.max() > 1.0:
        img /= 255.0

    img = np.expand_dims(img,axis=0)
    img = np.transpose(img,(0,3,1,2))
    return img

def read_image(path, as_tensor: bool = False) -> torch.Tensor | np.ndarray:
    """Read image from disk in RGB format.

    Args:
        path (str, Path): path to the image file
        as_tensor (bool, optional): If True, returns the image as a tensor. Defaults to False.

    Example:
        >>> image = read_image("test_image.jpg")
        >>> type(image)
        <class 'numpy.ndarray'>
        >>>
        >>> image = read_image("test_image.jpg", as_tensor=True)
        >>> type(image)
        <class 'torch.Tensor'>

    Returns:
        image as numpy array
    """
    image = Image.open(path).convert("RGB")
    return to_dtype(to_image(image), torch.float32, scale=True) if as_tensor else np.array(image) / 255.0

def _load_metadata(path= None) -> dict | DictConfig:  # noqa: PLR6301
        """Load the meta data from the given path.

        Args:
            path (str | Path | dict | None, optional): Path to JSON file containing the metadata.
                If no path is provided, it returns an empty dict. Defaults to None.

        Returns:
            dict | DictConfig: Dictionary containing the metadata.
        """
        metadata: dict[str, float | np.ndarray | torch.Tensor] | DictConfig = {}
        print(path)

        if path is not None:
            config = OmegaConf.load(path)
            metadata = cast(DictConfig, config)
        return metadata

def normalize_min_max(
    targets: np.ndarray | np.float32 | torch.Tensor,
    threshold: float | np.ndarray | torch.Tensor,
    min_val: float | np.ndarray | torch.Tensor,
    max_val: float | np.ndarray | torch.Tensor,
) -> np.ndarray | torch.Tensor:
    """Apply min-max normalization and shift the values such that the threshold value is centered at 0.5."""
    normalized = ((targets - threshold) / (max_val - min_val)) + 0.5
    print('-'*20)
    print(normalized)
    print(targets)
    if isinstance(targets, np.ndarray | np.float32 | np.float64):
        normalized = np.minimum(normalized, 1)
        normalized = np.maximum(normalized, 0)
    elif isinstance(targets, torch.Tensor):
        normalized = torch.minimum(normalized, torch.tensor(1))  # pylint: disable=not-callable
        normalized = torch.maximum(normalized, torch.tensor(0))  # pylint: disable=not-callable
    else:
        msg = f"Targets must be either Tensor or Numpy array. Received {type(targets)}"
        raise TypeError(msg)
    return normalized

def _normalize(
        pred_scores: torch.Tensor | np.float32,
        metadata,
        anomaly_maps: torch.Tensor | np.ndarray | None = None,
    ) -> tuple[np.ndarray | torch.Tensor | None, float]:
        """Apply normalization and resizes the image.

        Args:
            pred_scores (Tensor | np.float32): Predicted anomaly score
            metadata (dict | DictConfig): Meta data. Post-processing step sometimes requires
                additional meta data such as image shape. This variable comprises such info.
            anomaly_maps (Tensor | np.ndarray | None): Predicted raw anomaly map.

        Returns:
            tuple[np.ndarray | torch.Tensor | None, float]: Post processed predictions that are ready to be
                visualized and predicted scores.
        """
        # min max normalization
        #print(f'metadata is {metadata}')
        if "pred_scores.min" in metadata and "pred_scores.max" in metadata:
            if anomaly_maps is not None and "anomaly_maps.max" in metadata:
                anomaly_maps = normalize_min_max(
                    anomaly_maps,
                    metadata["pixel_threshold"],
                    metadata["anomaly_maps.min"],
                    metadata["anomaly_maps.max"],
                )
            pred_scores = normalize_min_max(
                pred_scores,
                metadata["image_threshold"],
                metadata["pred_scores.min"],
                metadata["pred_scores.max"],
            )
            #print(f'process pred_scores is {pred_scores}')

        return anomaly_maps, float(pred_scores)

# load ONNX model
onnx_model_path = '/home/ubuntu/ltt-files/anomalib-main/export/yinzhang/weights/onnx/modelModify.onnx'
metadata = _load_metadata('/home/ubuntu/ltt-files/anomalib-main/export/yinzhang/weights/onnx/metadata.json')

model = onnx.load(onnx_model_path)

# check model
onnx.checker.check_model(model)

# init ONNX Runtime 
session = ort.InferenceSession(onnx_model_path, providers=['CPUExecutionProvider','CUDAExecutionProvider'])
print(f"get_providers: {session.get_providers()}")
# get input
input_name = session.get_inputs()[0].name
img_files = [f for f in os.listdir('/home/ubuntu/ltt-files/anomalib-main/datasets/bottle/test/broken_small/') if f.endswith(('.jpg', '.png'))]

# for i in img_files:
img = read_image('/home/ubuntu/ltt-files/anomalib-main/datasets/yinzhang_imgs/train/'+'22 (1).jpg')
img = preprocess(img.copy())
#print(img)
outputs = session.run(None, {input_name: img})
print(outputs)
predictions = outputs[0]
start = time.time()
anomaly_map = predictions.squeeze()
pred_score = anomaly_map.reshape(-1).max()
_, pred_score = _normalize(pred_scores=pred_score, metadata=metadata)
    # print(f'times is {time.time()-start}')
    # if pred_score/2 >= 0.31506848335266113:
    #     print('NORMAL')
    # else:
    #     print('ABNORMAL')
    #print(pred_score)

[FedericoDeBona]

Do you think this is similar to #2448?