Hausdorff Distance metric gets 'inf'

[hsiangyuzhao]

Hi, thank you guys for developing such a good package for medical image computation. I am using MONAI to perform lesion segmentation on BraTS dataset and I use Dice and Hausdorff distance for metrics. But when training I found that for some certain cases the hausdorff distance would be inf (not nan), and I have no idea the cause of it. Would you guys help me to find out the cause?
Relevant code is pasted below:

as BraTS provides labels for enhancing tumor, NET/NCR and peritumoral edema, I need to transform these discrete labels into overrlaping ones (enhancing tumor, tumor core and whole tumor), the relevant code is pasted below:

[Nic-Ma]

Hi @hsiangyuzhao ,

Thanks for your interest here.
MONAI provides a tutorial that train segmentation task with Brats dataset:
https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/brats_segmentation_3d.ipynb
Maybe you can compare it with your program first?
If still has the issue, please paste some training log then @yiheng-wang-nv can help try to figure it out.

Thanks.

[hsiangyuzhao]

Hi @rijobro , my scipy version is 1.5.0.

[rijobro]

Ok, so I suspect it's due to the different scipy versions. Anyway, we can make things easier for the user by adding in these sorts of checks:

if torch.all(y_pred == 0):
    warnings.warn("y_pred is all 0, this may result in nan/inf distance")
if torch.all(y == 0):
    warnings.warn("y is all 0, this may result in nan/inf distance")

[yiheng-wang-nv]

Hi @rijobro , thanks for the advice, I submitted a PR (#2205) to add the warnings.

[umgpy]

Hi @rijobro
I am trying to use Dice loss along with dice metric, but at the same time I would also like to use HD95 and average surface distance as an additional metric for reporting.
However, after a couple of epochs both HD95 and ASSD start to give Inf and the loss ends up being Nan.
While on training DSC as a metric there is no such behavior.

code

hd95_metric = HausdorffDistanceMetric(include_background=True, percentile=95, reduction="mean")
hd95_metric_batch = HausdorffDistanceMetric(include_background=True, percentile=95, reduction="mean_batch")

sd_metric =  SurfaceDistanceMetric(include_background=True, symmetric=True, reduction="mean")
sd_metric_batch =  SurfaceDistanceMetric(include_background=True, symmetric=True, reduction="mean_batch")`

...

model.eval()
        with torch.no_grad():

            for val_data in val_loader:
                ...
                val_outputs = inference(val_inputs)
                val_outputs = [post_trans(i) for i in decollate_batch(val_outputs)]
                ## trying to compute all three metrics but or this run only had SurfaceDIstance 
                #dice_metric(y_pred=val_outputs, y=val_labels)
                #dice_metric_batch(y_pred=val_outputs, y=val_labels)
                #hd95_metric(y_pred=val_outputs, y=val_labels)
                #hd95_metric_batch(y_pred=val_outputs, y=val_labels)
                sd_metric(y_pred=val_outputs, y=val_labels)
                sd_metric_batch(y_pred=val_outputs, y=val_labels)      
 
                assd_metric = sd_metric.aggregate().item()
                assd_metric_values.append(assd_metric)
                assd_metric_batch = sd_metric_batch.aggregate()
                assd_metric_1 = assd_metric_batch[0].item()
                assd_metric_values_1.append(assd_metric_1)
                assd_metric_2 = assd_metric_batch[1].item()
                assd_metric_values_2.append(assd_metric_2)
                assd_metric_3 = assd_metric_batch[2].item()
                assd_metric_values_3.append(assd_metric_3)
                sd_metric.reset()
                sd_metric_batch.reset()

epoch 10/300
1/8, train_loss: 0.5534, step time: 1.5471
2/8, train_loss: 0.5787, step time: 1.5402
3/8, train_loss: 0.5726, step time: 1.5288
4/8, train_loss: 0.5950, step time: 1.5261
5/8, train_loss: 0.5621, step time: 1.5294
6/8, train_loss: 0.5870, step time: 1.5272
7/8, train_loss: 0.5832, step time: 1.5286
8/8, train_loss: 0.5712, step time: 1.5279
epoch 10 average loss: 0.5754
current epoch: 10 current mean assd: 21.4569 BG: 10.9622 fem: 29.2373 pel: 24.1713
best mean dice: -1.0000 at epoch: -1
time consuming of epoch 10 is: 1241.7538

epoch 11/300
1/8, train_loss: 0.5905, step time: 1.5489
2/8, train_loss: 0.5844, step time: 1.5277
3/8, train_loss: 0.6046, step time: 1.5266
4/8, train_loss: 0.5408, step time: 1.5271
5/8, train_loss: 0.5791, step time: 1.5284
6/8, train_loss: 0.5734, step time: 1.5251
7/8, train_loss: 0.5421, step time: 1.5311
8/8, train_loss: 0.6211, step time: 1.5263
epoch 11 average loss: 0.5795

/home/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/hausdorff_distance.py:160: UserWarning: the prediction of class 0 is all 0, this may result in nan/inf distance.
warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/home/umgpy/Monai_8_test_env/lib/python3.8/site-packages/numpy/lib/function_base.py:4486: RuntimeWarning: invalid value encountered in subtract
diff_b_a = subtract(b, a)
/home/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/hausdorff_distance.py:160: UserWarning: the prediction of class 1 is all 0, this may result in nan/inf distance.
warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/home/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/hausdorff_distance.py:160: UserWarning: the prediction of class 2 is all 0, this may result in nan/inf distance.
warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/home/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/surface_distance.py:155: UserWarning: the prediction of class 0 is all 0, this may result in nan/inf distance.
warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/home/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/surface_distance.py:155: UserWarning: the prediction of class 1 is all 0, this may result in nan/inf distance.
warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/home/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/surface_distance.py:155: UserWarning: the prediction of class 2 is all 0, this may result in nan/inf distance.
warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")

current epoch: 11 current mean assd: inf BG: inf fem: inf pel: inf
best mean dice: -1.0000 at epoch: -1
time consuming of epoch 11 is: 633.2988

epoch 12/300
1/8, train_loss: 0.6075, step time: 1.5569
2/8, train_loss: 0.6360, step time: 1.5258
3/8, train_loss: nan, step time: 1.5197
4/8, train_loss: 0.6048, step time: 1.5274
5/8, train_loss: 0.6066, step time: 1.5282
6/8, train_loss: 0.5886, step time: 1.5263
7/8, train_loss: nan, step time: 1.5196
8/8, train_loss: 0.5142, step time: 1.5269
epoch 12 average loss: nan
current epoch: 12 current mean assd: inf BG: inf fem: inf pel: inf
best mean dice: -1.0000 at epoch: -1
time consuming of epoch 12 is: 634.2761

[al3ms]

Hi
Did you solve the problem? Regarding using Dice loss along with dice metric, and same timeHD95 and average surface distance?

[dscarmo]

Writing this response if somebody finds this in the future, distance metrics are not valid when one of the two surfaces is not present in the prediction.

I am pretty sure the BRATS dataset has some cases where the more severe type of tumor is not present, invalidating surface distance.

In fact the MONAI warning is explaining this, "UserWarning: the prediction of class 1 is all 0, this may result in nan/inf distance."

[emi-dm]

@KumoLiu do you know how to solve it?

[emi-dm]

In my case, I created a model to segment the liver and liver cancer in MSD Task 03. However, the problem is that, in some cases, the volume labels do not contain a certain class, causing the SurfaceDistanceMetric to return infinity.

[KumoLiu]

Hi @emi-dm, if the ground truth does not contain the class, you may skip the calculate. If the pred doesn't contain, you may enhance the model performance.

[emi-dm]

This is my code for model validation now @KumoLiu :

Post Transforms

post_label = AsDiscrete(to_onehot=3)
post_trans = Compose([AsDiscrete(argmax=True, to_onehot=3)])
post_trans_to_write = Compose([Activations(softmax=True), AsDiscrete(argmax=True)])

Validation in the loop

            model.eval()
            eval_loss =  0.0
            val_loader = tqdm(loaders["validation"], desc=f"Epoch {epoch+1}/{epochs} - Validation")
            with torch.no_grad():
                for data in val_loader:
                    inputs, labels = data["image"].to(device), data["label"].to(device)
                    logits = sliding_window_inference(inputs, (128, 128, 32), 10, model, overlap=0.75)
                    loss = criterion(logits, labels)
                    eval_loss += loss.item()
                    outputs = [post_trans(i) for i in decollate_batch(logits)]
                    labels_convert = [post_label(i) for i in decollate_batch(labels)]
                    dice(y_pred=outputs, y=labels_convert)
                    sdm(y_pred=outputs, y=labels_convert)
                    
                eval_dice = dice.aggregate().item()
                eval_sdm = sdm.aggregate().item()
                eval_loss /= len(loaders["validation"])
                print(f"Epoch: {epoch} | Eval Loss: {loss.item()} | Eval Dice: {eval_dice} | Eval SDM: {eval_sdm}")
                dice.reset()
                sdm.reset()
                if eval_dice > best_dice and eval_loss < less_eval_loss:
                    best_dice = eval_dice
                    less_eval_loss = eval_loss

                    # save_dir : Experiment_vN / models
                    val_results_dir = output_dir/f"Epoch_{epoch}"
                    if not val_results_dir.exists():
                        val_results_dir.mkdir(parents=False, exist_ok=True)
                    
                    for i, data in enumerate(loaders["validation"], start=1):
                        inputs, labels = data["image"].to(device), data["label"].to(device)
                        logits = sliding_window_inference(inputs, (128, 128, 32), 10, model, overlap=0.75)
                        outputs = [post_trans_to_write(i) for i in decollate_batch(logits)]
                        # TODO: Compute metrics and save the results in csv file
                        model_weights_dir = save_dir/"best_model.pth"
                        torch.save(model.state_dict(), model_weights_dir)
                        eval_utils.torch_to_nifti(inputs[0], labels[0], outputs[0], epoch=epoch, vol_id=i, output_folder=val_results_dir)
            logger.debug(f"Epoch: {epoch}, Eval Dice: {best_dice}")

How could I modify it to avoid possible inf or nan results?