Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CheckpointSaver logs wrong filenames when final_filename is set; doesnt log saving of key metric checkpoint #6024

Closed
jak0bw opened this issue Feb 17, 2023 · 2 comments · Fixed by #6026

Comments

@jak0bw
Copy link
Contributor

jak0bw commented Feb 17, 2023

Describe the bug
When final_filename is set during CheckpointSaver creation it still uses the default name for logging. The correct (the set name) is used for saving. If save key metric is set, the saving of best metric Checkpoint is not logged at all

To Reproduce
Steps to reproduce the behavior:

  1. Create CheckpointSaver and set final_filename, save_key_metric etc. (see screenshot)
  2. Run CheckpointSaver until Checkpoint is saved and look at logged output

Expected behavior
Use correct filename for saved final checkpoint and log saving/deletion of best metric checkpoint

Screenshots
image
image

Environment

================================
Printing MONAI config...

MONAI version: 1.1.0
Numpy version: 1.24.2
Pytorch version: 1.13.1+cu117
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: a2ec375
MONAI file: C:\Users\jakob\miniconda3\envs\sr3915\lib\site-packages\monai_init_.py

Optional dependencies:
Pytorch Ignite version: 0.4.10
Nibabel version: 5.0.0
scikit-image version: 0.19.3
Pillow version: 9.4.0
Tensorboard version: NOT INSTALLED or UNKNOWN VERSION.
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.14.1+cu117
tqdm version: 4.64.1
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 5.9.4
pandas version: 1.5.3
einops version: NOT INSTALLED or UNKNOWN VERSION.
transformers version: 4.26.0
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

================================
Printing system config...

System: Windows
Win32 version: ('10', '10.0.19045', 'SP0', 'Multiprocessor Free')
Win32 edition: Core
Platform: Windows-10-10.0.19045-SP0
Processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
Machine: AMD64
Python version: 3.9.15
Process name: python.exe
Command: ['C:\Users\jakob\miniconda3\envs\sr3915\python.exe', '-c', 'import monai; monai.config.print_debug_info()']
Open files: [popenfile(path='C:\Program Files\WindowsApps\Microsoft.LanguageExperiencePackde-DE_19041.56.184.0_neutral__8wekyb3d8bbwe\Windows\System32\de-DE\tzres.dll.mui', fd=-1), popenfile(path='C:\Program Files\WindowsApps\Microsoft.LanguageExperiencePackde-DE_19041.56.184.0_neutral__8wekyb3d8bbwe\Windows\System32\de-DE\kernel32.dll.mui', fd=-1), popenfile(path='C:\Program Files\WindowsApps\Microsoft.LanguageExperiencePackde-DE_19041.56.184.0_neutral__8wekyb3d8bbwe\Windows\System32\de-DE\KernelBase.dll.mui', fd=-1)]
Num physical CPUs: 4
Num logical CPUs: 8
Num usable CPUs: 8
CPU usage (%): [8.1, 1.3, 8.5, 2.1, 4.3, 1.7, 4.3, 78.2]
CPU freq. (MHz): 4001
Load avg. in last 1, 5, 15 mins (%): [0.0, 0.0, 0.0]
Disk usage (%): 81.0
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 16.0
Available memory (GB): 6.9
Used memory (GB): 9.0

================================
Printing GPU config...

Num GPUs: 1
Has CUDA: True
CUDA version: 11.7
cuDNN enabled: True
cuDNN version: 8500
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_37']
GPU 0 Name: NVIDIA GeForce GTX 1060 6GB
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 10
GPU 0 Total memory (GB): 6.0
GPU 0 CUDA capability (maj.min): 6.1

KumoLiu added a commit to KumoLiu/MONAI that referenced this issue Feb 18, 2023
Signed-off-by: KumoLiu <[email protected]>
@KumoLiu
Copy link
Contributor

KumoLiu commented Feb 18, 2023

Hi @jak0bw, thanks for pointing this out, have created a PR to fix it.

@jak0bw
Copy link
Contributor Author

jak0bw commented Feb 20, 2023

@KumoLiu Thank you for handling this error. I didn't look deeply into the code but does your PR fix both errors

  1. The wrong save file is logged
  2. The saving of the best metric file is not logged at all

Or only one of them?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants