Skip to content

Commit

Permalink
docs: update classification
Browse files Browse the repository at this point in the history
  • Loading branch information
@ZiniuYu
ZiniuYu committed Sep 30, 2022
1 parent 1e87dec commit 5b2f782
Showing 1 changed file with 28 additions and 33 deletions.
61 changes: 28 additions & 33 deletions docs/user-guides/benchmark.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,9 +71,6 @@ We include the disk usage (in delta) and the peak RAM and VRAM usage (in delta)
| ViT-L-14::openai | 0.610 | 0.793 | 0.599 | 0.767 | 0.494 | 0.717 |
| ViT-L-14-336::openai | 0.616 | 0.812 | 0.629 | 0.779 | 0.533 | 0.741 |

## Zero-shot classification


````{dropdown} Zero-shot retrieval: MS COCO Captions
| model_fullname | image_retrieval_recall@5 | text_retrieval_recall@5 |
Expand Down Expand Up @@ -106,34 +103,32 @@ We include the disk usage (in delta) and the peak RAM and VRAM usage (in delta)
````

````{dropdown} Zero-shot classification: ImageNetV2
| model_fullname | acc1 | acc5 | mean_per_class_recall |
|----------------------------------|--------|--------|-----------------------|
| RN50::openai | 0.5287 | 0.8148 | 0.5291 |
| RN50::yfcc15m | 0.2139 | 0.4253 | 0.2145 |
| RN50::cc12m | 0.2238 | 0.4563 | 0.2244 |
| RN101::openai | 0.5608 | 0.8314 | 0.5617 |
| RN101::yfcc15m | 0.2212 | 0.4397 | 0.2216 |
| RN50x4::openai | 0.5944 | 0.8584 | 0.5946 |
| RN50x16::openai | 0.6427 | 0.8837 | 0.643 |
| RN50x64::openai | 0.6703 | 0.907 | 0.6702 |
| ViT-B-32::openai | 0.5594 | 0.8339 | 0.5595 |
| ViT-B-32::laion400m_e31 | 0.5226 | 0.794 | 0.5233 |
| ViT-B-32::laion400m_e32 | 0.5232 | 0.7947 | 0.5235 |
| ViT-B-32::laion2b_e16 | 0.5729 | 0.8391 | 0.5737 |
| ViT-B-32::laion2b_s34b_b79k | 0.5814 | 0.8392 | 0.5808 |
| ViT-B-16::openai | 0.6186 | 0.8735 | 0.6189 |
| ViT-B-16::laion400m_e31 | 0.5942 | 0.8527 | 0.5941 |
| ViT-B-16::laion400m_e32 | 0.5965 | 0.8542 | 0.5963 |
| ViT-B-16-plus-240::laion400m_e31 | 0.6139 | 0.8631 | 0.6146 |
| ViT-B-16-plus-240::laion400m_e32 | 0.6147 | 0.8646 | 0.614 |
| ViT-L-14::openai | 0.6983 | 0.9092 | 0.6986 |
| ViT-L-14::laion400m_e31 | 0.6543 | 0.886 | 0.6547 |
| ViT-L-14::laion400m_e32 | 0.6539 | 0.8857 | 0.6543 |
| ViT-L-14::laion2b_s32b_b82k | 0.6774 | 0.9024 | 0.6783 |
| ViT-L-14-336::openai | 0.7094 | 0.9164 | 0.7094 |
| ViT-H-14::laion2b_s32b_b79k | 0.7087 | 0.9166 | 0.7091 |
| ViT-g-14::laion2b_s12b_b42k | 0.6956 | 0.9086 | 0.6962 |
## Zero-shot classification

````
| model_fullname | imagenetv2 | voc2007 | vtab/caltech101 | vtab/cifar10 | vtab/cifar100 | vtab/dtd | vtab/flowers | vtab/pets | vtab/svhn | vtab/eurosat | vtab/resisc45 | vtab/pcam | vtab/diabetic_retinopathy | vtab/clevr_count_all | vtab/clevr_closest_object_distance | vtab/dsprites_label_x_position | vtab/dsprites_label_orientation | vtab/smallnorb_label_azimuth | vtab/smallnorb_label_elevation | vtab/dmlab | vtab/kitti_closest_vehicle_distance |
|---------------------------------|------------|---------|-----------------|--------------|---------------|----------|--------------|-----------|-----------|--------------|---------------|-----------|---------------------------|----------------------|------------------------------------|--------------------------------|---------------------------------|------------------------------|--------------------------------|------------|-------------------------------------|
| RN101 openai | 0.561 | 0.651 | 0.780 | 0.807 | 0.476 | 0.432 | 0.652 | 0.869 | 0.226 | 0.314 | 0.547 | 0.583 | 0.280 | 0.242 | 0.130 | 0.031 | 0.021 | 0.054 | 0.111 | 0.139 | 0.263 |
| RN101 yfcc15m | 0.221 | 0.243 | 0.469 | 0.299 | 0.125 | 0.117 | 0.210 | 0.177 | 0.137 | 0.151 | 0.099 | 0.479 | 0.584 | 0.109 | 0.159 | 0.031 | 0.019 | 0.055 | 0.097 | 0.153 | 0.252 |
| RN50 cc12m | 0.224 | 0.438 | 0.582 | 0.395 | 0.178 | 0.135 | 0.095 | 0.331 | 0.102 | 0.148 | 0.117 | 0.535 | 0.293 | 0.184 | 0.222 | 0.031 | 0.025 | 0.047 | 0.096 | 0.161 | 0.155 |
| RN50 openai | 0.529 | 0.650 | 0.772 | 0.715 | 0.403 | 0.415 | 0.660 | 0.857 | 0.303 | 0.408 | 0.453 | 0.636 | 0.171 | 0.217 | 0.148 | 0.034 | 0.014 | 0.056 | 0.110 | 0.145 | 0.170 |
| RN50 yfcc15m | 0.214 | 0.215 | 0.402 | 0.291 | 0.116 | 0.122 | 0.167 | 0.174 | 0.157 | 0.172 | 0.123 | 0.533 | 0.358 | 0.151 | 0.158 | 0.032 | 0.024 | 0.053 | 0.120 | 0.160 | 0.336 |
| RN50x16 openai | 0.643 | 0.680 | 0.810 | 0.813 | 0.522 | 0.524 | 0.724 | 0.898 | 0.409 | 0.433 | 0.589 | 0.625 | 0.715 | 0.195 | 0.213 | 0.030 | 0.026 | 0.050 | 0.116 | 0.146 | 0.229 |
| RN50x4 openai | 0.594 | 0.682 | 0.781 | 0.794 | 0.451 | 0.486 | 0.698 | 0.887 | 0.367 | 0.335 | 0.532 | 0.569 | 0.318 | 0.205 | 0.082 | 0.031 | 0.026 | 0.056 | 0.108 | 0.162 | 0.233 |
| RN50x64 openai | 0.670 | 0.740 | 0.834 | 0.851 | 0.598 | 0.531 | 0.788 | 0.936 | 0.481 | 0.577 | 0.628 | 0.539 | 0.073 | 0.227 | 0.200 | 0.034 | 0.025 | 0.056 | 0.125 | 0.158 | 0.311 |
| ViT-B-16 laion400m_e31 | 0.594 | 0.767 | 0.838 | 0.917 | 0.712 | 0.513 | 0.694 | 0.892 | 0.380 | 0.503 | 0.585 | 0.593 | 0.062 | 0.289 | 0.245 | 0.031 | 0.030 | 0.059 | 0.100 | 0.152 | 0.200 |
| ViT-B-16 laion400m_e32 | 0.597 | 0.768 | 0.837 | 0.917 | 0.712 | 0.513 | 0.692 | 0.892 | 0.385 | 0.501 | 0.585 | 0.598 | 0.077 | 0.287 | 0.245 | 0.032 | 0.029 | 0.060 | 0.099 | 0.151 | 0.183 |
| ViT-B-16 openai | 0.619 | 0.783 | 0.819 | 0.908 | 0.669 | 0.449 | 0.712 | 0.890 | 0.313 | 0.559 | 0.582 | 0.507 | 0.036 | 0.209 | 0.158 | 0.030 | 0.023 | 0.053 | 0.122 | 0.155 | 0.263 |
| ViT-B-16-plus-240 laion400m_e31 | 0.614 | 0.764 | 0.832 | 0.925 | 0.733 | 0.555 | 0.706 | 0.904 | 0.355 | 0.569 | 0.615 | 0.551 | 0.093 | 0.240 | 0.159 | 0.041 | 0.026 | 0.056 | 0.111 | 0.149 | 0.280 |
| ViT-B-16-plus-240 laion400m_e32 | 0.615 | 0.764 | 0.833 | 0.928 | 0.738 | 0.555 | 0.711 | 0.902 | 0.362 | 0.581 | 0.613 | 0.551 | 0.095 | 0.238 | 0.160 | 0.043 | 0.027 | 0.054 | 0.110 | 0.148 | 0.281 |
| ViT-B-32 laion2b_e16 | 0.573 | 0.788 | 0.831 | 0.941 | 0.754 | 0.539 | 0.691 | 0.893 | 0.388 | 0.503 | 0.619 | 0.506 | 0.195 | 0.192 | 0.167 | 0.031 | 0.024 | 0.052 | 0.110 | 0.189 | 0.176 |
| ViT-B-32 laion2b_s34b_b79k | 0.581 | 0.791 | 0.839 | 0.936 | 0.755 | 0.557 | 0.716 | 0.909 | 0.410 | 0.482 | 0.610 | 0.598 | 0.734 | 0.153 | 0.189 | 0.029 | 0.034 | 0.062 | 0.113 | 0.159 | 0.262 |
| ViT-B-32 laion400m_e31 | 0.523 | 0.731 | 0.818 | 0.883 | 0.678 | 0.521 | 0.659 | 0.856 | 0.220 | 0.470 | 0.510 | 0.549 | 0.259 | 0.155 | 0.161 | 0.033 | 0.021 | 0.053 | 0.117 | 0.173 | 0.122 |
| ViT-B-32 laion400m_e32 | 0.523 | 0.733 | 0.817 | 0.885 | 0.677 | 0.523 | 0.658 | 0.854 | 0.223 | 0.476 | 0.510 | 0.548 | 0.240 | 0.153 | 0.161 | 0.033 | 0.021 | 0.054 | 0.117 | 0.173 | 0.118 |
| ViT-B-32 openai | 0.559 | 0.764 | 0.815 | 0.898 | 0.643 | 0.443 | 0.664 | 0.873 | 0.135 | 0.504 | 0.537 | 0.623 | 0.447 | 0.232 | 0.164 | 0.037 | 0.024 | 0.061 | 0.127 | 0.193 | 0.274 |
| ViT-g-14 laion2b_s12b_b42k | 0.696 | 0.811 | 0.851 | 0.971 | 0.839 | 0.682 | 0.776 | 0.943 | 0.603 | 0.648 | 0.718 | 0.560 | 0.580 | 0.332 | 0.175 | 0.036 | 0.031 | 0.060 | 0.115 | 0.190 | 0.138 |
| ViT-H-14 laion2b_s32b_b79k | 0.709 | 0.777 | 0.850 | 0.975 | 0.847 | 0.678 | 0.801 | 0.945 | 0.563 | 0.726 | 0.699 | 0.542 | 0.297 | 0.268 | 0.169 | 0.032 | 0.027 | 0.054 | 0.111 | 0.140 | 0.110 |
| ViT-L-14 laion2b_s32b_b82k | 0.677 | 0.805 | 0.851 | 0.966 | 0.833 | 0.629 | 0.758 | 0.932 | 0.459 | 0.646 | 0.668 | 0.563 | 0.116 | 0.312 | 0.161 | 0.032 | 0.020 | 0.056 | 0.108 | 0.224 | 0.229 |
| ViT-L-14 laion400m_e31 | 0.654 | 0.758 | 0.839 | 0.947 | 0.774 | 0.598 | 0.757 | 0.917 | 0.378 | 0.632 | 0.671 | 0.487 | 0.058 | 0.242 | 0.149 | 0.030 | 0.026 | 0.053 | 0.109 | 0.186 | 0.200 |
| ViT-L-14 laion400m_e32 | 0.654 | 0.756 | 0.839 | 0.946 | 0.774 | 0.605 | 0.756 | 0.919 | 0.380 | 0.622 | 0.675 | 0.493 | 0.061 | 0.243 | 0.149 | 0.030 | 0.026 | 0.053 | 0.110 | 0.186 | 0.203 |
| ViT-L-14 openai | 0.698 | 0.783 | 0.835 | 0.956 | 0.758 | 0.554 | 0.792 | 0.932 | 0.571 | 0.626 | 0.633 | 0.520 | 0.733 | 0.194 | 0.161 | 0.032 | 0.023 | 0.045 | 0.115 | 0.163 | 0.218 |
| ViT-L-14-336 openai | 0.709 | 0.781 | 0.837 | 0.949 | 0.744 | 0.556 | 0.783 | 0.937 | 0.560 | 0.615 | 0.638 | 0.608 | 0.733 | 0.200 | 0.158 | 0.032 | 0.024 | 0.046 | 0.113 | 0.158 | 0.262 |

0 comments on commit 5b2f782

Please sign in to comment.