Prompt Learning via Meta-Regularization

Official implementation of CVPR 2024 paper "Prompt Learning via Meta-Regularization".

Jinyoung Park, Juyeon Ko, Hyunwoo J. Kim.

Department of Computer Science and Engineering, Korea University

Installation

For installation and other package requirements, please follow the instructions detailed in INSTALL.md

Data Preparation

Please follow the instructions at DATASETS.md to prepare all datasets.

ProMetaR Training

We provide bash scripts in scripts/ for training ProMetaR. Make sure to update the DATA variable with dataset path in the script file and run the commands from the main directory ProMetaR/. Below we provide training and testing instructions for ProMetaR.

ProMetaR

(1) Base-to-Novel class generalization setting

The base-to-novel ProMetaR configuration is provided in config file at configs/trainers/ProMetaR/vit_b16_c2_ep10_batch4_4+4ctx.yaml. All hyper-parameters such as learning rate, number of epochs, prompt length and prompt depth etc., can be modified using this config file.

Run the commands below to train ProMetaR on eurosat.

# Other possible dataset values includes [caltech101, food101, dtd, ucf101, oxford_flowers, oxford_pets, fgvc_aircraft, stanford_cars, sun397, eurosat]

# seed=1
# trains and evaluates on base classes
bash scripts/prometar/base2new_train.sh eurosat 1
# evaluates on novel classes
bash scripts/prometar/base2new_test.sh eurosat 1

# seed=2
# trains and evaluates on base classes
bash scripts/prometar/base2new_train.sh eurosat 2
# evaluates on novel classes
bash scripts/prometar/base2new_test.sh eurosat 2

# seed=3
# trains and evaluates on base classes
bash scripts/prometar/base2new_train.sh eurosat 3
# evaluates on novel classes
bash scripts/prometar/base2new_test.sh eurosat 3

Averaging results over 3 seeds:

Once the above trainings and evaluations are completed, the output/ directory should have the following structure:

output
|–– base2new/
|   |–– test_new/
|   |   |–– eurosat/
|   |   |   |–– shots_16/
|   |   |   |   |–– ProMetaR/
|   |   |   |   |   |–– vit_b16_c2_ep10_batch4_4+4ctx/
|   |   |   |   |   |   |–– seed1/
|   |   |   |   |   |   |–– seed2/
|   |   |   |   |   |   |–– seed3/
|   |–– train_base/
|   |   |–– eurosat/
|   |   |   |–– shots_16/
|   |   |   |   |–– ProMetaR/
|   |   |   |   |   |–– vit_b16_c2_ep10_batch4_4+4ctx/
|   |   |   |   |   |   |–– seed1/
|   |   |   |   |   |   |–– seed2/
|   |   |   |   |   |   |–– seed3/

Now use the script parse_test_res.py and run the commands below to calculate the averaged results:

# prints averaged results for base classes
python output/base2new/train_base/eurosat/shots_16/ProMetaR/vit_b16_c2_ep10_batch4_4+4ctx --test-log
# averaged results for novel classes
python output/base2new/test_new/eurosat/shots_16/ProMetaR/vit_b16_c2_ep10_batch4_4+4ctx --test-log

The above steps can be repeated for other individual datasets.

---

Citation

If you find our work, or this repository useful, please consider giving a star ⭐ and citation.

@InProceedings{Park_2024_CVPR,
    author    = {Park, Jinyoung and Ko, Juyeon and Kim, Hyunwoo J.},
    title     = {Prompt Learning via Meta-Regularization},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023}
}

Contact

If you have any questions, please create an issue on this repository or contact at [email protected].

Acknowledgements

Our code is based on PromptSRC, along with Co-CoOp and CoOp repository. We thank the authors for releasing their code. If you use our model and code, please consider citing these works as well.