GroupNormalization_keras

keras implementation of group normalization. https://arxiv.org/abs/1803.08494

Group Normalization

Yuxin Wu and Kaiming He

[WIP Alert]

This repository is still work in progress.

The functionality of Group Normalization has not been fully checked. The implementation could be wrong.

Usage

from GroupNormalization import GroupNormalization

# GroupNormalization(axis=-1, epsilon=1e-6, group=32, **kwargs)

G = 8

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), kernel_initializer='he_normal', input_shape=input_shape))
model.add(GroupNormalization(group=G))
model.add(Activation('relu'))
...

Experiments with group normalization

• Experiment notebook

• Blog post

Experiment 1: Comparison between BatchNorm, GroupNorm and InstanceNorm

Setup

• Dataset: Fashion MNIST
• Architecture
  • 
• Batch size: 1
• Optimizer: Adam
• Learning rate: from 1e-3 with callback ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=3, min_lr=1e-5)
• Epochs: 13

1. Training loss

2. Validation loss

3. Training accuracy

4. Validation accuracy

Experiment 2: More comparisons

a. GroupNorm w/ optimizer AMSGrad, batch size = 1

 - Training time: 3 hrs on Google Colab

b. GroupNorm w/ optimizer AMSGrad, batch size = 128, epochs = 39

 - Training time: 17 mins on Google Colab

1. Training loss

2. Validation loss

3. Training accuracy

4. Validation accuracy

Acknowledgments

Code borrows from DingKe. Instance normalization implementation is from keras-contrib.