CapsNet_for_NER

Capsule Network adapted for Name Entity Recognition with the CoNLL-2003 Shared Task.

---

Usage

Step 1. Install Keras>=2.0.7 with TensorFlow>=1.2 backend.

pip install tensorflow
pip install keras

Step 2. Clone this repository to local.

git clone https://github.com/Chucooleg/CapsNet_for_NER capsnet-ner
cd capsnet-ner

Step 3. Run jupyter notebook.

jupyter notebook

Step 4. Train

Training was done in code/model_training_tmpl.ipynb with CapsNet and CNN. Variations of window size, embedding type, additional features and decoder were used.

Rank	Model	Decoder	Window	Embed	+Fea	F1
1	CAP	off	11	glove	yes	92.24
2	CAP	off	11	glove	yes	92.15
3	CAP	on	7	glove	yes	92.11
24	CAP	off	9	glove	yes	90.93
26	CAP	off	11	glove	yes	90.79
27	CNN	off	11	glove	yes	90.59
33	CNN	off	9	glove	yes	89.49

Test Results

F1 Scores

CapsNet named entity recognition f1 scores on ConLL-2003. The results can be reproduced by launching /code/model_testing.ipynb.

Model	Precision	Recall	F1
Chiu (2016)	91.39	91.85	91.62
CapsNet	87.59	87.33	87.46
CapsNet+decoder	86.40	87.17	86.78
CNN Baseline	85.93	86.23	86.08
CoNLL-2003 Baseline	71.91	50.90	59.61

Files

/code

File	Description
buildCapsModel.py	capsnet model implementation
buildCNNModel.py	cnn model implementation
capsulelayers.py	capsnet modules - slightly adapted from Xifeng Guo's implementation
evaluation_helper.py	helper functions for model evaluation
Examine_History.ipynb	code to plot and investigate model training history and dev results
error_analysis_demo.ipynb	notebook to generate a model evaluation report
error_analysis_testset.ipynb	notebook to generate model evaluation reports for test set
glove_helper.py	helper code for loading Glove embeddings
loadutils.py	helper functions for loading and storing models and the data set
model_testing.ipynb	demo code for best model and baseline test set performance
model_training_tmpl.ipynb	notebook to orchestrate and run model training sessions
trainCapsModel.py	interface code to train a capsnet model
trainCNNModel.py	interface code to train a CNN model
/common	helper code for building and manipulating a vocabulary

/data

CoNLL-2003 data set

/code/result

Final models and their training history

setup GCP gpu for tensorflow and keras

https://hackernoon.com/launch-a-gpu-backed-google-compute-engine-instance-and-setup-tensorflow-keras-and-jupyter-902369ed5272

Internal Link	About
Team Project Proposal Link	For submission

CapsNet Concepts	About
Dynamic Routing Between Capsules	Sabour, Frosst, Hinton (2017)
Transforming Auto-encoders	Hinton, Krizhevsky, Wang (2011)
Named Entity Recognition with Bidirectional LSTM-CNNs	Chiu and Nichols (2016)
Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition	Erik F. Tjong Kim SangandFien De Meulder, 2016
Capsule Networks (CapsNets) –	Video concept intro

CapsNet Implementation	About
Tensorflow Implementation 1	.ipynb based on Dynamic Routing between Capsules (MNIST)
Video of above	Video walkthrough (MNIST)
How to implement Capsule Nets using Tensorflow	Video walkthrough (MNIST)
CapsNet-Keras	repo Keras w/ TensorFlow backend (MNIST)
CapsNet-Tensorflow	repo TensorFlow (MNIST)
Dynamic Routing Between Capsules	repo PyTorch (MNIST)
CapsNet for Natural Language Processing	repo CapsNet for Sentiment Analysis

NER Dataset	About
OntoNotes Release 5.0	download, intro
CoNLL-2003 Dataset	download, intro
Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition	Introduction of the dataset
CoNLL-2003 Shared Task	CoNLL-2003 Benchmark papers
OntoNotes coreference annotation and modeling	OntoNotes Benchmark papers
Named Entity Recognition: Exploring Features	Explore faetures for NER task. Both CoNLL-2003 and OntoNotes version 4 are used.

|