From 8770fabb324b391ba9dd2b196771b92824f01f8c Mon Sep 17 00:00:00 2001 From: Shagun Sodhani Date: Sat, 16 Mar 2019 11:58:46 -0400 Subject: [PATCH] fixed some typos --- site/_config.yml | 4 +- ...nal Language in Multi-Agent Populations.md | 2 +- ...r Learning Hierarchical Representations.md | 2 +- ...as Memory for Recurrent Neural Networks.md | 46 +++++++++++++++++++ site/_site | 2 +- 5 files changed, 51 insertions(+), 5 deletions(-) create mode 100755 site/_posts/2019-02-05-Linguistic Knowledge as Memory for Recurrent Neural Networks.md diff --git a/site/_config.yml b/site/_config.yml index 6ff63043..ee1d3555 100755 --- a/site/_config.yml +++ b/site/_config.yml @@ -9,14 +9,14 @@ title: 'Papers I Read' tagline: 'Notes and Summaries' description: 'I am trying a new initiative - A Paper A Week. This blog will hold all the notes and summaries.' # url: 'https://shagunsodhani.in/test' -baseurl: 'https://shagunsodhani.in/papers-I-read' +baseurl: 'https://shagunsodhani.github.io/papers-I-read' paginate: 10 plugins: [jekyll-paginate] # About/contact author: name: Shagun Sodhani - url: https://shagunsodhani.in + url: https://shagunsodhani.github.io email: sshagunsodhani@gmail.com # Custom vars diff --git a/site/_posts/2018-09-12-Emergence of Grounded Compositional Language in Multi-Agent Populations.md b/site/_posts/2018-09-12-Emergence of Grounded Compositional Language in Multi-Agent Populations.md index 4216cccf..4ae7ac1c 100755 --- a/site/_posts/2018-09-12-Emergence of Grounded Compositional Language in Multi-Agent Populations.md +++ b/site/_posts/2018-09-12-Emergence of Grounded Compositional Language in Multi-Agent Populations.md @@ -3,7 +3,7 @@ layout: post title: Emergence of Grounded Compositional Language in Multi-Agent Populations comments: True excerpt: -tags: ['2018', 'AAAI 2018', 'Emergent Language', 'Multi-Agent', Natural Language Processing', AAAI, AI, NLP] +tags: ['2018', 'AAAI 2018', 'Emergent Language', 'Multi-Agent', 'Natural Language Processing', AAAI, AI, NLP] --- ## Introduction diff --git a/site/_posts/2018-10-11-Poincare Embeddings for Learning Hierarchical Representations.md b/site/_posts/2018-10-11-Poincare Embeddings for Learning Hierarchical Representations.md index 41b3dc9a..ae512a0c 100755 --- a/site/_posts/2018-10-11-Poincare Embeddings for Learning Hierarchical Representations.md +++ b/site/_posts/2018-10-11-Poincare Embeddings for Learning Hierarchical Representations.md @@ -3,7 +3,7 @@ layout: post title: Poincaré Embeddings for Learning Hierarchical Representations comments: True excerpt: -tags: ['2017', 'Hyperbolic Embedding', 'Graph Representation', Natural Language Processing', 'Poincare Ball Model', AI, Embedding, Graph, NLP, Tree] +tags: ['2017', 'Hyperbolic Embedding', 'Graph Representation', 'Natural Language Processing', 'Poincare Ball Model', AI, Embedding, Graph, NLP, Tree] --- ## Introduction diff --git a/site/_posts/2019-02-05-Linguistic Knowledge as Memory for Recurrent Neural Networks.md b/site/_posts/2019-02-05-Linguistic Knowledge as Memory for Recurrent Neural Networks.md new file mode 100755 index 00000000..d0c447b1 --- /dev/null +++ b/site/_posts/2019-02-05-Linguistic Knowledge as Memory for Recurrent Neural Networks.md @@ -0,0 +1,46 @@ +--- +layout: post +title: Linguistic Knowledge as Memory for Recurrent Neural Networks +comments: True +excerpt: +tags: ['2017', 'Natural Language Processing', AI, Memory, NLP, QA, RNN] +--- + +* [Link to the paper](https://arxiv.org/abs/1703.02620) + +* Training RNNs to model long term dependencies is difficult but in some cases, the information about dependencies between elements (of the sequence) may be present in the form of symbolic knowledge. + +* For example, when encoding sentences, coreference, and hypernymy relations can be extracted between tokens. + +* These elements(tokens) can be connected with each other with different kind of edges resulting in the graph data structure. + +* One approach could be to model this knowledge(encoded in the graph) using a graph neural network (GNN). + +* The authors prefer to encode the information into 2 DAGs (via topological sorting) as training the GNN could add some extra overhead. + +* This results into the Memory as Acyclic Graph Encoding RNN (MAGE-RNN) architecture. Its GRU version is referred to as MAGE-GRU. + +* Given an input sequence of tokens [x1, x2, ..., xT] and information about which tokens relate to other tokens, a graph G is constructed with different (possibly typed) edges. + + +* Given the graph *G*, two DFS orderings are computed - forward DFS and backward DFS. + +* MAGE-RNN uses separate networks for accessing the forward and backward DFS orders. + +* A separate hidden state is maintained for each entity type to separate memory content from addressing. + +* For any DFS order (forward or backward), the representation at time *t* is given as the concatenation of representation of different edge types at that time. + +* The hidden states (for different edge types at time t) are updated in the topological order using the current state of all incoming edges at xt. + +* The representation of the DFS order is given as the sequence of all the previous representations. + +* In some cases, elements across multiple sequences could be related to each other. In that case, the graph is decomposed into a collection of DAGs and use MAGE-GRU on the DAGs by taking one random permutation of the sequences and decomposing it into the forward and the backward graphs. + +* The model is evaluated on the task of text comprehension with coreference on bAbi dataset (story based QA), LAMBADA dataset (broad context language modeling) and CNN dataset (cloze-style QA). + +* MAGE-GRU was used as a replacement for GRU units in bi-directional GRUs and GA-Reader architecture. + +* DAG-RNN and shared version of MAGE-GRU (with shared edge types) are the other baselines. + +* For all the cases, the model with MAGE-GRU works the best. diff --git a/site/_site b/site/_site index 3179cc9d..3ac6f22c 160000 --- a/site/_site +++ b/site/_site @@ -1 +1 @@ -Subproject commit 3179cc9de4bdc9c769e2e673f84ae0ca37c8da62 +Subproject commit 3ac6f22c8545bfc062147293307ceff4b2bcd674