Introduction
- R-NET is an end-to-end trained neural network model for machine comprehension.
- It starts by matching the question and the given passage (using gated attention based RNN) to obtain question-aware passage representation.
- Next, it uses a self-matching attention mechanism to refine the passage representation by matching the passage against itself.
- Lastly, it uses pointer networks to determine the position of the answer in the passage.
- Link to the paper
Datasets
- SQuAD
- MS-MARCO
Architecture
- Question / Passage Encoder
- Concatenate the word level and character level embeddings for each word and feed into a bidirectional GRU to obtain question and passage representation.
- Gated Attention based RNN
- Given question and passage representation, sentence pair representation is generated via soft-alignment of the words in the question and in the passage.
- The newly added gate captures the relation between the question and the current passage word as only some parts of the passage are relevant for answering the given question.
- Self Matching Attention
- The passage representation obtained so far would not capture most of the context.
- So the current representation is matched against itself so as to collect evidence from the entire passage and encode the evidence relevant to the current passage word and question.
- Output Layer
- Use pointer network (initialized using attention pooling over answer representation) to predict the position of the answer.
- Loss function is the sum of negative log probabilities of start and end positions.
- Results
- R-NET is ranked second on SQuAD Leaderboard as of 7th August, 2017 and achieves best-published results on MS-MARCO dataset.
- Using ideas like sentence ranking, using syntax information performing multihop inference and augmenting question dataset (using seqToseq network) do not help in improving the performance.
