Incorporating structure into neural models for language processing

Abstract

Structured data is abundant in the world, as is the multitude of NLP applications seeking to perform inferences over such data. Despite their success, modern neural network models often struggle to incorporate structured information. In this thesis, we investigate how to build effective neural network models to incorporate structured data for natural language understanding. Graphs are a natural form of representation for structural information, and the recently proposed Graph Neural Networks (GNNs) allow neural networks to perform inference over graphs through learnable message passing functions. We begin by introducing effectively the first GNN model suitable for the directed, multirelational data found in common forms of structured data relevant to NLP applications, such as knowledge bases (KBs). We apply our technique as a structural encoder for relational link prediction, achieving state-of-the-art results. We then introduce two variants for factoid question answering over KBs, relying either on choosing individual answer vertices or on choosing a best path to the answer. We furthermore propose a novel model for fact verification over open collections of tables, combining a RoBERTa-encoder for linearised tables with a cross-attention mechanism for fusing evidence documents. A significant challenge is the uninterpretable, black-box nature of such encoders. To alleviate this problem, we finally introduce GraphMask, a novel technique for interpreting the predictions of GNNs. We apply this technique to analyze the predictions of two NLP models from the literature