QuaRel: A Dataset and Models for Answering Questions about Qualitative\n Relationships

Abstract

Many natural language questions require recognizing and reasoning with\nqualitative relationships (e.g., in science, economics, and medicine), but are\nchallenging to answer with corpus-based methods. Qualitative modeling provides\ntools that support such reasoning, but the semantic parsing task of mapping\nquestions into those models has formidable challenges. We present QuaRel, a\ndataset of diverse story questions involving qualitative relationships that\ncharacterize these challenges, and techniques that begin to address them. The\ndataset has 2771 questions relating 19 different types of quantities. For\nexample, "Jenny observes that the robot vacuum cleaner moves slower on the\nliving room carpet than on the bedroom carpet. Which carpet has more friction?"\nWe contribute (1) a simple and flexible conceptual framework for representing\nthese kinds of questions; (2) the QuaRel dataset, including logical forms,\nexemplifying the parsing challenges; and (3) two novel models for this task,\nbuilt as extensions of type-constrained semantic parsing. The first of these\nmodels (called QuaSP+) significantly outperforms off-the-shelf tools on QuaRel.\nThe second (QuaSP+Zero) demonstrates zero-shot capability, i.e., the ability to\nhandle new qualitative relationships without requiring additional training\ndata, something not possible with previous models. This work thus makes inroads\ninto answering complex, qualitative questions that require reasoning, and\nscaling to new relationships at low cost. The dataset and models are available\nat http://data.allenai.org/quarel.\n