(QA)$^2$: Question Answering with Questionable Assumptions

• Najoung Kim, Phu Mon Htut, Samuel R. Bowman and Jackson Petty

[ACL] [arXiv]

Abstract. Naturally occurring information-seeking questions often contain questionable assumptions—assumptions that are false or unverifiable. Questions containing questionable assumptions are challenging because they require a distinct answer strategy that deviates from typical answers for information-seeking questions. For instance, the question “When did Marie Curie discover Uranium?” cannot be answered as a typical “when” question without addressing the false assumption “Marie Curie discovered Uranium”. In this work, we propose (QA)$^2$ (Question Answering with Questionable Assumptions), an open-domain evaluation dataset consisting of naturally occurring search engine queries that may or may not contain questionable assumptions. To be successful on (QA)$^2$, systems must be able to detect questionable assumptions and also be able to produce adequate responses for both typical information-seeking questions and ones with questionable assumptions. Through human rater acceptability on end-to-end QA with (QA)$^2$, we find that current models do struggle with handling questionable assumptions, leaving substantial headroom for progress.

    title = "({QA})$^2$: Question Answering with Questionable Assumptions",
    author = "Kim, Najoung  and
      Htut, Phu Mon  and
      Bowman, Samuel R.  and
      Petty, Jackson",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.472",
    doi = "10.18653/v1/2023.acl-long.472",
    pages = "8466--8487",