Reasoning in NLP

Abstract

Our current deep learning based NLP paradigm (circa 2021) has inherent limitations. A large model is trained on large corpora, resulting in a strong capacity for memorizing and imitating. However, it only learns data correlations in the training corpora. This can lead to many problems even for state-of-the-art models (e.g. BERT), for example: (1) the loss of human commonsense in the model; (2) failing to explain “why” for machine decision; (3) bias; (4) failing to extrapolate to unseen instances.

In this project, we hypothesize reasoning is a promising approach to address the limitation of the current NLP paradigm. We are interested in commonsense reasoning (for world knowledge), causal reasoning (for supporting evidence in text), and human-in-the-loop reasoning (for human’s input when machine reaches its upper bound). 

Reasoning in NLP has been explored by our community for a long time. Among all directions, we are particularly interested in unsupervised/self-supervised methods. This is because human annotation is expensive, and it is impossible to annotate every NLP phenomenon. We believe unsupervised/self-supervised methods are smart and efficient approaches to bridge an unseen problem with existing NLP models and knowledge sources.

Project Members

Publications

To come.

Progress

Yisong is currently exploring reasoning in a classic NLP task. He has made preliminary efforts with unsupervised methods and causal intervention. He will share the preprints when ready. 

Recourses

Yisong’s reading list on GitHub. 

Meetings

Due to semi-privacy, we don’t publicize Yisong’s meeting with Min. Our internal WING members can DM me through Slack for slides. External guests can request by emailing Yisong.

References

  • Cyc: toward programs with common sense. Lenat DB, Guha RV, Pittman K, Pratt D, Shepherd M. Communications of the ACM. 1990 Aug 1;33(8):30-49. [PDF@acm]
  • Unsupervised commonsense question answering with self-talk. Shwartz V, West P, Bras RL, Bhagavatula C, Choi Y.  EMNLP 2020. [PDF@aclweb]

Image credit: Sebastian Ortuzar from Pinterest.