Our current deep learning based NLP paradigm (circa 2021) has inherent limitations. A large model is trained on large corpora, resulting in a strong capacity for memorizing and imitating. However, it only learns data correlations in the training corpora. This can lead to many problems even for state-of-the-art models (e.g. BERT), for example: (1) the loss of human commonsense in the model; (2) failing to explain “why” for machine decision; (3) bias; (4) failing to extrapolate to unseen instances.
In this project, we hypothesize reasoning is a promising approach to address the limitation of the current NLP paradigm. We are interested in commonsense reasoning (for world knowledge), causal reasoning (for supporting evidence in text), and human-in-the-loop reasoning (for human’s input when machine reaches its upper bound).
Reasoning in NLP has been explored by our community for a long time. Among all directions, we are particularly interested in unsupervised/self-supervised methods. This is because human annotation is expensive, and it is impossible to annotate every NLP phenomenon. We believe unsupervised/self-supervised methods are smart and efficient approaches to bridge an unseen problem with existing NLP models and knowledge sources.
Yisong is currently exploring reasoning in a classic NLP task. He has made preliminary efforts with unsupervised methods and causal intervention. He will share the preprints when ready.
Yisong’s reading list on GitHub.
Due to semi-privacy, we don’t publicize Yisong’s meeting with Min. Our internal WING members can DM me through Slack for slides. External guests can request by emailing Yisong.
- Cyc: toward programs with common sense. Lenat DB, Guha RV, Pittman K, Pratt D, Shepherd M. Communications of the ACM. 1990 Aug 1;33(8):30-49. [PDF@acm]
- Unsupervised commonsense question answering with self-talk. Shwartz V, West P, Bras RL, Bhagavatula C, Choi Y. EMNLP 2020. [PDF@aclweb]