Example Project

we investigate whether a generative LM (e.g., GPT- 2) is able to access its memory sequentially or randomly. Through carefully-designed synthetic tasks, we reveal that LMs manage to sequentially access their memory while encountering challenges in randomly accessing memorized content.

We propose Discursive Socratic Questioning (DiSQ), a new evaluation measure for discourse semantics. Inspired by the Socratic method, DiSQ involves asking models about key event relations, testing their robustness to counterfactuals, and ensuring consistency with equivalent questions. Experiments show that GPT-4 achieves only 41% of the DiSQ scores. We recommend using context and discourse connectives as essential linguistic features to enhance discourse comprehension.

Can emojis convey intricate meanings like English phrases? We present the Emoji-Lexical Composition (ELCo) dataset with 1,655 instances spanning 209 concepts, from tangible to abstract ideas. ELCo enables analysis of emoji and lexical composition. Our Emoji-based Textual Entailment (EmoTE) task reveals challenges for current models, but fine-tuning improves performance significantly.