The 2nd Computational Linguistics Scientific Document Summarization Shared Task (CL-SciSumm 2016)

Results were presented at the BIRNDL 2016 Workshop, a workshop co-located after
Joint Conference on Digital Libraries (JCDL '16), Newark, New Jersey, USA
Task Sponsored by Microsoft Research Asia

Last updated: Fri Dec 2 20:08:05 SGT 2016

 

 

Call for Participation

You are invited to participate in the CL-SciSumm Shared Task at BIRNDL 2016. The shared task will be on automatic paper summarization in the Computational Linguistics (CL) domain. The output summaries will be of two types: faceted summaries of the traditional self-summary (the abstract) and the community summary (the collection of citation sentences ‘citances’). We also propose to group the citances by the facets of the text that they refer to.

This task follows up on the successful CL Pilot Task conducted as a part of the BiomedSumm Track at the Text Analysis Conference 2014 (TAC 2014). In this task, a training corpus of ten topics from CL research papers was released. Participants were invited to enter their systems in a task-based evaluation. Nine teams from four countries expressed an interest in participating in the shared task; three teams submitted system descriptions and findings. We also released the SciSumm14 manually annotated dataset, comprising of ACL Computational Linguistics research papers and summaries. It offers a community summary of a reference paper based on its collection of citing sentences “citances”. Furthermore, each citance is mapped to its referenced text in the reference paper and tagged with the information facet it represents. In our proposed shared task, we will extend this by releasing pairs of training and test datasets – each pair comprising the annotated citing sentences for a research paper, and the summaries of the research paper.

The CL-SciSumm 2016 corpus is expected to be of interest to a broad community including those working in computational linguistics and natural language processing, text summarization, discourse structure in scholarly discourse, paraphrase, textual entailment and text simplification.

We have secured support for the costs of the shared task annotation from Microsoft Research Asia. The National University of Singapore will be primarily responsible for the task's oversight.

Tasks

Given: A topic consisting of a Reference Paper (RP) and up to ten Citing Papers (CPs) that all contain citations to the RP. In each CP, the text spans (i.e., citances) have been identified that pertain to a particular citation to the RP.

Task 1A: For each citance, identify the spans of text (cited text spans) in the RP that most accurately reflect the citance. These are of the granularity of a sentence fragment, a full sentence, or several consecutive sentences (no more than 5).

Task 1B: For each cited text span, identify what facet of the paper it belongs to, from a predefined set of facets.

Task 2 (optional bonus task): Finally, generate a structured summary of the RP from the cited text spans of the RP. The length of the summary should not exceed 250 words.

Evaluation: Task 1 will be scored by overlap of text spans measured by number of sentences in the system output vs the gold standard created by human annotators. Task 2 will be scored using the ROUGE family of metrics between (i) the system output and the gold standard summary from the reference spans (ii) the system output and the asbtract of the reference paper. Again, Task 2 is optional.

Registration

The 2016 shared task has concluded. However you can still download the corpus and publish your results at other venues. If you use the data and publish please let us know and cite this paper:
@inproceedings{jaidka2016overview,
title={Overview of the CL-SciSumm 2016 Shared Task},
author={Jaidka, Kokil and Chandrasekaran, Muthu Kumar and Rustagi, Sajal and Kan, Min-Yen},
booktitle={In Proceedings of Joint Workshop on Bibliometric-enhanced Information Retrieval and NLP for Digital Libraries (BIRNDL 2016)},
year={2016}
}

Organizations wishing to participate in the CL Shared Task track at BIRNDL 2016 are invited to register on EasyChair by 30 March 2016. Participants are advised to register as soon as possible in order to receive timely access to evaluation resources, including development and testing data. Registration for the task does not commit you to participation - but is helpful to know for planning. All participants who submit system runs are welcome to present their system at the BIRNDL Workshop.

Dissemination of CL-SciSumm work and results other than in the workshop proceedings is welcomed, but the conditions of participation specifically preclude any advertising claims based on these results. Any questions about conference participation may be sent to the organizers mentioned below.

 

 

Corpus

https://github.com/WING-NUS/scisumm-corpus

The corpus for the CL-SciSumm task has been created by randomly sampling ten documents from the ACL Anthology corpus and selecting their citing papers. The training, development and testing set will be made publicly available at the GitHub link above at dates specified below.

The training set of 10 articles is already available for download and can be used by participants to pilot their systems. The development set of 10 articles, a part of the same corpus, will be released in April, which participants can add to the training set to tune their system parameters. Finally the test set of 10 articles will be released in May. The system outputs from the test set should be submitted to the task organizers, for the collation of the final results to be presented at the workshop.

 

 

Important Dates

Please consult the BIRNDL Workshop for official dates for the workshop.

EventDate
Call for Participation releasedFebruary 2016
Training Set ReleasedFebruary 2016
Deadline for Registration and Short System Descriptions Due30 March 2016
Development Set Posted8 April 2016
Notification of Acceptance of Presentation Proposals22 April 2016
Test Set Released7 May 2016
System Runs and Preliminary System Reports Due in EasyChair20 May 2016
Camera-Ready Contributions Due in EasyChair3 June 2016
Participant Presentations at BIRNDL Workshop23 June 2016 in Newark, New Jersey, USA

All deadlines for the CL-SciSumm shared task are calculated as 11:59pm Baker Island Time (BIT: UTC/GMT-12). Block booking of the hotel rooms by participants need to be done before 15 May 2016, to ensure early bird rates.

 

 

Background

The CL-SciSumm task provides resources to encourage research in a promising direction of scientific paper summarization, which considers the set of citation sentences (i.e., "citances") that reference a specific paper as a (community created) summary of a topic or paper (Nanba, Kando and Okumura, 2011; Qazvinian and Radev, 2008). Citances for a reference paper are considered a synopses of its key points and also its key contributions and importance within an academic community. The advantage of using citances is that they are embedded with meta-commentary and offer a contextual, interpretative layer to the cited text. The drawback, however, is that though a collection of citances offers a view of the cited paper, it does not consider the context of the target user (Sparck Jones, 2007; Teufel and Moens, 2002; Nenkova and McKeown, 2011; Jaidka, Khoo and Na, 2013a), verify the claim of the citation or provide context from the reference paper, in terms the type of information cited or where it is in the referenced paper.

CL-SciSumm explores summarization of scientific research, for the computational linguistics research domain. An ideal summary of computational linguistics research papers would be able to summarize previous research by drawing comparisons and contrasts between their goals, methods and results, as well as distil the overall trends in the state of the art and their place in the larger academic discourse. Literature surveys and review articles in CL do help readers to gain a gist of the state-of-the-art in research for a topic. However, literature survey writing is labor-intensive and a literature survey is not always available for every topic of interest. What are needed, are resources which automate the synthesis and updating of automatic summaries of CL research papers.

Existing scientific summarization systems have automatically generated related work sections for a target paper by instantiating a hierarchical topic tree (Hoang and Kan, 2010), generating model citation sentences (Mohammad et al., 2009) or implementing a literature review framework (Jaidka et al., 2013). However, the limited availability of evaluation resources and human-created summaries constrains research in this area. The goal of the TAC 2014 CL Shared Task is to highlight the challenges and relevance of the scientific summarization problem, support research in automatic scientific document summarization and provide evaluation resources to push the current state of the art.

 

 

BIRNDL 2016 Workshop

http://wing.comp.nus.edu.sg/birndl-jcdl2016

The BIRNDL 2016 workshop will be held on 23 June 2016, in New Jersey, USA. The workshop is a forum both for presentation of results (including failure analyses and system comparisons), and for more lengthy system presentations describing techniques used, experiments run on the data, and other issues of interest to NLP researchers. TAC track participants who wish to give a presentation during the workshop will submit a short abstract describing the experiments they performed. As there is a limited amount of time for oral presentations, the abstracts will be used to determine which participants are asked to speak and which will present in a poster session.

 

 

Organising Committee

Muthu Kumar Chandrasekaran - muthu.chandra@comp.nus.edu.sg

I am broadly interested in natural language processing and its applications to information retrieval; specifically, in retrieving and organising information from asynchronous conversation media such as scholarly publications: the mainstream medium of scholarly communication such as discussion forums and debate forums. Rich rhetorical discourse in their content can be effectively used to achieve a desired organisation such as summarisation. Communication of scholarly research needs to be summarised to avoid redundant or outdated research and ensure faster progress to pressing problems. I helped create a small corpus and organise a pilot task for scientific document summarisation in the 2014 Text Analysis Conference. I am currently a doing my Ph.D. research on a similarly motivated problem on Massive Open Online Course (MOOC) discussion forums on recommending salient student discussions for instructors to intervene given their limited bandwidth.

Kokil Jaidka - kokil@pmail.ntu.edu.sg

Dr Kokil Jaidka is a Ph.D. in Information Studies from Nanyang Technological University, Singapore. Her expertise is in multidocument summarization, natural language processing and applied linguistics. Currently, in her capacity as a computer scientist, she is conducting social media analyses to solve problems for user behavior profiling and topic detection. She is also exploring transaction records in the digital marketing domain for research problems related to outbound marketing. Beyond information science, she is also the data specialist for an international, collaborative political communication research project on Asian democracies.

Min-Yen Kan - kanmy@comp.nus.edu.sg

My research interests fall under the areas of digital libraries, natural language processing, information retrieval, human-computer interaction. Specifically, they include document structure acquisition, verb analysis, digital library resource annotation and and applied text summarization. My research goal aims to investigate how natural language processing and information retrieval can be applied to improve scholarly publication and knowledge discovery.