Improved Temporal Relation Classification using Dependency Parses and Selective Crowdsourced Annotations
Research Area: Natural Language Processing Year: 2012
Type of Publication: In Proceedings Keywords: Temporal relations, information extraction, crowdsourcing, convolution kernels, clause structure, dependency parsing
Authors:
  • Jun-Ping Ng
  • Min-Yen Kan
 
   
Abstract:
We study the problem of classifying the temporal relationship between events and time expres- sions in text. In contrast to previous methods that require extensive feature engineering, our approach is simple, relying only on a measure of parse tree similarity. Our method generates such tree similarity values using dependency parses as input to a convolution kernel. The resulting system outperforms the current state-of-the-art. To further improve classifier performance, we can obtain more annotated data. Rather than rely on expert annotation, we assess the feasibility of acquiring annotations through crowdsourcing. We show that quality temporal relationship annotation can be crowdsourced from novices. By leveraging the problem structure of temporal relation classification, we can selectively acquire annotations on problem instances that we assess as more difficult. Employing this annotation strategy allows us to achieve a classification accuracy of 73.2%, a statistically significant improvement of 8.6% over the previous state-of-the-art, while trimming annotation efforts by up to 37%. Finally, as we believe that access to sufficient training data is a significant barrier to current temporal relationship classification, we plan to share our collected data with the research community to promote benchmarking and comparative studies.
Digital version