Developing new methods for evaluation of document summaries

Text summarization is the task of forming a summary of a given document. There are two types of summaries: extracts (just selecting some of the important sentences in the document as the summary) and abstracts (generating new sentences from scratch to form the summary). In this project, we deal only with extract type of summarization. The classical metric used to evaluate the success of a summarization system is the Rouge metric.

We will begin the project by reading the paper “ROUGE: A Package for Automatic Evaluation of Summaries”, Lin, C-Y., Workshop On Text Summarization Branches Out, 2004. We will collect a set of documents and their summaries. Then we will implement the Rouge methods explained in the paper to evaluate the goodness of these summaries. Then we will try to develop more advanced techniques for summary evaluation.