Evaluation is a key bottleneck in much NLP and IR research. To decide whether a summary is a good summary or a search result is relevant, there is often little substitute for having human participants examine the results of a computer algorithm. Our group practices joint evaluation exercises, in which members of the group are voluntarily asked to help each other out in becoming subjects for evaluation.

Evaluation tools

Computer Centre has acquired the Eazy-Survey software package for those of you who need to do a survey as part of your research. See the following URL for more information: http://www.nus.edu.sg/comcen/apps/ezsurvey.htm.

Aside from that, if you're willing to learn perl you can use some of Min's code to customize a survey and link it to a simple Berkeley DB database to store and retrieve your results.

WING-eval group

Visitors, if you have time, feel free to join our evaluation pool. Periodically we will send out notices to the WING-eval mailing list announcing new evaluations that require human subjects. Subjects are paid between 8 to 15 dollars an hour for their participation in evaluations. See below for details about current and open evaluations.

All current evaluations / experiments

N.B. if the project below is older than three months and a project duration is not specified, you may want to check with the maintainer to see whether the evaluation is still active.