GENERATION CHALLENGES 2009
--------------------------

To be held in conjunction with ENLG 2009.

Following the NSF/SIGGEN Workshop on Shared Tasks and Comparative
Evaluation in NLG in April 2007
(http://www.ling.ohio-state.edu/~mwhite/nlgeval07/), and the INLG'06
Special Session on Sharing Data and Comparative Evaluation
(http://www.nltg.brighton.ac.uk/home/Anja.Belz/inlg06-specsess.html),
there has recently been a lot of activity involving shared tasks in
Natural Language Generation (NLG).  The Pilot Attribute Selection for
Generating Referring Expressions (ASGRE) Challenge took place between May
and September 2007; the Referring Expression Generation (REG) Challenge
2008 is currently underway; and other challenges are being prepared.

To provide a common forum for these activities, we are organising
Generation Challenges 2009, an umbrella event designed to bring
together a variety of shared-task evaluation efforts that involve the
generation of natural language.  The following is a provisional list
of tasks that have been proposed so far:

1. The GIVE Challenge (Koller, Byron, Moore, Oberlander & Striegnitz):
   Generation of natural-language instructions to aid human task-solving
   in a virtual environment.

2. GREC Task (Belz, Gatt & Viethen): Generation of references to named
   entities in discourse; exact task definition to be announced.

3. TUNA Progress Test (Gatt & Belz): An opportunity to improve on
   REG'08 Task 3, i.e. mapping from TUNA domain representations to
   referring expressions.

At this point, we invite any team of researchers interested in
organising a shared task as part of the Generation Challenges 2009
event to submit a task proposal.

This is not intended to be a selective process.  Anyone interested in
running a shared task as part of Generation Challenges 2009 is in
principle welcome to do so.  We would simply like to get an idea of how
many tasks there are likely to be for organisational purposes, and
also to be able to offer advice and support where needed.  In the
unlikely event that more tasks are proposed than can be accommodated,
we would make a selection together with the ENLG'09 organisers.

Each team of task organisers will be reponsible for organising and
evaluating their task.  The Generation Challenges 2009 coordinators
will merely coordinate issues relating to the ENLG'09 special session,
presentations and reports.

Please send task proposals to gre-stec@itri.brighton.ac.uk by 31 May
2008.  Please bear in mind the timeline below, and include information
regarding organisers, task description, size and state of completion
of data to be used, evaluation plans and provisional timeline.

Don't hesitate to get in touch with any questions.  Best wishes,

Anja Belz and Albert Gatt


Generation Challenges 2009 Timeline:

Deadline for submission of task proposals       31 May 2008
Recommended first call for participation        Jun 2008
Recommended availability of training data       Jul-Aug 2008
Recommended evaluation period                   Feb 2009
Expected deadline for camera-ready reports      end Feb 2009
ENLG'09 (provisional)                           30-31 March 2009