Frequently asked questions
This page lists all the questions that we have been asked by participants during the ASGRE and REG Challenges (updated as appropriate to current procedures)..
Q: Can members/contributors be added to a participating team after the initial registration?
A: Yes, you can update your registration at any point.
Q: Does the use of the term training data in the Participants Pack imply that this is a competition only for statistical and machine-learning approaches?
A: No, absolutely not. Any type of method, approach or system can participate.
-
Q: About the size of training/test set - will this be the whole TUNA corpus?
A: No, the Shared Tasks just involve singulars (780 items). Of this, we'll be holding back about 20% for the test set. The remainder will be divided into training (60%) and development set (20%).
Q: Do I need to run the evaluation software for just the training set (but not the test set), just to report these results in the submission?
A: Yes, you'll need to run the evaluation software that we provide on the training and development sets only (to put scores in your report), not on the test data.
Q: Some of the referring expressions in the TUNA corpus are unambiguous or inaccurate (do not describe the referent), why is this?
A: The corpus records referring expressions produced by experimental subjects. While in most cases subjects produced accurate and unambiguous referring expressions, in a few cases they did not.
Q: In the TUNA corpus, why is a hairColour attribute given for a subject with hasHair:0?
A: hairColour records the colour of a person's hair or the colour of a person's beard; "facialHairColour" might perhaps have been a better name. So it can appear with hasBeard:1 as well as hasHair:1. If a subject has both hair and a beard, they always have the same hairColour.
Q: In the TUNA corpus, are there multiple descriptions for the same referential task?
A: Sort of. Some trials have the same entity as the target referent and the same set of distractors, but the target referent is always in a different position.
Thus, if you look at all attribute sets for a particular entity (identifiable by its ID), and discard the location attributes (x-dimension, y-dimension), then you will in a sense have multiple descriptions for the same entity.
Q: If I'm reading Sect 4.8 correctly, then we can look at Irene's template realizer w/o that affecting eligibility to enter tasks 2-3. Is that right?
A: No, the idea is that if you reuse a module then you can't enter the corresponding task. So if you reuse Irene's realiser, then you can't enter TUNA Task 2, and if you reuse one of the attribute selection modules then you can't enter TUNA Task 1.
Q: In the training data s19t2, I found in the attribute-set twice a
y-dimensionwith different values. The word string would make sense for attributename="x-dimension",value="1"name="y-dimension" value="2". This could be wrong?A: Thanks for pointing this out. This is an error on the part of the annotators who originally annotated the TUNA corpus. Unfortunately, these errors tend to arise in any data set of reasonable size, either because annotators disagree, or simply through oversight.
Given that such errors seem inevitable, it would be unfeasible to follow up each error report of this kind by regenerating the data sets. It seems more reasonable to allow people to take action regarding the errors they find on their own initiative, particularly because such errors tend to be rare.
Q: I have accidentally deleted my test data! Can I download it again?
A: Yes. Visit this download page to do so.
Q: I can / have to write for each of the tasks (tracks) (eg. tuna-as, tuna-r, tuna-reg) up to two pages. In total 3x2 = 6 pages or only 2 pages in total?
A: One report for each track/team combination, so a maximum of 6 pages in your case. But the 2-page limit for each report is a maximum, so you could always e.g. have a very short REG report, and slightly longer AS and R reports. The main thing is not to repeat content.
Further questions
If you have further questions, please .
Last modified: 2008-04-02 14:58