QA WSD @ CLEF 2008 http://ixa2.si.ehu.es/qawsd/index.php 2014-11-25T16:24:44Z Joomla! 1.5 - Open Source Content Management 2008-01-22T16:43:12Z 2008-01-22T16:43:12Z http://ixa2.si.ehu.es/qawsd/index.php?option=com_content&view=article&id=15:clef-2008-introduction&catid=1:latest Administrator olopezdelaca001@ehu.es

<div> <h1><strong>QA WSD @ CLEF 2008</strong></h1> </div> <div> <h3>Word Sense Disambiguation for Question Answering</h3> <br /> <p class="clr">Following the 2007 joint <a href="http://ixa2.si.ehu.es/qawsd/../semeval-clir" target="_blank">SemEval-CLEF task</a> , a related pilot task is planned with the aim of exploring the contribution of Word Sense Disambiguation to  Question Answering. In this case, the organizers of the task will provide document collections (from the news domain) and topics which have been automatically tagged with Word Senses from WordNet using several state-of-the-art Word Sense Disambiguation systems.  <a href="http://ixa2.si.ehu.es/qawsd/../clirwsd">Robust WSD CLIR @ CLEF</a> is a closely related task on robust retrieval.</p> <h2><strong>News</strong></h2> </div> <div> <div> <div> <div> <ul> <li>2009: The CLEF QA exercise has changed, and thus the QA-WSD exercise won't be repeated. </li> <li> <p>Sep 13: <a href="http://clef.isti.cnr.it/2008/working_notes/CLEF2008WN-Contents1.html" target="_blank"><span style="text-decoration: underline;"><span style="color: #800080;">working notes</span></span></a> available</p> </li> <li> <p>Apr. 24: <strong>New dates for submission (see below) <br /></strong></p> </li> <li> <p>Apr. 14:<strong><strong> </strong></strong>Small bug in WSD results of UBC fixed. Please download again</p> </li> </ul> </div> </div> </div> <h2><strong>Mailing list</strong></h2> <div> <div> <div> <ul> <li>You can <a href="http://groups.google.com/group/qawsd" target="_blank">browse</a> the e-mail list. To join enter your e-mail in form to the left, and press subscribe. </li> </ul> </div> </div> </div> <h2></h2> <h2>Instructions for participation</h2> <p>Please note the following steps in order to participate:</p> <blockquote>1. Please read the main <a href="http://nlp.uned.es/clef-qa/QA@CLEF08_Guidelines-for-Participants.pdf">QA task guidelines</a> and the <a href="http://ixa2.si.ehu.es/qawsd/data/wsd-qa%40clef08_guidelines-for-participants.pdf">specific guidelines for the QA WSD task</a> (<a href="http://ixa2.si.ehu.es/qawsd/data/WSD-QA%40CLEF08_Guidelines-for-Participants.doc">doc</a>).</blockquote> <blockquote>2. <a href="http://www.clef-campaign.org/2008/2008_howto.html">Registration</a> is via the CLEF <a href="http://www.clef-campaign.org/">website</a> <strong>and</strong> the QA <a href="http://nlp.uned.es/clef-qa/">website</a> . Participants must sign an agreement restricting use of the data and regulating publication and dissemination of results. Registration closes 28 April.</blockquote> <blockquote>3.  Join the mailing list (see subscribe button to the left) for updates.</blockquote> <blockquote>4.  Download the document collection from 1 March.</blockquote> <blockquote>5.  Download the English and Spanish WordNets (see below)</blockquote> <blockquote>6.  Download the test topics from 19 May. <strong>(New date)</strong><br /></blockquote> <blockquote>7.  Submit results by 27 May <strong>(New date)</strong> (details for submission to be released).</blockquote> <p><span><strong>IMPORTANT:</strong> </span></p> <ul> <li><span>Remember that the results must be submitted <strong>WITHIN 5</strong> days from the first test sets download, and anyway <strong>not later than May 27</strong>. </span></li> <li><span>Thus, 5 days from downloading - and downloading must be done between 19 and 23 May. </span></li> <li><span>Any submission after 5 days from the download, or after May 27, will not be accepted. </span></li> </ul> <h2>Time Schedule</h2> </div> <div>·        Registration Opens - 15 February 2008 (<strong>closes 28 April 2008</strong>)<br />·        Release of documents with annotations - from 1 March 2008<br />·        Question Release - 19 May 2008<br />·        Submission of Runs by Participants - 23 May 2008, 12 a.m. (Central Europe Time) <br />·        Release of Relevance Assessments and Individual Results - 1 July 2008<br />·        Submission of Paper for Working Notes - 25 July, 12 a.m. (Central Europe Time) 2008<br />·        Workshop - 17-19 September 2008</div> <div><br /> <h2>Description</h2> The QA-WSD task will bring semantic and retrieval evaluation together. The participants will be offered the same queries and document collections from the main QA exercise which have been annotated by systems for word sense disambiguation (WSD). The goal of the task is to test whether WSD can be used beneficially for Question Answering.<br /><br />The exercise scenario is event-targeted QA on a news document collection. In the QA-WSD track only English monolingual and Spanish to English bilingual tasks are offered, i.e. English will be the only target language, and queries will be available on both English and Spanish. The queries will be the same as for the main QA exercise, and the participation will follow the same process, except for the use of the sense-annotated data.<br /><br />A large number of questions will be topic-related, i.e. clusters of questions which are related to the same topic and possibly contain anaphoric references between one question and the other questions. Contrary to the main QA task, Wikipedia articles are not included, and thus systems need to reply to the questions that have an answer in the news document collection.<br /><br />The goal of this task is to evaluate whether word sense information can help in certain queries. For this reason, participants are required to send two runs for each of the monolingual/bilingual tasks where they participate: one which does not use sense annotations and another one which does use sense annotations. Whenever possible, the only difference between the two runs should be solely the use or not of the sense information. Participants which send a single run will be discarded from the evaluation.<br /><br />The WSD data is based on WordNet version 1.6 and will be supplemented with data from the English and Spanish WordNets in order to test different expansion strategies. Several leading WSD experts will run their systems, and provide those WSD results for the participants to use.<br /><br /><br /> <h2><a name="formats" title="formats"></a>Data formats, additional files, and source of WSD tags</h2> <p>These are the DTDs for the disambiguated <a href="http://ixa2.si.ehu.es/qawsd/data/query.wsd.dtd.txt">queries</a> and <a href="http://ixa2.si.ehu.es/qawsd/data/docs.wsd.dtd.txt">documents</a>. Please check these sample disambiguated <a href="http://ixa2.si.ehu.es/qawsd/data/query-example.wsd">query</a> (<a href="http://ixa2.si.ehu.es/qawsd/data/query-example.xml">original</a> ) and <a href="http://ixa2.si.ehu.es/qawsd/data/doc-example.wsd">document</a> (<a href="http://ixa2.si.ehu.es/qawsd/data/doc-example.xml">original</a>).</p> <p>Spanish topics will be disambiguated using the first sense heuristic. English topics and documents will be annotated with the following word sense disambiguation systems:</p> <blockquote> <p>1. Agirre, Eneko &amp; Lopez de Lacalle, Oier (2007). UBC-ALM: Combining k-NN with SVD for WSD. Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval 2007).  pp. 341-345. Prague, Czech Republic.</p> </blockquote> <blockquote> <p>2. Chan, Yee Seng, &amp; Ng, Hwee Tou, &amp; Zhong, Zhi (2007). NUS-PT: Exploiting Parallel Texts for Word Sense Disambiguation in the English All-Words Tasks. Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval 2007).  pp. 253-256. Prague, Czech Republic.</p> </blockquote> <p>In order to expand from WordNet synset numbers to words in English and Spanish, you will need the following:</p> <ul> <li>The English WordNet version 1.6 available from <a href="http://wordnet.princeton.edu/wordnet/download/old-versions/">here</a></li> <li>The Spanish WordNet, available free for research from <a href="http://www.lsi.upc.edu/%7Enlp/web/index.php?option=com_wrapper&amp;Itemid=85">here</a></li> </ul> </div> <div>These are the ad-hoc collections (documents) which were available at CLEF 2001:</div> <div>- LA Times 94 (with WSD data) : 72,027,935 tokens<br /> - Glasgow Herald 95 (with WSD data): 27,731,946 tokens</div> <div> <div> <h2 style="text-align: left"><span style="font-family: Arial; font-size: small;"><span style="font-weight: 400; font-family: Arial"><span style="font-weight: bold">Contact</span></span></span></h2> <div style="text-align: left">Eneko Agirre, University of the Basque Country, e.agirre at ehu es</div> <div style="text-align: left">Anselmo Peñas, UNED, anselmo at lsi uned es</div> <p class="MsoNormal" style="margin: 3pt 0cm; text-align: left"><span style="font-family: Arial; font-size: small;"> </span></p> </div> </div>

<div> <h1><strong>QA WSD @ CLEF 2008</strong></h1> </div> <div> <h3>Word Sense Disambiguation for Question Answering</h3> <br /> <p class="clr">Following the 2007 joint <a href="http://ixa2.si.ehu.es/qawsd/../semeval-clir" target="_blank">SemEval-CLEF task</a> , a related pilot task is planned with the aim of exploring the contribution of Word Sense Disambiguation to  Question Answering. In this case, the organizers of the task will provide document collections (from the news domain) and topics which have been automatically tagged with Word Senses from WordNet using several state-of-the-art Word Sense Disambiguation systems.  <a href="http://ixa2.si.ehu.es/qawsd/../clirwsd">Robust WSD CLIR @ CLEF</a> is a closely related task on robust retrieval.</p> <h2><strong>News</strong></h2> </div> <div> <div> <div> <div> <ul> <li>2009: The CLEF QA exercise has changed, and thus the QA-WSD exercise won't be repeated. </li> <li> <p>Sep 13: <a href="http://clef.isti.cnr.it/2008/working_notes/CLEF2008WN-Contents1.html" target="_blank"><span style="text-decoration: underline;"><span style="color: #800080;">working notes</span></span></a> available</p> </li> <li> <p>Apr. 24: <strong>New dates for submission (see below) <br /></strong></p> </li> <li> <p>Apr. 14:<strong><strong> </strong></strong>Small bug in WSD results of UBC fixed. Please download again</p> </li> </ul> </div> </div> </div> <h2><strong>Mailing list</strong></h2> <div> <div> <div> <ul> <li>You can <a href="http://groups.google.com/group/qawsd" target="_blank">browse</a> the e-mail list. To join enter your e-mail in form to the left, and press subscribe. </li> </ul> </div> </div> </div> <h2></h2> <h2>Instructions for participation</h2> <p>Please note the following steps in order to participate:</p> <blockquote>1. Please read the main <a href="http://nlp.uned.es/clef-qa/QA@CLEF08_Guidelines-for-Participants.pdf">QA task guidelines</a> and the <a href="http://ixa2.si.ehu.es/qawsd/data/wsd-qa%40clef08_guidelines-for-participants.pdf">specific guidelines for the QA WSD task</a> (<a href="http://ixa2.si.ehu.es/qawsd/data/WSD-QA%40CLEF08_Guidelines-for-Participants.doc">doc</a>).</blockquote> <blockquote>2. <a href="http://www.clef-campaign.org/2008/2008_howto.html">Registration</a> is via the CLEF <a href="http://www.clef-campaign.org/">website</a> <strong>and</strong> the QA <a href="http://nlp.uned.es/clef-qa/">website</a> . Participants must sign an agreement restricting use of the data and regulating publication and dissemination of results. Registration closes 28 April.</blockquote> <blockquote>3.  Join the mailing list (see subscribe button to the left) for updates.</blockquote> <blockquote>4.  Download the document collection from 1 March.</blockquote> <blockquote>5.  Download the English and Spanish WordNets (see below)</blockquote> <blockquote>6.  Download the test topics from 19 May. <strong>(New date)</strong><br /></blockquote> <blockquote>7.  Submit results by 27 May <strong>(New date)</strong> (details for submission to be released).</blockquote> <p><span><strong>IMPORTANT:</strong> </span></p> <ul> <li><span>Remember that the results must be submitted <strong>WITHIN 5</strong> days from the first test sets download, and anyway <strong>not later than May 27</strong>. </span></li> <li><span>Thus, 5 days from downloading - and downloading must be done between 19 and 23 May. </span></li> <li><span>Any submission after 5 days from the download, or after May 27, will not be accepted. </span></li> </ul> <h2>Time Schedule</h2> </div> <div>·        Registration Opens - 15 February 2008 (<strong>closes 28 April 2008</strong>)<br />·        Release of documents with annotations - from 1 March 2008<br />·        Question Release - 19 May 2008<br />·        Submission of Runs by Participants - 23 May 2008, 12 a.m. (Central Europe Time) <br />·        Release of Relevance Assessments and Individual Results - 1 July 2008<br />·        Submission of Paper for Working Notes - 25 July, 12 a.m. (Central Europe Time) 2008<br />·        Workshop - 17-19 September 2008</div> <div><br /> <h2>Description</h2> The QA-WSD task will bring semantic and retrieval evaluation together. The participants will be offered the same queries and document collections from the main QA exercise which have been annotated by systems for word sense disambiguation (WSD). The goal of the task is to test whether WSD can be used beneficially for Question Answering.<br /><br />The exercise scenario is event-targeted QA on a news document collection. In the QA-WSD track only English monolingual and Spanish to English bilingual tasks are offered, i.e. English will be the only target language, and queries will be available on both English and Spanish. The queries will be the same as for the main QA exercise, and the participation will follow the same process, except for the use of the sense-annotated data.<br /><br />A large number of questions will be topic-related, i.e. clusters of questions which are related to the same topic and possibly contain anaphoric references between one question and the other questions. Contrary to the main QA task, Wikipedia articles are not included, and thus systems need to reply to the questions that have an answer in the news document collection.<br /><br />The goal of this task is to evaluate whether word sense information can help in certain queries. For this reason, participants are required to send two runs for each of the monolingual/bilingual tasks where they participate: one which does not use sense annotations and another one which does use sense annotations. Whenever possible, the only difference between the two runs should be solely the use or not of the sense information. Participants which send a single run will be discarded from the evaluation.<br /><br />The WSD data is based on WordNet version 1.6 and will be supplemented with data from the English and Spanish WordNets in order to test different expansion strategies. Several leading WSD experts will run their systems, and provide those WSD results for the participants to use.<br /><br /><br /> <h2><a name="formats" title="formats"></a>Data formats, additional files, and source of WSD tags</h2> <p>These are the DTDs for the disambiguated <a href="http://ixa2.si.ehu.es/qawsd/data/query.wsd.dtd.txt">queries</a> and <a href="http://ixa2.si.ehu.es/qawsd/data/docs.wsd.dtd.txt">documents</a>. Please check these sample disambiguated <a href="http://ixa2.si.ehu.es/qawsd/data/query-example.wsd">query</a> (<a href="http://ixa2.si.ehu.es/qawsd/data/query-example.xml">original</a> ) and <a href="http://ixa2.si.ehu.es/qawsd/data/doc-example.wsd">document</a> (<a href="http://ixa2.si.ehu.es/qawsd/data/doc-example.xml">original</a>).</p> <p>Spanish topics will be disambiguated using the first sense heuristic. English topics and documents will be annotated with the following word sense disambiguation systems:</p> <blockquote> <p>1. Agirre, Eneko &amp; Lopez de Lacalle, Oier (2007). UBC-ALM: Combining k-NN with SVD for WSD. Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval 2007).  pp. 341-345. Prague, Czech Republic.</p> </blockquote> <blockquote> <p>2. Chan, Yee Seng, &amp; Ng, Hwee Tou, &amp; Zhong, Zhi (2007). NUS-PT: Exploiting Parallel Texts for Word Sense Disambiguation in the English All-Words Tasks. Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval 2007).  pp. 253-256. Prague, Czech Republic.</p> </blockquote> <p>In order to expand from WordNet synset numbers to words in English and Spanish, you will need the following:</p> <ul> <li>The English WordNet version 1.6 available from <a href="http://wordnet.princeton.edu/wordnet/download/old-versions/">here</a></li> <li>The Spanish WordNet, available free for research from <a href="http://www.lsi.upc.edu/%7Enlp/web/index.php?option=com_wrapper&amp;Itemid=85">here</a></li> </ul> </div> <div>These are the ad-hoc collections (documents) which were available at CLEF 2001:</div> <div>- LA Times 94 (with WSD data) : 72,027,935 tokens<br /> - Glasgow Herald 95 (with WSD data): 27,731,946 tokens</div> <div> <div> <h2 style="text-align: left"><span style="font-family: Arial; font-size: small;"><span style="font-weight: 400; font-family: Arial"><span style="font-weight: bold">Contact</span></span></span></h2> <div style="text-align: left">Eneko Agirre, University of the Basque Country, e.agirre at ehu es</div> <div style="text-align: left">Anselmo Peñas, UNED, anselmo at lsi uned es</div> <p class="MsoNormal" style="margin: 3pt 0cm; text-align: left"><span style="font-family: Arial; font-size: small;"> </span></p> </div> </div>