Conference And Labs Of The Evaluation Forum

The Conference and Labs of the Evaluation Forum (formerly Cross-Language Evaluation Forum), or CLEF, is an organization promoting research in multilingual information access (currently focusing on European languages). Its specific functions are to maintain an underlying framework for testing information retrieval systems and to create repositories of data for researchers to use in developing comparable standards.
The organization holds a conference every September in Europe since a first constituting workshop in 2000. From 1997 to 1999, TREC, the similar evaluation conference organised annually in the USA, included a track for the evaluation of Cross-Language IR for European languages. This track was coordinated jointly by NIST and by a group of European volunteers that grew over the years. At the end of 1999, a decision by some of the participants was made to transfer the activity to Europe and set it up independently. The aim was to expand coverage to a larger number of languages and to focus on a wider range of issues, including monolingual system evaluation for languages other than English. Over the years, CLEF has been supported by a number of various EU funded projects and initiatives.
Peters, Carol, and Martin Braschler. "European research letter: Cross‐language system evaluation: The CLEF campaigns." Journal of the Association for Information Science and Technology 52.12 (2001): 1067-1072.

CLEF 2019 marked the 20th anniversary of the conference and it was celebrated by publishing a book on the lessons learned in 20 years of evaluation activities.

Structure of CLEF

Before 2010, CLEF was organised as a workshop co-located with the European Conference on Digital Libraries, consisting of a number of evaluation labs or tracks, similarly to TREC. In 2010, CLEF moved to become a self-sufficiently organised conference with evaluation labs, laboratory workshops, and a main conference track. In 2012, INEX, a workshop on retrieval and access to structured text, previously organised annually at Schloß Dagstuhl, merged with CLEF to become one of its evaluation labs.

Prior to each CLEF conference, participants in evaluation labs receive a set of challenge tasks. The tasks are designed to test various aspects of information retrieval systems and encourage their development. Groups of researchers propose and organize campaigns to satisfy those tasks and the results are used as benchmarks for the state of the art in the specific areas.,Fredric C. Gey, Noriko Kando, and Carol Peters "Cross-Language Information Retrieval: the way ahead" in ''Information Processing & Management''
vol. 41, no. 3, p.415-431 May 2005,

In the beginning, CLEF focussed mainly on fairly typical information retrieval tasks, but has moved to more specific tasks. For example, the 2005 interactive image search task worked with illustrating non-fiction texts using images from Flickr. and the 2010 medical retrieval task focused on retrieval of computed tomography, MRI, and radiographic images. In 2017, CLEF accommodated a number of tasks e.g. on identifying biological species from photographs or video clips, on stylistic analysis of authorship, and on health related information access.

List of CLEF workshops and conferences

References

External links

CLEF homepage

CLEF history before 2010

Information retrieval organizations

{{Compu-conference-stub