Informativeness for Adhoc IR evaluation: a measure that prevents assessing individual documents

Romain Deveaud (1, 2) , Véronique Moriceau (3) , Josiane Mothe (1, 4) , Eric San Juan (2)

1 IRIT-SIG - Systèmes d’Informations Généralisées
2 LIA - Laboratoire Informatique d'Avignon
3 LIMSI - Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur
4 UT2J - Université Toulouse - Jean Jaurès

Informativeness measures have been used in interactive in- formation retrieval and automatic summarization evaluation. Indeed, as opposed to adhoc retrieval, these two tasks cannot rely on the Cranfield evaluation paradigm in which retrieved documents are compared to static query relevance document lists. In this paper, we explore the use of informativeness measures to evaluate adhoc task. The advantage of the proposed evaluation framework is that it does not rely on an exhaustive reference and can be used in a changing environment in which new documents occur, and for which relevance has not been assessed. We show that the correlation between the official system ranking and the informativeness measure is specifically high for most of the TREC adhoc tracks.

Mots clés: Information retrieval; Evaluation; Informativeness; Adhoc retrieval; SIGEVI

Domaines: Théorie de l'information [cs.IT]; Recherche d'information [cs.IR]

Romain Deveaud, Véronique Moriceau, Josiane Mothe, Eric San Juan. Informativeness for Adhoc IR evaluation: a measure that prevents assessing individual documents. 38th European Conference on Information Retrieval (ECIR 2016), Mar 2016, Padoue, Italy. pp.818-823, ⟨10.1007/978-3-319-30671-1_73⟩. ⟨hal-01567065⟩