Background
Indexing and classification methods to assist with information retrieval have a long history dating back to the earliest libraries and collections however systematic evaluation of their effectiveness began in earnest in the 1950s with the rapid expansion in research production across military, government and education and the introduction of computerised catalogues. At this time there were a number of different indexing, classification and cataloguing systems in operation which were expensive to produce and it was unclear which was the most effective. Cyril Cleverdon, Librarian of the College of Aeronautics, Cranfield, England, began a series of experiments of print indexing and retrieval methods in what is known as the Cranfield paradigm, or Cranfield tests, which set the standard for IR evaluation measures for many years. Cranfied developed a test called ‘known-item searching’ - to check whether an IR system returned the documents that were known to be relevant or correct for a given search. Cranfield’s experiments established a number of key aspects required for IR evaluation: a test collection, a set of queries and a set of pre-determined relevant items which combined would determine precision and recall. Cleverdon's approach formed a blueprint for the successfulApplications
Evaluation of IR systems is central to the success of any search engine including internet search, website search, databases and library catalogues. Evaluations measures are used in studies of information behaviour,Online measures
Online metrics are generally created from search logs. The metrics are often used to determine the success of an A/B test.Session abandonment rate
Session abandonment rate is a ratio of search sessions which do not result in a click.Click-through rate
Session success rate
Session success rate measures the ratio of user sessions that lead to a success. Defining "success" is often dependent on context, but for search a successful result is often measured using dwell time as a primary factor along with secondary user interaction, for instance, the user copying the result URL is considered a successful result, as is copy/pasting from the snippet.Zero result rate
''Zero result rate'' (''ZRR'') is the ratio of Search Engine Results Pages (SERPs) which returned with zero results. The metric either indicates a recall issue, or that the information being searched for is not in the index.Offline metrics
Offline metrics are generally created from relevance judgment sessions where the judges score the quality of the search results. Both binary (relevant/non-relevant) and multi-level (e.g., relevance from 0 to 5) scales can be used to score each document returned in response to a query. In practice, queries may bePrecision
Precision is the fraction of the documents retrieved that areRecall
Recall is the fraction of the documents that are relevant to the query that are successfully retrieved. : In binary classification, recall is often called sensitivity. So it can be looked at as ''the probability that a relevant document is retrieved by the query''. It is trivial to achieve recall of 100% by returning all documents in response to any query. Therefore, recall alone is not enough but one needs to measure the number of non-relevant documents also, for example by computing the precision.Fall-out
The proportion of non-relevant documents that are retrieved, out of all non-relevant documents available: : In binary classification, fall-out is closely related to specificity and is equal to . It can be looked at as ''the probability that a non-relevant document is retrieved by the query''. It is trivial to achieve fall-out of 0% by returning zero documents in response to any query.F-score / F-measure
The weighted harmonic mean of precision and recall, the traditional F-measure or balanced F-score is: : This is also known as the measure, because recall and precision are evenly weighted. The general formula for non-negative real is: : Two other commonly used F measures are the measure, which weights recall twice as much as precision, and the measure, which weights precision twice as much as recall. The F-measure was derived by van Rijsbergen (1979) so that "measures the effectiveness of retrieval with respect to a user who attaches times as much importance to recall as precision". It is based on van Rijsbergen's effectiveness measure . Their relationship is: : where F-measure can be a better single metric when compared to precision and recall; both precision and recall give different information that can complement each other when combined. If one of them excels more than the other, F-measure will reflect it.Average precision
Precision and recall are single-value metrics based on the whole list of documents returned by the system. For systems that return a ranked sequence of documents, it is desirable to also consider the order in which the returned documents are presented. By computing a precision and recall at every position in the ranked sequence of documents, one can plot a precision-recall curve, plotting precision as a function of recall . Average precision computes the average value of over the interval from to : : That is the area under the precision-recall curve. This integral is in practice replaced with a finite sum over every position in the ranked sequence of documents: : where is the rank in the sequence of retrieved documents, is the number of retrieved documents, is the precision at cut-off in the list, and is the change in recall from items to . This finite sum is equivalent to: : where is an indicator function equaling 1 if the item at rank is a relevant document, zero otherwise. Note that the average is over relevant documents in top-k retrieved documents and the relevant documents not retrieved get a precision score of zero. Some authors choose to interpolate the function to reduce the impact of "wiggles" in the curve. For example, the PASCAL Visual Object Classes challenge (a benchmark for computer vision object detection) until 2010 computed the average precision by averaging the precision over a set of evenly spaced recall levels : : where is an interpolated precision that takes the maximum precision over all recalls greater than : :. An alternative is to derive an analytical function by assuming a particular parametric distribution for the underlying decision values. For example, a ''binormal precision-recall curve'' can be obtained by assuming decision values in both classes to follow a Gaussian distribution.Precision at k
For modern (web-scale) information retrieval, recall is no longer a meaningful metric, as many queries have thousands of relevant documents, and few users will be interested in reading all of them.R-precision
R-precision requires knowing all documents that are relevant to a query. The number of relevant documents, , is used as the cutoff for calculation, and this varies from query to query. For example, if there are 15 documents relevant to "red" in a corpus (R=15), R-precision for "red" looks at the top 15 documents returned, counts the number that are relevant turns that into a relevancy fraction: . R-Precision is equal to recall at the R-th position. Part of ''Introduction to Information Retrieval'Mean average precision
Mean average precision (MAP) for a set of queries is the mean of theDiscounted cumulative gain
DCG uses a graded relevance scale of documents from the result set to evaluate the usefulness, or gain, of a document based on its position in the result list. The premise of DCG is that highly relevant documents appearing lower in a search result list should be penalized as the graded relevance value is reduced logarithmically proportional to the position of the result. The DCG accumulated at a particular rank position is defined as: : Since result set may vary in size among different queries or systems, to compare performances the normalised version of DCG uses an ideal DCG. To this end, it sorts documents of a result list by relevance, producing an ideal DCG at position p (), which normalizes the score: : The nDCG values for all queries can be averaged to obtain a measure of the average performance of a ranking algorithm. Note that in a perfect ranking algorithm, the will be the same as the producing an nDCG of 1.0. All nDCG calculations are then relative values on the interval 0.0 to 1.0 and so are cross-query comparable.Other measures
*Visualization
Visualizations of information retrieval performance include: * Graphs which chart precision on one axis and recall on the other * Histograms of average precision over various topics *Non-relevance measures
Queries per time
Measuring how many queries are performed on the search system per (month/day/hour/minute/sec) tracks the utilization of the search system. It can be used for diagnostics to indicate an unexpected spike in queries, or simply as a baseline when comparing with other metrics, like query latency. For example, a spike in query traffic, may be used to explain a spike in query latency.See also
*References
{{Machine learning evaluation metrics Information retrieval Information retrieval evaluation Internet search engines