Listener behavior
Both, MUSHRA and ITU BS.1116 tests call for trained expert listeners who know what typical artifacts sound like and where they are likely to occur. Expert listeners also have a better internalization of the rating scale which leads to more repeatable results than with untrained listeners. Thus, with trained listeners, fewer listeners are needed to achieve statistically significant results. It is assumed that preferences are similar for expert listeners and naive listeners and thus results of expert listeners are also predictive for consumers. In agreement with this assumption Schinkel-Bielefeld et al. found no differences in the rank order between expert listeners and untrained listeners when using test signals containing only timbre and no spatial artifacts. However, Rumsey et al. showed that for signals containing spatial artifacts, expert listeners weigh spatial artifacts slightly stronger than untrained listeners, who primarily focus on timbre artifacts. In addition to this, it has been shown that expert listeners make more use of the option to listen to smaller sections of the signals under test repeatedly and perform more comparisons between the signals under test and the reference. In contrast to the naive listener who produce a preference rating, expert listeners therefore produce an audio quality rating, rating the differences between the signal under test and the uncompressed original, which is the actual goal of a MUSHRA-test.Pre- or post-screening
The MUSHRA guideline mentions several possibilities to assess the reliability of a listener. The easiest and most common is to disqualify listeners who rate the hidden reference below 90 MUSHRA points for more than 15 percent of all test items. The hidden reference should be rated with 100 MUSHRA points so this is obviously a mistake. While it can happen that the hidden reference and a high-quality signal are confused, a rating of lower than 90 should only be given when the listener is certain that the rated signal is different than the original reference. The other possibility to assess a listener's performance is eGauge, a framework based on the analysis of variance. It computes ''agreement'', ''repeatability'' and ''discriminability'', though only the latter two are recommended for pre or post screening. ''Agreement'' analyses how well a listener agrees with the rest of the listeners. ''Repeatability'' looks at the variance when rating the same test signal again in comparison to the variance of the other test signals and ''discriminability'' analyses if listeners can distinguish between test signals of different conditions. As eGauge requires listening to every test signal twice, it is more effort to apply this than to post screen listeners based on the ratings of the hidden reference. However, if a listener has proven a reliable listener using eGauge, he or she can also be considered a reliable listener for future listening tests, provided the character of the test does not change; A reliable listener for stereo listening test is not necessarily equally good in perceiving artifacts in 5.1 or 22.2 format test items.Test items
It is important to choose critical test items; items which are difficult to encode and are likely to produce artifacts. At the same time, the test items should be ecologically valid; they should be representative of broadcast material and not some synthetic signals especially designed to be difficult to encode. A method to choose critical material is presented by Ekeroot et al. who propose a ranking by elimination procedure. While this is a good way to choose the most critical test items, it does not ensure inclusion a variety of test items prone to different artifacts. Ideally the character of a MUSHRA test item should not change too much for the whole duration of that item. Otherwise it can be difficult for the listener to decide on a rating if different parts of the items display different or stronger artifacts than others. Often shorter items lead to less variability than longer ones, as they are more stationary. However, even when trying to choose stationary items, ecologically valid stimuli will very often have sections that are slightly more critical than the rest of the signal. Thus, listeners who focus on different sections of the signal may evaluate it differently. In this case more critical listeners seem to be better in identifying the most critical regions of a stimulus than less critical listeners.Language of test items
While in ITU-T P.800 tests which are commonly used to evaluate telephone quality codecs the tested speech items should always be in the native language of the listeners, this is not necessary in MUSHRA tests. A study with Mandarin Chinese and German listeners found no significant difference between rating foreign language and native language test items. However, listeners needed more time and compared more when evaluating the foreign language items. So it seems that listeners compensate for any difficulties they may have in rating foreign language items. Such compensation is not possible in ITU-T P.800 ACR tests where items are heard only once and no comparison to the reference is possible. There, foreign language items are rated as being of lower quality when listeners' language proficiency is low.References
{{reflistExternal links