The beta-binomial: A preliminary comparison of smaller samples having many replications versus larger samples having fewer replications

The experiment investigated whether a body of difference test data, arranged into small consumer samples with many replications, would elicit more or less overdispersion than an equal number of tests from same body of data, rearranged into a series of larger consumer samples, with fewer replications. Four separate experiments were performed, each with a separate set of 200 consumers. In two of the experiments, consumers used 2‐AFC tests while in the remaining two, consumers used triangle tests. Data were then assigned to samples of different size ranging from 100 to 28. The number of replicate tests assigned for each sample size was arranged so that the total number of tests performed was always close to 200. For the two experiments involving 2‐AFC tests, smaller physical differences were assessed in one experiment and larger physical differences in the other. Same was true for the two experiments using triangle tests. For the two experiments where smaller physical differences were assessed, overdispersion tended to increase as sample sizes became larger. For the two experiments where larger physical differences were assessed, overdispersion tended to decrease as sample sizes became larger. These apparently contradictory results were explained by the sensitivity variation hypothesis. Practical applications: Beta‐binomial statistics are used for significance testing for difference tests, when the experimenter combines the number of subjects with the number of replicate tastings, to increase test power. This is to circumvent the problems of overdispersion, the extra variance involved when this combination is made. Nearly all the work on overdispersion is theoretical. Very little experimental work has been published. From prior experimentation, the sensitivity variation hypothesis was developed, which indicates that overdispersion simply depends on the performance of the particular consumers in the sample and is not governed by the type of test used. This experiment successfully applied the sensitivity variation hypothesis to a set of seemingly contradictory results, regarding the overdispersion elicited in a series of four experiments. It provides further much needed experimental insight, into the effects of sample size and numbers of replications as well as the sources of overdispersion per se.