C. R. Pernet, M. Latinus, T. E. Nichols, G. A. Rousselet


Publication year



Periodical Number

1872-678X (Electronic)



Author Address

Centre for Clinical Brain Sciences, Neuroimaging Sciences, University of Edinburgh, Edinburgh, UK. Electronic address: FAU - Latinus, M Institut de Neurosciences de la Timone UMR 7289, Aix Marseille Universite, CNRS, 13385 Marseille, France. FAU - Nichols, T E Department of Statistics, Warwick University, Coventry, UK. FAU - Rousselet, G A Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, UK.

Full version

BACKGROUND: In recent years, analyses of event related potentials/fields have moved from the selection of a few components and peaks to a mass-univariate approach in which the whole data space is analyzed. Such extensive testing increases the number of false positives and correction for multiple comparisons is needed. METHOD: Here we review all cluster-based correction for multiple comparison methods (cluster-height, cluster-size, cluster-mass, and threshold free cluster enhancement – TFCE), in conjunction with two computational approaches (permutation and bootstrap). RESULTS: Data driven Monte-Carlo simulations comparing two conditions within subjects (two sample Student’s t-test) showed that, on average, all cluster-based methods using permutation or bootstrap alike control well the family-wise error rate (FWER), with a few caveats. CONCLUSIONS: (i) A minimum of 800 iterations are necessary to obtain stable results; (ii) below 50 trials, bootstrap methods are too conservative; (iii) for low critical family-wise error rates (e.g. p=1%), permutations can be too liberal; (iv) TFCE controls best the type 1 error rate with an attenuated extent parameter (i.e. power<1).