Is there Anything Wrong with the MTMM Approach to Question Evaluation?

In 1959 Campbell and Fiske introduced the Multitrait Multimethod (MTMM) approach to evaluate the convergence and discriminant validity of measurement instruments. 1984 Frank Andrews adjusted the procedure to estimate the quality of questions for survey research. He also did the first meta-analysis across many experiments to evaluate the effect of the different question characteristics on the quality of the questions. After his death my research group continued his valuable work by doing more experiments, doing a new meta analysis and developing a program for prediction of the quality of questions (Oberski, Kuipers, & Saris, 2005) on the bases of the findings of the meta analysis (Saris & Gallhofer, 2007). In 2001 the European Social Survey (ESS) decided to include in its data collection MTMM experiments to evaluate the quality of questions, to compare the quality across countries and to correct for possible differences in quality across these countries. In order to apply this approach in the ESS the design of the MTMM experiment was adjusted to reduce memory effects by repetition of the same questions. Saris, Satorra, and Coenders (2004) developed for this purpose the Split Ballot MTMM design. This design has been used in all rounds of the ESS in more than 20 countries.In a recent book, edited by Madans, Miller, Maitland and Willis (2011) published by Wiley in the series on survey research three authors criticize the MTMM approach. Because these criticisms come from three very well known and respected scholars, Duane Alwin, Jon Krosnick and Peter Mohler and they have been published in the very prestigious survey research series of Wiley I feel obliged to indicate that there is nothing wrong with the MTMM approach but that the criticisms are unjustified.