dissect the design
You gave a therapy, and quality of life went up. Now you want to show that the therapy did that. But a sceptic has another explanation ready in no time: it was just the passing of time, it was that questionnaire beforehand, it only worked because they had already been thinking about it. That is what the Solomon four-group design (Richard L. Solomon, 1949) is for: four groups, one therapy, and in the improvement of each group different pieces: what gets better by itself (history and maturation, H/M), what the therapy does (the treatment, X), what the pretest itself does (the testing effect, T), and what pretest and therapy do together on top of that (pretest sensitization, S). You take them apart, one piece per step, until only S is left. That way you get there before the sceptic does: you can say exactly what you corrected for.
A group of people gets a therapy that is meant to improve their quality of life. You measure that quality of life with a questionnaire; the score runs from 0 to 100, and higher is better. Random assignment divides the people into four groups. Two groups also fill in the questionnaire before the therapy: the pretest. All four fill it in at the end: the posttest. The table shows the mean score per group.
One thing you may assume from the start: random assignment put the groups together, so on average they started out equal. The other assumptions come along the way, at the place where you need them.
steps, and then an encore of : then everything turns out to be a regression. The first step sets down the equation for group I, where all four pieces sit. Each step after that finds one piece and fills it into that equation, until only S is left. If you do not know, open the hint. If you want to see how it goes, open the worked solution. All steps use the same numbers.
This is what you assumed along the way. For each assumption: why that is allowed, and whether the design takes care of it or you measure it.
| what you assume | why that is allowed | does the design take care of it, or do you measure it? |
|---|---|---|
| The four groups started out equal on average. | Random assignment (R) decided who went into which group. | The design takes care of it. One bit you can check: the pretests of I and II are and . |
| Groups III and IV had the same score beforehand as the mean of the pretests of I and II. | That follows from the first row. This is called substitution: you fill in a number in a place where you measured nothing. | The design. Measuring is impossible here: give group III a pretest, and it has become group I. |
| What gets better by itself gets better by the same amount in all four groups. | The groups run through the same period, in the same world. | The design: all groups at the same time. |
| The pieces add up. | That is a convention: what is left in group I after you take away the other pieces, you call S. | A convention. You do not measure it, you choose it. |
Fill it in without looking. Click in a cell to type, or print the page and use a pen.
| effect | which groups do you compare? | the calculation | what you assume |
|---|---|---|---|
| H/M | |||
| X | |||
| T | |||
| S |
Six sliders, laid out like the design table: the two pretests and the four posttests. Groups III and IV borrow their pretest, just as in the ladder; that number moves along. The calculations follow along, in the same order as the ladder: first H/M, then X, then T, and S last. The buttons set up a known case.
Also try: the pretests of I and II not equal. See what happens to T and S then, and to the regression on the posttest.
If you measure before and after in your own research, the improvement you see holds everything at once: what gets better by itself, your intervention, and what the pretest itself did. With one group you cannot take those pieces apart. For that you need groups that differ in one thing only, the way II and IV here differ only in the pretest. So for every effect you want to know, look for the two groups that yield it. If you cannot find them in your design, you cannot see that effect on its own either.
And that is what you can tell the sceptic: “it got better by itself” you have answered as soon as you have a group without therapy, “it was the questionnaire” only once you also have a group without a pretest. Whatever your design cannot answer, you say so yourself, before they ask.
And S is a difference in differences: the effect of the therapy on quality of life with a pretest, minus the effect of the therapy on quality of life without a pretest. That is called an interaction, and that is what chapter 10 of the (Dutch) book, Verschil in verschil, is about.
| here | also called |
|---|---|
| H/M | history and maturation (growing up, recovering) |
| X | the treatment, the intervention |
| T | the testing effect, testing. Watch out: some books call this pretest sensitization, and then they do not mean S. |
| S | pretest sensitization, the interaction of testing and treatment: the interaction of pretest and therapy |
And not every book numbers the groups the way this page does. So always look first at who got a pretest and who got the therapy.
This page works with means, with the borrowed pretest added. That is how Solomon (1949) worked it out himself, and to find S that is fine. If you also want to test whether S really differs from zero, you need more than means: you look at the scores of all the people. There you get stuck, because groups III and IV have no pretest of their own, so no improvement per person either. Campbell and Stanley (1963) therefore recommended another route: test on the posttests, with two factors: pretest (yes or no) and therapy (yes or no), the two switches from the encore. That is exactly the regression on the posttest from the encore, and the interaction in it is S. If the pretests of I and II are equal, it gives the same number as the calculations.
Mind you: the main effect of therapy in such an analysis is the mean of the effect with and without a pretest, ( + ) / 2 = , and not the of b2: that weight is only the effect without a pretest.
From the (Dutch) book De regressieve ruggengraat · Chapter 10 — Verschil in verschil (difference in differences) · sensitization is a difference in differences
The design and the arithmetic: Solomon, R. L. (1949). An extension of control group design. Psychological Bulletin, 46(2), 137–150. https://doi.org/10.1037/h0062958
The test on the posttests: Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Rand McNally.