Second movement · Coupling differences

Chapter 3 — The Little Rectangle

Covariance, correlation and simple regression

1. The return

The next morning the diamonds were still lying on the moss.

The crow picked one up in his beak and held it up to the light. “This one is big. And it shines quite a bit.”

“But this one is smaller and shines more,” said the beaver, pointing.

“And this one is big, but… dull,” said the gull.

The elephant looked out over all twelve for a while, the way only elephants look — without hurry, without touching anything.

“The big ones mostly shine more,” he said. “Mostly.”

And there, right in the middle of that mostly, lay that one big dull one.

 

Mostly. That word is going to carry this chapter. Not always — mostly. The big ones mostly shine more; big mostly goes together with shiny. That’s a new kind of observation: not about one diamond, and not about the average diamond either, but about how their differences join in with each other.

And now a game. Suppose the dam catches one more tonight — a thirteenth. The beaver puts it on the scales: 1.8 carats. The shine meter is somewhere under the moss and nobody can be bothered to look for it. What do you guess?

Chapter 1 has an answer to that, and it’s not a bad one: guess the mean, 50, and you’ll be 21.74 off, on average. That was all you could do then — you knew nothing about this stone, so you guessed what you knew about all the stones.

But now you do know something. You know its carat. And you’ve just heard the elephant say that the big ones mostly shine more. 1.8 is on the big side — so your guess goes up. Above 50. Everyone by the water can feel that.

How much above 50, nobody can feel. That you have to work out. And to work it out, you first have to know exactly what the elephant saw: how strongly do big and shiny actually go together, in this chest? That’s this chapter. The stone stays on the moss in the meantime. At the end I’ll come back to give it its number — and then I’ll tell you honestly what a number like that is worth.

2. The cloud and its middle

Table 3.1. The diamonds from the chest, with their carat (the unit of weight for gemstones, 1 ct = 0.2 g) and shine (shine meter, scale 0–100).
id name carat shine (0–100)
1 Anna 1.0 20
2 Bram 1.1 30
3 Cees 1.1 30
4 Dora 1.3 30
5 Else 1.6 50
6 Faas 1.6 70
7 Geer 1.7 80
8 Hans 1.8 60
9 Ida 1.9 70
10 Joop 1.9 80
11 Kees 2.1 30
12 Leen 2.1 50

The twelve are listed in table 3.1, with their carat and their shine — you’ll need them throughout this chapter, and this way you don’t have to go back to chapter 1 for them.

Back to the cloud of points chapter 1 closed with: carat on one axis, shine on the other, twelve points. Statisticians call the crossing of the two dashed lines — the mean carat, the mean shine — the centroid: the balance point of the cloud. And at that crossing — in our case, by chance, not by law — somebody lives: Else. The average diamond to the last decimal; we’ve known that since chapter 1, honest footnote and all: the centroid is usually an empty address, and our cloud happens to have a resident. Figure 3.1 draws the cloud again, with its middle circled.

The same scatterplot as at the end of chapter 1, carat against shine, every name included. Around Else, at the crossing of the two dashed lines of the means, is a burgundy ring with the word centroid in burgundy. To the right of it, at carat 1.8, a burgundy dotted line runs the full height, with no end point, straight through Hans; at the bottom there is a question mark: the thirteenth stone, its shine still unknown. Else, Faas and Leen sit exactly on a dashed line and belong to none of the four parts. At carat 1.1 and shine 30, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2.

Figure 3.1. The cloud from chapter 1, now with its balance point circled; the two dashed lines divide the twelve into four parts, and at carat 1.8 the thirteenth stone is waiting for its shine.

Else is going to teach us something more in this chapter, and it’s no small thing: whoever is exactly average takes part in none of the sums. She says nothing. She only counts — as one of the twelve.

And somewhere to the right of that crossing, at carat 1.8, lies the stone from a moment ago: we know its place on one axis, not its height on the other. That’s why, in figure 3.1 above, it stands as a question mark at the bottom, with a dotted line going up that ends nowhere.

From that crossing, we’re going to look. Not at the whole cloud at once — at one diamond at a time.

3. High or low? — the arms

Pick any diamond from the cloud. First question: is it high or low on carat? So: look only left and right — and ignore the height completely for a moment. That’s harder than it sounds: your eyes want to see the whole position, but I’m only asking about one axis at a time.

Say the answer is low. In class I let my right arm hang — low, dangling a little, with exaggerated relish. Figure 3.2 is a strip of four small sketches, and picture 1 does the same with an arrow: from the mean to the stone, along the bottom.

Four small hand-drawn sketches side by side, numbered 1 to 4, each the same faint cloud with one stone drawn large and dark. 1: a stone at the bottom left; an arrow along the carat axis runs from the dashed line of the mean leftwards to it, labelled low. 2: the same stone, now also with the dashed line of the mean shine, and a second arrow down along the shine axis, also labelled low. 3: a stone at the top right; both arrows point the other way, high and high. 4: a stone at the bottom right; the carat arrow says high, the shine arrow says low.

Figure 3.2. The arms, one axis at a time. Each arrow runs from the mean to the stone: high on carat is to the right, high on shine is upwards. Picture 1: only the carat arm. Picture 2: the same stone, now with the mean shine as a dashed line too, and the shine arm added. Picture 3: a stone at the top right. Picture 4: that one big dull one.

Second question, same diamond: and on shine — high or low? Low as well. Down goes my left arm: down too, dangling too. Two arms, the same way — in picture 2 that’s a second arrow, along the left-hand side.

Now a diamond at the top right of the cloud. High on carat — arm up. High on shine — the other arm up too. Two arms the same way again, only now both in the air (picture 3).

What did these two diamonds have in common?

In class I let the room flounder here for a moment. With you, all I can do is hold the page still.

Not their position — one sat at the bottom left, the other at the top right. What they had in common: both arms pointed the same way. Low-low and high-high are, for this question, the same kind of diamond: one that chose the same direction on both axes.

And then there are diamonds where the arms contradict each other: high on carat, low on shine. One arm up, one down — picture 4, and you’ll recognise it: the elephant’s big dull one. Its arrows contradict each other: this stone doesn’t join in with the elephant’s mostly.

Last question of the scene: which diamond goes furthest on both variables? Not the one that sits furthest out on one axis — the one that sits far from the middle on both at once. To catch ‘far on X’ and ‘far on Y’ in one number, you need a product. Multiplying. And when you multiply two distances, you get —

there it is again: an area.

And now all the poses at once, in figure 3.3 — you’ve just been through four of them.

Nine stick figures on squared paper in three rows of three, shine running upwards and carat to the right. A dashed vertical line marks exactly average on carat, a dashed horizontal line exactly average on shine. Each figure holds one arm for carat and one for shine, up, flat or down: the left arm is carat, c, and the right arm shine, s, as the letters on the bottom-left figure say. Under each figure is the sign of its contribution: plus, in green, at the bottom left (both arms down) and the top right (both arms up); minus, in burgundy, at the top left and the bottom right, where the arms disagree; and 0 for the five figures on a dashed line, where at least one arm is flat.

Figure 3.3. Nine arm poses, one for each combination of low, average and high: one arm (c) does carat, the other (s) does shine, and the sign underneath says what that pose contributes — arms together, arms against each other, or one arm flat.

4. The rectangle

In chapter 1 we turned every miss into a square: the deviation times itself. Now we have two misses per diamond — one on carat, one on shine. Multiply them together and you don’t get a square but a rectangle (a square is only a rectangle whose sides happen to match): width = the carat deviation, height = the shine deviation.

\[(X_i - \overline{X})(Y_i - \overline{Y})\]

Look at Anna first, in figure 3.4, a strip of four small sketches. In picture 1 her two misses are the blue strokes from chapter 1, only now there’s one lying sideways as well.

Four small hand-drawn sketches side by side, numbered 1 to 4, each the same faint cloud with the two dashed lines of the means. 1: Anna, a large dark dot at the bottom left, with two blue strokes, one up to the dashed line of the mean shine and one sideways to the dashed line of the mean carat. 2: the same two strokes now enclose a green rectangle with a plus in it. 3: a stone at the bottom right, Kees, encloses a burgundy rectangle with a minus in it. 4: a stone directly above the crossing, Faas, has only one short blue stroke, upwards, and a 0 beside it: no rectangle.

Figure 3.4. A little rectangle is born. Picture 1: Anna’s two misses, each from its own dashed line of the mean: a width and a height. Picture 2: together they span a patch, and it counts: plus. Picture 3: Kees spans a patch that counts against: minus. Picture 4: Faas differs only on shine; his width is zero, so he spans no patch: 0.

Multiply those two, and a patch appears (picture 2). Minus times minus is plus, so Anna counts on the plus side.

That rectangle can be three things:

  • Positive — both arms the same way (low-low or high-high: minus × minus = plus, plus × plus = plus).
  • Negative — arms opposite (high-low or low-high: plus × minus = minus).
  • Nothing — and this is the most important of the three. A diamond that is exactly average on one of the two axes has a rectangle with a side of zero. No area. Co means together — a diamond that differs on only one variable doesn’t vary together. That one doesn’t join in.

The last two are in pictures 3 and 4. Kees — that one big dull one from picture 4 of figure 3.2 — spans a patch that counts against him: plus times minus is minus. And Faas spans nothing. Width zero, no area: he doesn’t join in.

With twelve diamonds we can simply count who contributes what, in table 3.2:

Table 3.2. The twelve cross-products (‘rectangles’).
name carat deviation shine deviation rectangle contribution
Anna −0.6 −30 +18 positive
Bram −0.5 −20 +10 positive
Cees −0.5 −20 +10 positive
Dora −0.3 −20 +6 positive
Else 0 0 0 nothing — sits right on the crossing
Faas 0 +20 0 nothing — differs only on shine
Geer +0.1 +30 +3 positive
Hans +0.2 +10 +2 positive
Ida +0.3 +20 +6 positive
Joop +0.3 +30 +9 positive
Kees +0.5 −20 −10 negative — there he is again
Leen +0.5 0 0 nothing — differs only on carat
The same scatterplot with names, carat against shine, with the two dashed lines of the means. From each dot a rectangle is spanned to the crossing. At the bottom left and the top right they are green, for; where they overlap, the green gets darker. At the bottom right lies one burgundy rectangle, against: Kees's. Else, Faas and Leen lie on a dashed line and span nothing. Under the figure stand the three sums. At carat 1.1 and shine 30, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2. Their two rectangles coincide exactly.

Figure 3.5. Each diamond spans its rectangle to the crossing: green counts for, burgundy counts against — and a diamond on a dashed line has nothing to span.

Who wins? Figure 3.5 shows it. The positives bring in 64 between them, Kees takes back 10, three diamonds don’t join in. Sum: +54. So the cloud clearly leans one way: bigger goes with shinier. Kees objects, and loses.

And then we do what we always do with a sum: divide by the number of truly free misses. The average rectangle:

\[S_{XY} = \frac{\sum (X_i - \overline{X})(Y_i - \overline{Y})}{n-1}\tag{3.1}\]

\[\begin{aligned} S_{XY} &= \frac{\sum (X_i - \overline{X})(Y_i - \overline{Y})}{n-1} \\[0.4em] &= \frac{54}{11} \\[0.4em] &= 4.91 \end{aligned}\]

This number is called the covariance. It is moving together, on average: to association what the average square was to spread. For the stone on the moss, that plus sign means: the cloud leans upwards, so your guess does too. But how much — 4.91 carat-shines tells you nothing about that. Nobody thinks in carat-shines.

5. The same rectangle, in rulers

Is 4.91 a lot?

You can’t say, and that’s not for lack of experience. Had the shine meter counted to a thousand instead of a hundred, this would say 49.1 — a rectangle ten times the size, without anything about the cloud changing. Not a point moved. Only the marks on the meter. Figure 3.6 puts the two one above the other.

Two panels one above the other with the same twelve stones, carat against shine, without names, with the dashed lines of the means. In each panel Anna's rectangle, the dot at the bottom left, is coloured green up to the crossing. To the eye the two drawings are the same; only the numbers along the shine axis differ, up to a hundred at the top, up to a thousand at the bottom, and with those numbers the average rectangle given in each panel. In both panels, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2.

Figure 3.6. Exactly the same cloud, twice. At the top, the shine meter up to a hundred, average rectangle 4.91; at the bottom, the same stones on a meter up to a thousand, rectangle 49.1. Not a point moved — only the marks on the meter.

So a number like that is raw: it hangs on whatever scale somebody once happened to decide to measure on. And raw is rubbish. We know the way out from chapter 1 — measure both in their own ruler. Carat’s ruler is 0.39, shine’s is 21.74.

A box drawn on squared paper, 0.39 carats wide and 21.74 shine high: one ruler by one ruler, with an area of 8.48. Inside it, against the left-hand side, a green rectangle with 4.91 in it, the average rectangle, filling a little over half the box. Underneath: 4.91 divided by 8.48 is .58, which is r.
Figure 3.7. The box of one ruler by one ruler, and inside it the average rectangle: it fills .58 of it.

Those two rulers span a rectangle too (figure 3.7): 0.39 carats wide, 21.74 shine high, area 0.39 × 21.74 = 8.48. That’s the rectangle of one ruler by one ruler, and you can measure any other rectangle against it. How many times does our average rectangle fit into it? 4.91 divided by 8.48 is .58 times.

That number is called the correlation.

\[r_{XY} = \frac{S_{XY}}{S_X \cdot S_Y}\tag{3.2}\]

\[\begin{aligned} r_{XY} &= \frac{S_{XY}}{S_X \cdot S_Y} \\[0.4em] &= \frac{4.91}{0.39 \times 21.74} \\[0.4em] &= .58 \end{aligned}\]

The same rectangle as in section 4 — only no longer measured in carat-shines, but in rulers. Carat cancels against carat, shine against shine, and what’s left is a bare number without a unit. Had the cloud leaned the other way — heavy going with dull — there’d be a minus here.

And one thing you may take on trust from me, because we’re not going to prove it here: the average rectangle never fits more than once into the ruler-by-ruler rectangle. So the correlation always sits between −1 and +1. At −1 all the points lie exactly on a falling diagonal, at 0 the cloud leans neither way, at +1 they lie exactly on a rising diagonal. Our .58 sits somewhere in between: a cloud that clearly leans, and clearly wobbles. Have a look at the five small clouds in figure 3.8 — the fourth is ours.

Five small square clouds side by side, each of twelve points, with no numbers along the axes; above each cloud stands its correlation. From left to right: all the points on a falling diagonal, a loose falling cloud, a cloud that leans neither way, then our own diamonds, leaning up at a slant and wobbling, and at the far right all the points on a rising diagonal. In the fourth cloud, ours, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2.

Figure 3.8. Five small clouds of twelve points, from r = −1 to r = +1; the fourth cloud is our own diamonds. Bare axes, because there is nothing to read off them.

Notice what is not in this number: a direction. Carat with shine is the same rectangle as shine with carat — width times height is height times width. The covariance knows no order, and so neither does the correlation. They say how strongly two things move together, and leave open who follows whom. Hold on to that, because in the next section we’re going to choose a side.

effect size how big it is also: effect-size measure

Now that this number is on the table, I can tell you what kind of number it is. A number that says how big something is — how strong an association, how hefty a difference — is called an effect size. Our .58 is one. That’s all it is, and that’s exactly its strength: it answers the question how big is it? and keeps out of everything else — including the question of whether the association is real. If you ever read an article in which an association is merely ‘demonstrated’, without a word about how big it is, you now know what’s missing.

If you want to go further — r from z-scores.

You can also do it the other way round: instead of expressing the rectangle in rulers afterwards, measure each deviation in its own ruler beforehand — standardising — and only then multiply. Then r is the average rectangle of those ruler distances. That’s the z-score version, and the book has more to say about it later.

6. The slope

The correlation says how strongly. But for the stone on the moss you wanted to know how much — and nobody thinks in rulers, any more than in carat-shines. You want shine per carat.

For that, you reach for the raw rectangle again. Raw is rubbish, I said — but handy for some formulas, and this is one of them. On top of the fraction sits carat-times-shine: the rectangles. Divide that by carat-times-carat — the squares of carat from chapter 1, which together make 1.68 (that’s also where the ruler of 0.39 came from: 1.68 divided by eleven, and the square root) — and what’s left is shine per carat. That’s not just a trick with units: a rectangle divided by a square of the same width is height-per-width, and height per width is a slope.

\[b_1 = \frac{S_{XY}}{S^2_X}\tag{3.3}\]

\[\begin{aligned} b_1 &= \frac{S_{XY}}{S^2_X} \\[0.4em] &= \frac{54/11}{1.68/11} \\[0.4em] &= \frac{54}{1.68} \\[0.4em] &= 32.14 \end{aligned}\]

The eleven appears top and bottom, so it cancels: you might just as well divide the sum of products by the sum of squares. Same answer, and no rounding crumbs along the way.

And look what happens to the units: carat-times-shine divided by carat-squared = shine per carat. This number says something, in plain words: each extra carat goes, on average, with 32 more shine points. Figure 3.9 draws that line through the cloud.

The same scatterplot with names, carat against shine. Only the dashed line of the mean shine is still there, and from each dot a thin blue stroke runs to that line, the misses from chapter 1; Else and Leen lie on it and have none. Through it all a straight green line climbs from bottom left to top right, crossing the dashed line exactly at Else. At carat 1.1 and shine 30, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2.

Figure 3.9. The same cloud, now with the line through it: from the dashed line of the mean, as in chapter 1, it climbs 32 shine points per carat.

And this is where the word comes in: the line that runs through the cloud in such a way that the misses are as small as possible is called the regression line. Mind you: a miss is now no longer the distance to the mean, because you’re no longer guessing with the mean — it’s the distance to the line. And ‘as small as possible’ means exactly what chapter 1 would lead you to expect: not the misses themselves (they add up to zero anyway, plus and minus cancel out), but their squares — the line makes the sum of the squared misses as small as it can. Statisticians call it the least-squares line. Why the word regression — that story is still to come.

least squares the smallest sum of squares also: OLS (ordinary least squares)

And that line runs through Else — through the crossing of the two means. Why, you’ll see in section 7; that it does, you can use right away. Because then you can walk from her to the stone: from carat 1.6 to 1.8 is 0.2 carats further, times 32 is a good 6 more shine. Your guess slides from 50 to 56 or thereabouts.

That’s not a formula yet — it’s a walk from someone you happen to know. In section 7 we turn it into something that’s also of use to someone who has never met Else.

A quick word about per

Read the unit of b out loud once more: 32.14 shine points per carat.

That’s something you’ve been able to do all your life. Cheese per kilo, kilometres per hour, euros per square metre — all of them differences per difference. Nobody calls them that, and everyone works with them. So this is no new kind of number; it’s a price per kilo, with shine instead of euros — table 3.3:

Table 3.3. Three rates in a row — the cheese, the road and our regression line — with, for each, the promise and the small print.
what it says what it does not say
€8.50 per kilo 1 kilo more costs €8.50 more not every piece of cheese costs €8.50 — that depends on its weight
90 km per hour 1 hour longer is 90 km further you don’t cover 90 km every hour — you stop at traffic lights
32.14 shine per carat 1 carat heavier goes with 32.14 more shine points not every heavier stone is brighter — that’s Kees

That right-hand column matters more than the left, because 32.14 easily sounds like a promise to every single stone — ours included. It isn’t one.

Look at the prepositions for a moment, because all three are in there: a difference in shine per difference in carat, between the diamonds. In points to the variables, between points to the stones, and per says how the two move together. The unit does the teaching.

The same slope, in rulers

Now back to the correlation one more time, because it was a rate already — you just couldn’t see it yet. One ruler more carat is 0.39 carats; times the rate of 32.14, that’s 12.53 more shine. And 12.53 shine, divided by a shine ruler of 21.74, is .58 of a shine ruler. There’s our r again, and now it comes with a sentence you can remember:

One ruler more carat, and your guess for shine goes up .58 of a ruler.

Watch the subject of that sentence, because it’s easy to lose. It’s your guess that goes up .58 of a ruler, not the shine itself. The real shine still wobbles around it quite a bit — how much, exactly, is in section 9.

And because they’re two measures of the same thing, you can convert one into the other:

\[b_1 = r \cdot \frac{S_Y}{S_X}\tag{3.4}\]

Read it in plain words and it’s almost boring: .58 of a shine ruler per carat ruler, times 21.74 shine per ruler, divided by 0.39 carats per ruler — and there’s 32.14 shine per carat again. (Check it with the rounded numbers above and you get 32.3 — that’s rounding, not a mistake; unrounded, out comes exactly our 32.14.)

One slope, two measures. Not two tools: the same line, with different marks along the axes.

One more thing, for when you open an article: there the slope-in-rulers is often called β, the standardised coefficient — and with one predictor, that’s simply our r.

And now the thing the slope can do and the rectangle can’t: choose a side. Turn the question round — not shine from carat, but carat from shine — and divide the rectangles by the squares of shine instead of those of carat: 54 divided by 5200 instead of 54 divided by 1.68. Out comes a different slope: 0.0104 carats per shine point.

And careful, because the cheese table wrong-foots you here: a price per kilo you turn round by taking one divided by it, but one divided by 32.14 is 0.031, and that is not it. The line the other way round is a different line — it makes different misses small, the horizontal ones instead of the vertical — and so a different rate. But the correlation stays exactly .58, whichever way you look: in the ruler-by-ruler rectangle it makes no difference which side you call the width. Association has no direction. A guess does — the moment you say this I know, that I want to know, you’ve chosen a side, and that choice lives in your question, not in the diamonds. In figure 3.10 the two lines stand side by side, both through Else.

The same scatterplot with names, carat against shine, the two dashed lines of the means and a burgundy ring around Else at their crossing. Two straight lines run through that ring: a green one, rising gently, shine from carat, and a grey dashed one, much steeper, carat from shine. At Kees, bottom right, both his misses are drawn in burgundy: straight up to the green line, and dashed, to the left, to the grey line. At carat 1.1 and shine 30, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2.

Figure 3.10. Two lines through the same cloud and through the same crossing: shine from carat (green, 32.14 shine per carat) and carat from shine (grey, 0.0104 carats per shine). At Kees you can see the two kinds of miss: vertical to the green line, horizontal to the grey. The correlation is .58 in both directions.

7. The whole line — and where it starts

One number is not yet a line. 32.14 is a direction — but there are infinitely many lines with that direction, all neatly parallel, into the cloud and out again. Which one is ours? Figure 3.11 is a strip of two small sketches, and picture 1 draws five of them.

Two small hand-drawn sketches side by side, numbered 1 and 2, each the same faint cloud. 1: five parallel green lines with the same slope, one above the other, and a question mark. 2: the two dashed lines of the means added, a burgundy ring where they cross, with Else in it; one line runs through the ring, drawn solid and dark green, and the other four are still there as pale dashed ghosts.

Figure 3.11. From direction to line. Picture 1: five lines with the same slope; the question mark is the question from the text: which one is ours? Picture 2: the two dashed lines of the means are added, and where they cross there is a ring, with Else inside it. Of the five lines, the one through that ring remains; the other four are still there as ghosts.

We already used the answer in section 6: the regression line always goes through the centroid — through the crossing of the two means. For us: straight through Else. Else again. And again she doesn’t notice a thing (picture 2).

And now, as promised: why does it always go through there? Look what would happen if it didn’t. Take a line whose misses aren’t in balance — add them up and something is left over. Say the twelve lie, on average, five shine points above it. Then shift that whole line five points up. The slope doesn’t change, every guess goes up by five, and so every miss goes down by five. What does that do to the sum of the squares? It drops — by twelve times five squared, 300. For nothing, just by shifting. Figure 3.12 puts the two side by side: on the left a line like that, lying too low; on the right, ours.

Two panels side by side, the same twelve stones without names, carat against shine; in both a burgundy ring around the dot at the crossing, and from each dot a burgundy stroke straight to the line, the miss. On the left a grey line with the right slope but five shine points too low: eight of the twelve strokes point upwards. On the right the green line through the ring: strokes up *and* down, in balance. In both panels, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2.

Figure 3.12. On the left the same slope, but five shine points too low: the misses point mainly upwards and add up to 60. On the right the line through the crossing: the misses add up to zero, and the squares together are 300 smaller.

So as long as something is left over, it can be done better, and then it wasn’t the least-squares line. With the right line that gain is used up: there the misses add up to zero. Just like the blue strokes around the mean in chapter 1 — some things just move house with you.

And with that the centroid is already there, just not in those words. If the misses cancel each other out, the average guess is exactly the mean shine. And the average guess is what the line says at the mean carat. Put those two side by side and it reads: at carat 1.6, the line says 50. That’s the crossing. That’s Else.

intercept where the line starts also: constant · notation: \(b_0\)

But ‘the line with direction 32.14 through Else’ is not a formula anyone else can do anything with. For that a line needs a starting point: where it meets the vertical axis, at carat zero. And now it turns out why our slope already had a number on its door in section 6 — \(b_1\): a second b is moving in, \(b_0\), the intercept. You find that starting point by simply walking back from the centroid:

\[b_0 = \overline{Y} - b_1\overline{X}\tag{3.5}\]

\[\begin{aligned} b_0 &= \overline{Y} - b_1\overline{X} \\[0.4em] &= 50 - 32.14 \times 1.6 \\[0.4em] &= −1.43 \end{aligned}\]

Together they make the whole line:

\[\hat{Y} = b_0 + b_1 X\tag{3.6}\]

\[\begin{aligned} \hat{Y} &= b_0 + b_1 X \\[0.4em] &= −1.43 + 32.14 \times X \end{aligned}\]

How to read this formula:

  • \(\hat{Y}\) — say ‘Y-hat’. The hat means: this is not what we saw, but what the line guesses. The predicted shine.
  • \(b_0\) — where the line starts (the shine guess at carat zero): −1.43.
  • \(b_1\) — how much the line rises per carat: 32.14 shine points.

Read it as: start at −1.43, and add 32.14 for every carat.

Does that intercept mean anything? Look at it again: it really says that. A diamond of zero carats would, according to this line, shine −1.43. Not a little shine — negative shine, on a scale that starts at zero. No such thing exists, and the line claims it anyway, with a straight face.

That’s because zero carats lies outside our diamonds. Our smallest is Anna, at 1.0 carat, and the line was drawn on what’s there, between 1.0 and 2.1. At carat zero it’s a long way from home, and there it makes a claim about a stone that isn’t there. Here, \(b_0\) is an address where nobody lives — not even an empty house, but a house number on a street that ends before you get there. To see that address at all, you have to open the window up as far as carat zero; figure 3.13 does that.

Scatterplot, carat on the horizontal axis starting at zero, shine on the vertical. The twelve stones stand faintly in a sand-coloured strip, the street where they live; there the line is drawn solid, to the left of it dashed, down to a burgundy dot on the vertical axis, just below the black line at shine zero, the floor. A burgundy circle on the line marks Else, where the line passes through. At the top left the same corner is magnified: there the line visibly dips below the floor. At carat 1.1 and shine 30, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2.

Figure 3.13. The window opened up to carat zero. The street where the stones live runs from 1.0 to 2.1; there the line is drawn solid, to its left dashed, down to the dot at −1.43 — below the floor of shine zero. That bit is so small that it’s shown again, magnified, at the top left.

That’s more often the case than not. An intercept may mean something, but it doesn’t have to, and whether it does depends on one question: does zero lie within your data, and does zero mean anything there? For height in centimetres or income in euros: no. For a difference score or a centred variable: yes. We come back to that in chapter 8, and there we do something about it.

A small scatterplot, carat horizontal, shine vertical; the twelve stones faint in the background, the line through them. A dotted arrow goes up from below at carat 1.8 to the line and then sideways to the vertical axis, where it says 56: the guess, an open green circle on the line. Just above it a dark dot, Hans, with his shine beside it, and a short burgundy stroke from the guess to Hans. At carat 1.1 and shine 30, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2.
Figure 3.14. Predicting is reading off: up at 1.8 to the line, across to 56. Hans hangs just above it.

And now the stone can have its number. The thirteenth, remember, with its 1.8 carats on the moss. In section 6 you walked to it from Else; now anyone can do it, even someone who has never met Else — and if you’d rather walk along the line with your finger than with a formula, you can do that in figure 3.14:

\[\hat{Y} = −1.43 + 32.14 \times 1.8 \approx 56\]

Call it 56. That the formula comes out the same as the walk from Else is not a roundabout check — it is what \(b_0\) is for: a line you can hand on to someone who doesn’t know our diamonds.

And Hans — also 1.8 carats — shines 60, 3.6 points above it; that’s how close, and how far, a guess like that leaves you.

One more thing, about the barrel. Our \(b_0\) and \(b_1\) were worked out on the chest — twelve diamonds. In the barrel (all the diamonds the dam could ever catch) there’s a line too, and as always the population counterparts get Greek letters: β0 and β1 (‘beta’). Ours are estimates of that β0 and β1 — statistics that estimate parameters, the Levi’s 501s from chapter 1.

And now watch, because this is the confusion chapter 1 announced. Earlier in this chapter the slope-in-rulers was also called β — the standardised coefficient you meet in articles. Here the same letter stands for something quite different: the slope in the barrel. Two meanings, one letter, eternal confusion. For us, for now, β is only this — the same b, but for the whole barrel.

8. DATA = FIT + RESIDUAL

In chapter 1 we played the lucky dip: draw a diamond blind, guess with the mean, and see how far off you were. Now we play it again — but this time you may look first. Not at the shine; at the carat. Your guess is no longer 50 for everyone; your guess is what the line says at that carat. First, the old game one more time, in figure 3.15: just the dashed line of the mean and the miss of each diamond.

Scatterplot, carat horizontal, shine vertical, each dot with its name. A dashed horizontal line gives the mean of shine, 50. From each stone a blue stroke runs straight to that line. Else and Leen lie exactly on it and have no stroke. Ida and Joop stand one above the other at the same carat, so that Ida's stroke falls inside Joop's. At carat 1.1 and shine 30, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2. Those two share one stroke as well.

Figure 3.15. The misses of chapter 1, one more time: only the dashed line of the mean and the total miss of each diamond.

With that, every shine falls into two pieces: the piece the line already knew, and the piece the diamond adds on its own.

\[Y_i = \hat{Y}_i + e_i \qquad \text{or} \qquad \text{DATA} = \text{FIT} + \text{RESIDUAL} \tag{3.7}\]

FIT is the prediction — the point on the line. RESIDUAL is the miss — the bit beside it, \(e_i = Y_i - \hat{Y}_i\). This is not a new formula but a new reading: every data point is a bit of model plus a bit of a mind of its own. This split is going to carry the rest of the book. Figure 3.16 cuts the misses of figure 3.15 open along the line, first for two stones.

The same cloud as in the previous figure, with names, the dashed line of the mean and now the regression line slanting through it too. Only Anna and Geer have strokes, three side by side: blue from the mean to the dot, green from the mean to the line, burgundy (the picture's own labels say red) from the line to the dot. At Anna, below the line, green and burgundy both point down; at Geer, above it, green is a stub and burgundy almost as long as blue. Their numbers are written in two lines of text, top left for Geer and bottom middle for Anna. At carat 1.1 and shine 30, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2.

Figure 3.16. The sequel to figure 3.15: the same misses, now cut open along the regression line — shown for Anna and Geer.

For Kees: the line says 66, he shines 30.

\[\underbrace{30}_{\text{DATA}} = \underbrace{66}_{\text{FIT}} + \underbrace{(−36)}_{\text{RESIDUAL}}\]

And for all twelve:

Table 3.4. DATA = FIT + RESIDUAL for all twelve diamonds.
name carat shine (DATA) on the line (FIT) off the line (RESIDUAL)
Anna 1.0 20 30.7 −10.7
Bram 1.1 30 33.9 −3.9
Cees 1.1 30 33.9 −3.9
Dora 1.3 30 40.4 −10.4
Else 1.6 50 50.0 0
Faas 1.6 70 50.0 +20.0
Geer 1.7 80 53.2 +26.8
Hans 1.8 60 56.4 +3.6
Ida 1.9 70 59.6 +10.4
Joop 1.9 80 59.6 +20.4
Kees 2.1 30 66.1 −36.1
Leen 2.1 50 66.1 −16.1

Get your calculator out and check two of them yourself — Anna and Geer. Anna: −1.43 + 32.14 × 1.0 = 30.7, and 30.7 + (−10.7) = 20. Does her row add up? And Geer’s? Every row in this table has to add up: FIT + RESIDUAL = DATA, no exceptions.

Three old acquaintances suddenly look different. Else has no residual now either — no square in chapter 1, no rectangle in section 4, and the line runs straight through her; she differs from nothing, so there is no line for her to differ from. Faas sits directly above her, so for him the line simply guesses the mean, and every bit of his distance is his own. And Leen had no square in chapter 1, but the line expects more of a big diamond: now he suddenly has a residual of −16 — being average stops being enough the moment someone knows your carat.

And Kees. Against the bare mean he was −20; against the line, −36. The line makes most misses smaller — compare the RESIDUAL column with the misses from chapter 1 — but Kees’s, and Leen’s as we just saw, it makes bigger. That’s no slip of the line; that’s information. The better the model, the more sharply the exception comes into view. Keep him in mind — still.

Oh, and: add up the RESIDUAL column. Zero (give or take rounding). Around the line the residuals keep each other in balance exactly as they did around the mean in chapter 1. Some things just move house with you.

Now the strokes themselves. Every blue total stroke (from the point to \(\overline{Y}\)) splits into a green fit stroke (from the line to \(\overline{Y}\) — the stretch the line bridges) and a burgundy residual stroke (from the point to the line — the bit that’s left over):

\[\underbrace{(Y_i - \overline{Y})}_{\text{blue: total}} = \underbrace{(\hat{Y}_i - \overline{Y})}_{\text{green: fit}} + \underbrace{(Y_i - \hat{Y}_i)}_{\text{burgundy: residual}}\]

Check it with Kees: blue −20 = green +16 + burgundy −36. With him the two pieces fight — the line pushes him up (it’s big, so it must be shiny) and reality pulls him hard back down. And check Geer yourself: blue +30 = green 3.2 + burgundy 26.8. With Geer the line does almost nothing; his shine is nearly all his own. Figure 3.17 does it for all twelve at once; Kees and Geer have theirs written out.

Scatterplot, carat horizontal, shine vertical, with names, the dashed line of the mean and the regression line. At each stone three thin strokes close together, always in this order from left to right: blue from the mean to the dot, green from the mean to the line, burgundy from the line to the dot. Stones that share a carat have their trio to the left and to the right of the dot, so that they don't cover each other; Else, on both lines, has none. Leen, exactly on the mean, has no blue stroke: only green going up and burgundy coming back. At the top Geer's sum is written out, at the bottom Kees's. At carat 1.1 and shine 30, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2. Those two have one trio between them.

Figure 3.17. DATA = FIT + RESIDUAL for all twelve: at each stone three thin strokes side by side — blue the total, green what the line bridges, burgundy what’s left over. Green and burgundy together end exactly where blue ends; at Kees the pieces fight, at Geer the line does almost nothing, and Else has none.

And those burgundy bits are the ones whose squares the line made as small as possible in section 6.

9. The garden in two

And now the move you know from chapter 1: square and add up.

First, one honest warning: per diamond the strokes add up, the squares don’t — for Kees, blue² = 400, but green² + burgundy² = 259 + 1303 (check it with his row from table 3.4: 16.1² and 36.1²). So for a single diamond the square-bookkeeping does not work. But add over all twelve at once, and suddenly it does, exactly:

\[SS_{\text{total}} = SS_{\text{regression}} + SS_{\text{residual}} \tag{3.8}\]

\[5200 = 1736 + 3464\]

\(SS_{\text{total}}\) you know already, by the way: it’s simply \(SS_Y\) from chapter 1, the 5200 — all the blue squares together. That garden is now divided in two (figure 3.18): a green part that lies on the line, and a burgundy part that falls beside it. That this works out exactly is no lucky rounding: the least-squares line is chosen precisely so that fit and residual have nothing left to tell each other — what the line already says isn’t in the misses a second time.

A rectangle on squared paper with a blue outline: the whole shine garden of 5200, 52 cells of ten by ten. A dashed line divides it into a green part on the left, 1736, on the line, and a burgundy part on the right, 3464, off the line. Underneath: 5200 = 1736 + 3464, and R² = 1736 / 5200 = .33, a third of the garden.

Figure 3.18. The whole shine garden of 5200 — all the blue squares from chapter 1 put together, 52 cells of ten by ten — in two: green 1736 on the line, burgundy 3464 off the line. A third of the garden stands in neat rows.

What the line costs — \(S_e\)

standard error of the estimate the residual ruler notation: \(S_e\) also: residual standard error

In chapter 1 the SD was your ruler: how far off a diamond is, on average, if you guess with the mean — 21.74 shine points. Now that we guess with the line, we want the same ruler again, but this time around the line: how far off is a diamond, on average, if you know its carat?

Same recipe as ever: take the squares of the misses (that’s \(SS_{\text{residual}}\), the burgundy 3464), divide by the number of truly free misses, and take the square root. Only — how many misses are still free? In chapter 1 the mean cost us one envelope — of the three, the third was fixed as soon as you knew the mean: \(n-1\). The line is more expensive: it takes two numbers to pin it down, \(b_0\) and \(b_1\). Two envelopes fixed, so \(n-2\) free misses.

\[S_e = \sqrt{\frac{SS_{\text{residual}}}{n-2}}\tag{3.9}\]

\[\begin{aligned} S_e &= \sqrt{\frac{SS_{\text{residual}}}{n-2}} \\[0.4em] &= \sqrt{\frac{3464}{10}} \\[0.4em] &= 18.6 \end{aligned}\]

This is the standard error of the estimate — but don’t let the name fool you: it’s simply a standard deviation, of the residuals. The SD was your ruler around the mean; \(S_e\) is exactly that same ruler, moved over to the line.

And look what’s happened: 21.74 has become 18.6. The ruler is shorter — that’s what knowing the carat buys you. Not that much shorter, mind, and that too is honest information: carat knows something about shine, but nowhere near everything. For the stone on the moss it means that with your guess of 56 — the shine the line gave it in section 7 — you’ll typically be about 18.6 off, sometimes more, sometimes less.

Figure 3.19 puts the two rulers one above the other: the band around the mean, and the same band moved over to the line.

Two panels one above the other with the same twelve stones, carat horizontal, shine vertical, without names. At the top a blue horizontal line at the mean of shine, with a light blue band of 21.74 above and below it; three stones fall outside it. At the bottom the green regression line, slanting upwards, with a light green band of 18.6 running along with it; four stones fall outside it. The lower band is a little narrower than the upper one. In both panels, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2.

Figure 3.19. The same cloud twice. Top: guessing with the mean, a band of ±21.74 around it. Bottom: guessing with the line, a band of ±18.6 slanting along with it. The band has got narrower — but not much narrower, and that is honest information.

What the line buys you — \(R^2\)

Divide the green part of the garden by the whole garden:

\[R^2 = \frac{SS_{\text{regression}}}{SS_{\text{total}}}\tag{3.10}\]

\[\begin{aligned} R^2 &= \frac{SS_{\text{regression}}}{SS_{\text{total}}} \\[0.4em] &= \frac{1736}{5200} \\[0.4em] &= .33 \end{aligned}\]

explained variation variance accounted for (VAF) notation: \(R^2\) also: explained variance, coefficient of determination, \(r^2\)

This is called the explained variation, or variance accounted for — VAF: the share of all the variation in shine that the line accounts for. Chapter 1 already promised it — explained variation is the part of the garden where neat rows of flowers stand. Here is the row, and it is the line. A third of the shine garden stands in neat rows; two thirds grows wild, burgundy, beside it.

You’ll come across explained variance just as often as explained variation, and it’s just as good — divide the green part and the whole garden by the same \(n - 1\), and their ratio stays exactly the same:

\[\frac{SS_{\text{regression}}}{SS_{\text{total}}} = \frac{SS_{\text{regression}} / (n - 1)}{SS_{\text{total}} / (n - 1)}\]

And then something lovely. Square the correlation — but take the unrounded one, .578: \(.578^2 = .33\). Exactly the same number. With one predictor, \(R^2\) is simply \(r^2\): the correlation — the slope in rulers — squared, is the share of the garden. And \(R^2\) is an effect size too — the same question as with \(r\), how big?, now answered in garden language.

10. The thirteenth stone

There’s one thing we’ve done this whole chapter without saying it out loud: we laid a straight line through the cloud. That’s not a neutral act but an assumption. The line adds — start at −1.43, add 32 per carat, at the bottom of the cloud just as much as at the top. The same amount added each time: a method like that is called additive, and it’s a lovely word. Figure 3.20 shows it as a staircase on the line: every step the same height.

Scatterplot, carat horizontal, shine vertical, the twelve stones faint in the background, the line through them in light green and its formula at the top left. On the line stands a staircase of four steps, starting on the line at the carat of the smallest stone, Anna, 1.0, each step a quarter of a carat: a black horizontal segment and then a burgundy upright one up to the line, with the rise, 8.0, beside it. All four upright segments are the same height, at the bottom of the cloud just as much as at the top. At carat 1.1 and shine 30, two stones lie exactly on top of each other, Bram and Cees; that is why the dot carries ×2.

Figure 3.20. The same amount added each time: every quarter of a carat further is 8.0 more shine points, at the bottom of the cloud just as much as at the top. A straight line is just adding up.

The assumption underneath goes by a more respectable name: linearity. The cloud itself may be curved; our line can’t be. That’s why the first check is always the dumbest and the best: look at the cloud. For our twelve, straight looks quite reasonable. Figure 3.21 shows what it looks like when it isn’t straight — deliberately without diamonds.

A made-up scatterplot, hours of practice horizontal and skill vertical, with no numbers on the axes. The points follow a curve that rises steeply and then flattens out; a straight green line runs through them. From each point a burgundy stroke runs to the line: at the left the points lie under the line, in the middle above it, at the right under it again, and those three stretches are labelled. Underneath: the misses lie below, above, below; the line is lying to you, neatly and systematically, and not one number warns you — so look at the cloud first.

Figure 3.21. A straight line on a curved relationship — made up, because our own cloud looks straight. The misses lie below, above, below: the line is lying to you, neatly and systematically, and not one number warns you.

Three more assumptions go with lines like this, all three about the residuals: that they’re independent of one another, that their spread is the same size all along the line (equal spread — the posh word is homoscedasticity; learn to say it out loud now), and that they’re normally distributed. For now, just the names: how you read those three off the residuals — which plot you ask for, and how you can tell it’s going wrong — is a job for the R exercise book, in the part on simple regression. What to hold on to here is the distinction: the line itself demands only straightness; the rest is about how the misses behave — and the misses only really matter once we start asking more of the line than description.

Now for the sobering part.

You’ve given the stone its number: 56, typically about 18.6 off, and a third of the garden explained.

And honestly — that one thing, giving a single stone a number, is something a psychologist, or any researcher, almost never does. When did you last have to predict one person’s shine? In research we mostly want to understand the mostly, not guess one case.

But look at what you’ve built along the way, because that you do all the time: the rectangle in rulers, the rate, and the split into what the line already knew and what the stone brought in itself. Those are the hand tools of this book’s fourth movement. There, the line can do something the correlation, in principle, can’t: hold a second variable still while you look at the first. Calculate it away. Correct for it.

And one more thing, for completeness. \(r\), \(b_1\) and the line are not three tools — they’re one slope, two measures. So if you wonder whether this mostly is real, or just a fluke of these twelve, you’re asking one question, not three. How you answer that question — we’ll come back to that later.

11. Work

Doing the sums with software belongs in the R exercise book: the book shows, the exercise book trains. There you do covariance, correlation and the regression line with your own data.

It shows, then. Everything you’ve done by hand in this chapter — the squares as small as they would go, \(b_1\), \(b_0\), the strokes at Kees — you ask for in one sentence:

lm(glans ~ karaat, data = diamantjes)

Shine (glans), explained by carat (karaat), in our diamonds (diamantjes). That sentence doesn’t spare you the sums; it packs them away: everything above is inside it, and you’re now the only one who knows what.

12. Closing

Then the crow took one on the quiet, and flew away.

The others watched him go. Nobody said anything.

 

Eleven diamonds. Tomorrow, perhaps, ten. Our cloud was never the whole world — it was whatever happened to get caught in a beaver’s dam. Would the pattern still be there with a different twelve? With eleven? How sure are we, really, of that mostly?

That’s the next chapter. The lucky-dip barrel is ready and waiting.