First movement · Difference under the microscope

Chapter 2 — The Shape of the Heap

Median, quartiles, boxplot and skewness

1. The distribution

“We will have to share them out, of course,” said the crow one afternoon, as if it had only just occurred to him. He said it too quickly for someone it had only just occurred to.

“Share them out,” the beaver repeated. He thought about it. “Twelve diamonds, four animals. Three each.”

“Three each,” said the crow, “but not just any three. I want the bright ones.”

“That’s not sharing out,” said the gull. “That’s picking out.”

They decided to do it fairly. The beaver laid the diamonds out in a row, from dull to bright — that took a while, because two of the diamonds disagreed about which of them was the duller, and the beaver had to lift them both three times.

Then there lay the row. From the dullest to the brightest, and everything in between.

“Look,” said the beaver, not without pride. “Only now can you see how they’re spread out. Here’s the middle. And if each of us takes a quarter of the row —” he pointed with his tail at the places where you chop the row into four, “— then nobody has anything to complain about. Except whoever gets the dull quarter.”

“I,” said the crow, “certainly won’t be.”

The elephant had not said anything yet. He looked at the row, for a very long time, the way only elephants look.

“If you share them out,” he said at last, “the row is gone.”

Nobody said anything back. The row lay there, from dull to bright, and had suddenly become something in its own right — something none of the four animals could have on their own.

They left the diamonds where they were. Together.

 

The beaver has, without meaning to, explained the word this chapter is about. A distribution is exactly that: how the whole lot is spread out once you lay it in a row. Where it heaps up, where it thins out, where the middle is, where the ends are. And the elephant supplied the warning straight away: the row itself is information. Keep only the mean, and you throw away the row.

2. The row — sorting, the middle, the quarters

First, sort

Here are the twelve once more, in table 2.1, so you don’t have to leaf back to chapter 1. In this chapter we look only at the shine column.

Table 2.1. The diamonds from the chest, with their carat (the unit of weight for gemstones, 1 ct = 0.2 g) and shine (shine meter, scale 0–100).
id name carat shine (0–100)
1 Anna 1.0 20
2 Bram 1.1 30
3 Cees 1.1 30
4 Dora 1.3 30
5 Else 1.6 50
6 Faas 1.6 70
7 Geer 1.7 80
8 Hans 1.8 60
9 Ida 1.9 70
10 Joop 1.9 80
11 Kees 2.1 30
12 Leen 2.1 50

In table 2.1 they’re listed by name. Now lay them out by shine, from low to high — figure 2.1:

The twelve diamonds side by side on one line, sorted by shine from dull on the left to bright on the right, rank 1 to 12. Each is a blue diamond shape, bigger for a heavier stone, with its name above it and its shine below: Anna 20, Bram 30, Cees 30, Dora 30, Kees 30, Else 50, Leen 50, Hans 60, Faas 70, Ida 70, Geer 80 and Joop 80. A dashed red line between rank 6 and rank 7 marks the middle, Med = 50; dotted lines between rank 3 and 4 and between rank 9 and 10 mark Q1 = 30 and Q3 = 70.

Figure 2.1. The shine of all twelve, sorted from dull to bright; the dashed line marks the middle, the dotted lines the quarters.

\[20,\ 30,\ 30,\ 30,\ 30,\ 50,\ 50,\ 60,\ 70,\ 70,\ 80,\ 80\]

That’s all sorting is. But look what it gives you: suddenly you can see shape. At the low end, a huddle of thirties; at the high end, a short run of seventies and eighties; and a gap in the middle where hardly anyone lives.

The middle — the median

median the middle also: Med, Mdn (in APA style), the value at the middle rank

With ranks there are always two questions about the middle diamond: where is it, and what is it? Its place is always rank \((n+1)/2\). For us: \(13/2 = 6.5\) — between rank 6 and rank 7. What lies there: 50 and 50. So the median is 50. (With an odd number you get a whole rank number, and the median sits exactly on a rank — then it is simply that value.)

The median chops your row into two equal pieces: six diamonds on the left, six on the right.

The quarters — quartiles

And now the beaver’s discovery: chop those halves in half again. \(Q_1\) is simply the median of the left half — six diamonds, midway between rank 3 and 4: 30 and 30, so \(Q_1\) = 30. \(Q_3\), the median of the right half: between rank 9 and 10, 70 and 70, so \(Q_3\) = 70. There are formulas for it, but I sometimes just do it on my fingers, you know — how many are there on the left? And then I count.

Together with the minimum (20) and the maximum (80), you now have five numbers — the five-number summary:

\[20 \quad 30 \quad 50 \quad 70 \quad 80\]

Between any two neighbours in that list lies exactly a quarter of the row — three ranks, in our case: the share-out the crow was after. A quarter in ranks, mind you, not necessarily in values: where scores are equal there’s a knot in the row, and in ours a knot like that lies right across \(Q_1\) — four stones on 30.

The boxplot — a drawing of the row

A horizontal boxplot of shine on a scale from 0 to 100. The box runs from Q1 = 30 to Q3 = 70, with a thick line inside it at the median, 50. The whiskers run out from the box to 20, the dullest diamond, and 80, the brightest, each ending in a short crossbar. The five numbers are written above: min 20, Q1 30, Med 50, Q3 70, max 80.

Figure 2.2. The row of figure 2.1 summed up in five numbers: the box runs from quartile to quartile, the line sits on the median, and the whiskers reach to the dullest and the brightest diamond.

A boxplot is nothing more than those five numbers as a drawing — figure 2.2 above. The box runs from \(Q_1\) to \(Q_3\) — that’s where the middle half lives. The line in the box is the median. And the lines reaching outwards are called the whiskers; they run to the dullest and the brightest diamond.

And then the reading — because that’s what it’s for. The width of each piece tells you where the diamonds are alike and where they spread apart. A narrow piece = a quarter of the cases on a small stretch of the scale: homogeneous, a huddle. A wide piece = that same quarter spread thin: heterogeneous. Read our own boxplot in figure 2.2: the whiskers are short, ten shine points each — at the low end the huddle around thirty, at the high end the short run of seventies and eighties. That’s where the diamonds are alike. The two halves of the box are each twice as wide: also three diamonds per piece, but spread thin across the gap in the middle.

Outliers and the one-and-a-half rule. How long may a whisker be? By agreement, at most \(1.5 \times\) the width of the box (the interquartile range, \(\mathit{IQR}\) = \(Q_3\) − \(Q_1\)). For us: \(\mathit{IQR}\) = 40, so whiskers may reach 60 below \(Q_1\) and 60 above \(Q_3\) — from −30 to 130. All twelve shine scores fall well inside that: no outliers, so no asterisks — the mark a boxplot gives an outlier.

3. The heap — stacking and shape

There’s a second way to give the row a shape: not laying it out but stacking it up. Put all the thirties on one stack, all the fifties on the next, and so on. Then you get a series of stacks — a histogram, but the picture to hold on to is this: the heap. Figure 2.3 stacks our twelve.

The twelve diamonds as blocks with their names on them, stacked on a shine scale from 0 to 100: Anna at 20; Bram, Cees, Dora and Kees at 30; Else and Leen at 50; Hans at 60; Faas and Ida at 70; Geer and Joop at 80. Nobody sits at 40. The tallest stack, four blocks high, stands at 30.

Figure 2.3. The same twelve, now stacked by shine score — each block carries a name, and the tallest stack stands at 30.

mode modal value also: the most frequent value

The tallest stack has a name of its own: the mode — the value that occurs most often. It’s what everyone’s wearing — mode is an old word for fashion. For us: 30, four diamonds high.

So now there are suddenly three ‘middles’ on the table. The mode: 30. The median: 50. The mean: 50 as well. Three summaries of the same twelve, and they don’t have to agree with each other — that’s not a mistake, that’s information about shape. Look where they live. The mode lives with the huddle at the low end, right in the thick of it. The median and the mean both live at 50 — out in the quiet stretch between the huddle and the run, where hardly anyone lives. Keep only the mean, and you throw away the row; here you can see what you throw away.

And heaps have shapes. Some are neatly symmetric: the same silhouette to the left and to the right of the middle. Some are skewed: most of it huddled at one end, and a long, thin tail off towards the other. The tail gives the name: tail to the right = right-skewed. Figure 2.4 puts the three shapes one above the other.

Three heaps of twenty blocks each on a score scale from 1 to 9, one above the other; in every heap a black line marks the median and a green lozenge the mean. Top, symmetric: the tallest stack stands at 5, and the line and the lozenge coincide at 5.0. Middle, right-skewed: the blocks pile up at the low end and a thin tail, marked with an arrow, runs out to the right; the median is 3.0 and the mean 3.8, to the right of it. Bottom, left-skewed: the mirror image, piled up at the high end with the tail running to the left; the median is 7.0 and the mean 6.2, to the left of it.

Figure 2.4. Three heaps of twenty blocks. At the top, symmetric: the same silhouette left and right. Below it, two skewed heaps; the arrow points out the tail, the side where the heap thins out. Notice the line and the lozenge in each heap now: in the skewed heaps they stand apart — the reason comes in the next section.

4. What the mean won’t tell you — the King

Suppose the King moves into your street.

Eight houses on King’s Lane. Seven of them are ordinary houses — four, five, six hundred thousand euros, roughly what a house costs. And then him, with something you need a driveway for: four and a half million.

From that day on, the average house price in your street is a million. You’ve done nothing. You live in the same house, with the same crooked gutter and the same neighbour who won’t cut his hedge. And yet you now live, officially, on a millionaires’ street. Put it on LinkedIn.

Now look at the median. The middle house is still the middle house: half a million. The median couldn’t care less. Whether a terraced house stands above it or a palace — it shrugs. Figure 2.5 shows the street twice, one above the other: without him and with him.

The same street drawn twice, one above the other, on a price axis in hundreds of thousands of euros. Top, before he moves in: seven ordinary houses as dots between 3 and 8; the median line and the green lozenge of the mean stand together at 5. Bottom, with him: the same seven plus one dot far to the right at 45, labelled 'him'. The median line has not moved from 5; the lozenge has moved to 10, with a pale ghost at its old place and an arrow from there to here.

Figure 2.5. The same street, twice. At the top, the seven ordinary houses: line and lozenge coincide. Below, his house joins them, far right. The lozenge (the mean) moves from half a million to a million; the line (the median) stays put.

That’s the word you learn here: the median is robust. The mean is sensitive — every score pulls on it, and an extreme score pulls extremely hard. And that’s no design fault. It’s exactly the property that made the mean the best guess in chapter 1: it’s the balance point where all the misses cancel out. Sit a giant on one end and he drags the whole balance point his way — the seesaw of figure 1.3, with a palace on one end. That’s the price of being the balance point.

Let me show you, just once, on King’s Lane. I’ll count in hundreds of thousands, so 3 means three hundred thousand euros. Sorted: 3, 4, 4, 5, 5, 6, 8, 45.

\(Q_1\) is the median of the four on the left: between 4 and 4, so 4. \(Q_3\) is the median of the four on the right: between 6 and 8, so 7. So the box is 3 wide — that’s your \(\mathit{IQR}\).

One and a half times 3 is 4.5. I always start on the side where it’s going to get weird: 7 + 4.5 = 11.5. That 11.5 is the fence: anything above it is an outlier. Forty-five. Asterisk. At the bottom: 4 − 4.5 = −0.5, and no house costs less than nothing, so nothing happens there.

And now watch, because this is where people come unstuck.

The fence is not a house. It’s an agreement, a line we draw somewhere. Nothing has to stand there, and usually nothing does. So the whisker does not run all the way to the fence — the whisker stops at the last real house before it. On King’s Lane that’s the house worth 8: the fence is at 11.5, but the whisker stops at 8, because further to the right nobody lives any more. Except him, and for the whisker he doesn’t count. Have a look at figure 2.6: the red dotted line is the fence, and no house stands there.

King's Lane drawn as a boxplot, with its eight houses as dots underneath, on a price axis in hundreds of thousands of euros. The box runs from Q1 = 4 to Q3 = 7, with the median line at 5. A red dotted line at 11.5 is labelled 'the fence'; no house stands there. The right whisker stops at 8, the last real house before the fence; the left one runs to 3. Far to the right, at 45, a red asterisk marked 'outlier'. Above the plot: the box from 4 to 7 is 3 wide; one and a half times 3 is 4.5; 7 + 4.5 = 11.5.

Figure 2.6. King’s Lane at the moment of the ritual. The box runs from 4 to 7, the fence at 11.5 is a red dotted line — an agreement, not a house — and the whisker stops at 8, the last real house before it. The dots under the box are the eight houses themselves; the asterisk at 45 is him.

And sometimes — look at Park Villas in figure 2.7, below — that last real house is exactly \(Q_1\) itself. Then the box has no whisker at all on that side.

Park Villas drawn as a boxplot, with its eight houses as dots underneath, on a price axis in hundreds of thousands of euros. The box runs from Q1 = 20 to Q3 = 28, with the median line at 24; the right whisker runs to 30. On the left there is no whisker: the lowest real house is 20, which is Q1 itself, and the text beside the box says so. A red dotted line at 8 is labelled 'the fence', and far to the left, at 2, a red asterisk marked 'outlier'. Above the plot: the box from 20 to 28 is 8 wide; one and a half times 8 is 12; 20 − 12 = 8.

Figure 2.7. Park Villas. The shed worth 2 is the outlier, the fence is at 8, and to the left of the box there is nothing to draw: the lowest real house is \(Q_1\), from 20 to 20. On the right the whisker runs as usual, to the house worth 30.

I know that look. Count along with me, because this is the moment people think something is broken. Park Villas: 2, 20, 20, 24, 24, 28, 28, 30. The 2 is the outlier — a shed among the villas, or someone who sold their house to their daughter. But the 2 counts in every calculation as normal: in \(Q_1\), in the \(\mathit{IQR}\), in the fence itself. Only in the drawing does it stand apart, as an asterisk. The boxplot disowns nobody; it marks them. So for the whisker there are seven houses left, and the cheapest of them costs 20.

Now \(Q_1\). The lower half is 2, 20, 20 and 24; its middle lies between the two 20s. So \(Q_1\) is 20 too.

There they are, in figure 2.7 above: the lowest real house and the first quartile, in exactly the same place. The whisker has to run from one to the other — from 20 to 20. There’s no room in between, so there’s nothing to draw. Not broken, not forgotten: quite simply, nobody lives between those two points.

That’s the honesty of a boxplot. It draws what is there, and when there’s nothing, it draws nothing. You won’t see many of these — but if you ever come across one, you now know you’ve done nothing wrong.

And with that — a mean that gets dragged along and a median that stays put — you have a free tell for shape: if the mean and the median lie almost on top of each other (as with our diamonds: both 50), the heap is in balance. If the mean lies clearly above the median, there’s a tail on the right pulling at it — right-skewed. Below it: left-skewed. Though this tell only sniffs out skew: ‘in balance’ doesn’t by itself mean ‘one neat hill’ — our own diamonds balance neatly at 50, with a gap in the middle. Two peaks, gaps, huddles: for those you have to keep drawing.

Four horizontal boxplots one above the other, each on its own price scale in hundreds of thousands of euros; in each, the value of the median stands above the box and the mean is a green lozenge with its value below it (only the top panel spells out the words 'median' and 'mean'). The Terrace: median 5, mean 5, the lozenge right on the line. Village Road: median 4.5, mean 5.5, the lozenge just to the right of the line, and no outlier. King's Lane: median 5, mean 10, the lozenge outside the box, between the end of the whisker and the dotted fence; a red asterisk far right at 45. Park Villas: median 24, mean 22, the lozenge to the left of the line; no whisker on the left, a dotted fence at 8 and a red asterisk at 2.

Figure 2.8. Four boxplots one above the other, with the median as a line and the mean as a green lozenge; King’s Lane and Park Villas carry the asterisk of their outlier.

Read the four streets of figure 2.8 from top to bottom and watch it happen: first the lozenge and the line lie on top of each other, then the lozenge creeps away, then it stands at twice the median — and in Park Villas it has slipped to the other side. Compare two numbers, and you can sniff out the shape before you’ve seen the picture.

The rule this chapter leaves behind: draw the heap first, then choose your summary. For neat, symmetric heaps, the mean and the SD tell the story. For skewed heaps with tails and giants, the median and the quartiles tell it more honestly. Summarise blind, and you summarise the wrong thing.

5. Work

Doing the sums with software belongs in the R exercise book: the book shows, the exercise book trains. There you sort, find the median and the quartiles, and draw the boxplot and the histogram with your own data.

6. Closing

The animals left the diamonds together, and we now know what the row gives us: a middle that doesn’t flinch at anything, quarters that tell you where things huddle, and a heap with a shape. We now have two languages for summarising — mean-and-spread, median-and-quartiles — and a tell to choose between them.

But one thing has been lying there quietly all along. We’ve combed through the shine of twelve diamonds — the middle, the spread, the shape of the heap. And all that time, since chapter 1, there was a second number next to every shine: the carat. We haven’t looked at it once.

Two rows of numbers, about the same twelve stones. Does a heavier stone also shine brighter? Or does one have nothing to do with the other?

That’s what the next chapter is about. And it starts with something you can already do: looking at how far one stone lies from the mean. Only now you do it for both rows at once, each with its own mean.