Book Review How Not to Be Wrong by Jordan Ellenberg

A while back (11/2/11) I reviewed the book Stats.con by James Penston. That book discussed how the statistics used in randomized clinical trials can be highly deceptive. How Not to be Wrong also covers some aspects of statistical misuse, in more detail, and certainly in a much more entertaining way. Some of his comments are funny as hell. 

Jordan Ellenberg

Consider the widespread use of a statistic call the p value, which estimates the probability that the result of a study could have just been a chance coincidence rather than an actual meaningful finding. A study is generally considered positive if the p value is 5% or less.

5% is of course not 0%. There is a one in twenty probability that the study results that are deemed positive were in fact negative. But what happens if journals only publish the positive studies and not the negative ones, when there might be a large number of negative studies, and when the positive study results are not reproduced (replicated) in a second study? Well, people start believing things that are not true, that's what.

The reason for this is because, as the author points out, improbable things actually happen quite frequently. Especially if you do lots and lots of things – like experiments. 

Another issue he mentions is that, if your sample size is too small, the chances increase dramatically that one of your subjects will be an outlier that dramatically but artificially changes the average for whatever characteristic you are measuring. With a small sample, you are more likely to get a few extra prodigies or slackers in a study of people's ability to perform certain tasks. A famous example: if Bill Gates walks into a bar with a few other people, the average guy in the room is a billionaire.  

Here’s how the author starts out a discussion of the p value problem (pages 145-46) :

"Imagine yourself a haruspex; that is, your profession is to make predictions about future events by sacrificing sheep and then examining the features of their entrails...You do not, of course, consider your predictions to be reliable merely because you follow the practices commanded by the Etruscan deities. That would be ridiculous. You require evidence. And so you and your colleagues submit all your work to the peer-reviewed International Journal of Haruspicy, which demands without exception that all published results clear the bar of statistical significance.    

      

Haruspicy, especially rigorous evidence-based haruspicy, is not an easy gig. For one thing, you spend a lot of your time spattered with blood and bile. For another, a lot of your experiments don't work. You try to use sheep guts to predict the price of Apple stock, and you fail; you try to model Democratic vote share among Hispanics, and you fail…The gods are very picky and it's not always clear precisely which arrangement of the internal organs and which precise incantations will reliably unlock the future. Sometimes different haruspices run the same experiment and it works for one but not the other — who knows why? It's frustrating…

      

But it's all worth it for those moments of discovery, where everything works, and you find that the texture and protrusions of the liver really do predict the severity of the following year's flu season, and, with a silent thank-you to the gods, you publish

      

You might find this happens about one time in twenty.

      

That's what I'd expect, anyway. Because I, unlike you, don't believe in haruspicy. I think the sheep's guts don't know anything about the flu data, and when they match up it's just luck. In other words, in every matter concerning divination from entrails, I'm a proponent of the null hypothesis [that there is no connection between the sheep entrails and the future]. So in my world, it's pretty unlikely that any given haruspectic experiment will succeed.

      

How unlikely? The standard threshold for statistical significance, and thus for publication in IJoH, is fixed by convention to be a p-value of .05, or 1 in 20... If the null hypothesis is always true — that is, if haruspicy is undiluted hocus-pocus —then only one in twenty experiments will be publishable.

      

And yet there are hundreds of haruspices, and thousands of ripped-open sheep, and even one in twenty divinations provides plenty of material to fill each issue of the journal with novel results, demonstrating the efficacy of the methods and the wisdom of the gods. A protocol that worked in one case and gets published usually fails when another harupex tries it, but experiments without statistically significant results do not get published, so no one ever finds out about the failure to replicate. And even if word starts getting around, there are always small differences the experts can point to that explain why the follow-up study didn't succeed."

The book covers many subjects about which the non-mathematically-inclined can learn to think in a mathematical way in order to avoid coming to certain wrong conclusions and to zero in on correct ones. Many of these, however, are irrelevant to this blog – the chapters on lotteries come to mind. I of course found those parts a bit less interesting. But the chapters relevant to medical studies are so right on.

Another important topic the author covers is known mathematically as regression to the mean. This phenomenon can lead, as examples, to overestimates about the genetic component of human traits and explains why fad diets always seem to work at first but then later on everyone seems to forget about them. As mentioned, when you average any measurement applied to human beings, the averages can be deceptive.  

In addition to the sample size considerations described above, you can get into trouble if you start with a sample that contains people who are higher or larger on average on the relevant variable that the average person in the general population.

If two tall people marry, their progeny will usually be, on average, tall compared to others in the general population. However, they are not all that likely to be taller than their parents. As Ellenberg states, “…the children of a great composer, or scientist, or political leader, often excel in the same field, but seldom so much as their illustrious parents” (p. 301). Their heredity mingles with chance environmental considerations, and pushes them back toward the population average. That is the meaning of regression to the mean.

To understand this, think about those who embark on weight loss diets. One needs to consider the fact that most people’s weight tends to fluctuate a few pounds either way depending on a lot of chance factors, such as their happening by an ice cream truck. And when are people most likely to start a diet? When their weight is at the top of their range! So by the law of averages, they are probably in many instances going to lose weight whether they diet or not. But when they do diet, guess what happens? They attribute the loss to the fantastic new diet!

I can not say for certain, but I wonder if studies on borderline personality disorders (BPD) yield misleading results because of regression to the mean. Long term follow-up studies on patients with the disorder seem to indicate that it seems to go away after a few years in a significant percentage of subjects. This finding is misleading, however, when you look closer. 

To make the BPD diagnosis, the subject needs to exhibit 5 of the 9 possible criteria. Many of the "improved" subjects merely went from 5 criteria down to 4 of them, and were therefore not diagnosed with BPD any longer. Actually, they became just what we call "subthreshold" for the disorder. Their problematic relationships, however, were still pretty much the same.

These results could mean that subjects with BPD may naturally vacillate between meeting criteria for the disorder and being subthreshold, or between exhibiting a high number of the criteria and a lower one. Which would mean that if they qualified for the diagnosis at the beginning of the long term follow-up study, a significant proportion of the long-term study subjects were at their worst. If so, the study results may indicate regression to the mean, and therefore say nothing else significant about the long term prognosis for the disorder.

Other important statistical issues the author discusses clearly and brilliantly include assumptions that two variables are related in a linear fashion when the are not (non-linearity - cause and effect relationships that are not based purely on an increase in one variable always leading to either an increase or decrease in another); torturing the data until it confesses (running multiple tests on your study data, controlling for different things, until something significant seems to pop up); and the following problem inherent in studies designed to see if two things like being married and smoking are correlated: 

"Surely the chance is very small that the proportion of married people is exactly the same as the proportion of smokers in the whole population. So, absent a crazy coincidence, marriage and smoking will be correlated, either positively or negatively."

Any one who is serious about critically evaluating the medical literature owes it to themselves to read this book.

Where the Analysts Went Wrong Part II

In the introduction to this post, I mentioned that a major problem with psychoanalytic formulations concerning the origins of personality dysfunction is that they presume that problematic interactions with parents and other family-of-origin members are only powerful in shaping personality functioning with young children.

When I first started getting interested in family systems ideas and started asking my adult patients about their current interactions with their parents and other members of their families of origin, it soon became clear that some of the interactions followed certain patterns than recurred again and again, and that these patterns served as triggers and reinforcers, as a behaviorist therapist might say, for the very feelings and behaviors that the patients were coming to therapy to try to change. 

Until I started asking about these interactions, as I described in my post Don’t Ask Don’t Tell, the patients had not described them in much detail. When they broke out in tears while telling me about them, however, it became hard for them to deny that the interactions were at least part of what was making them feel bad.

These were patients in individual psychotherapy, so I was not a first hand witness to these interactions, although later I found ways to see them in person. And my psychoanalytic and behaviorist psychotherapy supervisors had not discussed what to do about them.

It seemed to me that if my patients were just more assertive with their families, they might be able to change these problematic family interactions. The behaviorists had taught me about something called assertiveness training, so I tried that. The first time I tried it, I tried to teach a Chicana woman to stand up to her father. She wanted none of that. Wouldn’t even really discuss it. So, I thought, maybe it’s some sort of cultural force that I was up against in this particular case.

So how about with a patient from a somewhat more egalitarian culture? I taught an Anglo woman with traits of borderline personality disorder (BPD) to be assertive with her family. Her parents seemed to be subtly sabotaging her efforts to establish independence from them. When she was doing well, they ignored her. Or more like gave her the silent treatment and a cold shoulder. When she was in financial trouble, however, they were always right there to help out -although strangely they gave money to her teenage son rather than to her!

Every week in therapy she would dutifully practice assertiveness techniques, and would leave the session confident that she could address the issues with her family. The very next week, however, she would come back with her tail between her legs.  Her best efforts seemed to have been totally defeated, and she became even more unhappy than she had been, and even less self-confident.

I discovered that as a therapist I was absolutely no match for this woman’s parents in affecting her behavior, either for good or for ill. And it was not just her. I found out – again and again - that parents were way more potent influences on the patient than I as a therapist could ever be.

Behavior therapists do not seem to have figured out that the interactions of a patient and his or her family of origin are important triggers to many of the complaints that patients come to therapy with. Nonetheless, their behavioral interventions do often change certain aspects of a patient’s behavior, and therefore they feel that their patients are responding to treatment. In fact, if a patient’s family of origin is not too dysfunctional, they are correct. The patients change their behavior and the family basically accepts the change, so everything is cool.

But in significantly dysfunctional families? Not so much. The family therapists were right. The entire family will confront the patient in a variety of ways that all boil down to the message, “You are wrong, change back.” Many times I have even seen relatives such as aunts and uncles who previously had had little involvement with the patient come out of the woodwork screaming, “How can you do this to your mother?!?”  

Sometimes the situation would escalate to incredible extremes, with parents figuratively sticking their heads in the oven threatening suicide in response to the patient’s meager attempt at self actualization, or doing what they want and not what the parents seem to want. 

The psychoanalyst Karen Horneyonce opined that “basic anxiety” requires a sense of isolation from others, helplessness, and of being surrounded by hostility. Such perceptions and cognitions are closely related to attachment to others in one’s family system. The human brain encodes social events from the family exceptionally well, and is exquisitely sensitive to them.

“But,” I hear you protest, “a lot of patients with personality disorders are highly oppositional to their parents, seemingly doing the exact opposite of what the parents say that they want. So that theory can not be right!"

My answer to that: these people are oppositional to their parents because that’s what they think the parents need from them. The parents seem to need them to be black sheep. For a further discussion of this point, I refer you to my post about the role of the spoiler.

New developments in neuroscience are consistent with the proposition that parents can have strong effects on their children even as adults, even if they do not want to. Studies have shown that the perception of faces activates specific cells in the amygdala, which is the part of the brain responsible for fear reactions. Different cells there respond to different facial features, and certain cells respond only to one parent or the other. 

The amygdala is also strategically located for generating a rapid and specific autonomic nervous system and endocrine pattern in response to complex social signals. Lesions of amygdala in primates cause an inability to appraise social signals from other members of same species. An afflicted individual cannot distinguish whether another member of their species is coming towards them to fight with them or to mate with them. And again this is in adults, not juveniles.

In general, the attachment system seems to be one of the most important regulators of overall arousal. The amygdala is the first responder within fractions of a second; one’s initial fear orientation is not affected by conscious cognition. However, the signals then go to the other areas of the brain for further evaluation. Information regarding social context directly affects this appraisal process.

Attachment research indicates that the brain regions that compose the limbic system use input from the emotional states of attachment figures to regulate both internal and external responses. Individuals exhibiting so-called disorganized attachment have been found to have parents who display both frightened and frightening responses. 

In a sense, rage and panic are both communicated to and conditioned within the offspring of such parents. According to attachment researcher Mary Main, if parenting generates multiple, contradictory models of attachment, this creates a sense of insecurity in the offspring.

Complex limbic system reactions to the social environment have been found to be specific to important individuals within the family. Problematic reactions such as rage attacks can be seen to occur with one parent but not the other! If interactions with primary attachment figures are highly stressful over prolonged periods, this can have a profound effect on the development of a child’s brain that last a long, long time.

Early learning may be particularly difficult to inhibit. In general, it is much harder to unlearn fear than to learn it in the first place – a fact highly consistent with the experience of psychotherapists trying to extinguish chronic anxiety, particularly chronic interpersonal anxiety. 

Extinction of fear responses has also been found to be context specific. If a fear response is extinguished in one context, it may come right back if an animal is moved to a somewhat different environment. If the new environment is similar to another one such as the early family environment, fearful patterns of behavior learned early in life but inappropriate for the new environment may therefore be seen.

So, early influences are very powerful, but that does not mean that later experiences are inconsequential. When individuals grow up, their parents usually continue to act in ways that recapitulate social interactional sequences from the patient's early life experience. This parental behavior automatically both cues and reinforces old but engrained role relationship schemata (mental models of how to respond to different social cues).  

In turn, these reinforced schemata become more likely to be activated in the patient's current social interactions. This leads to reenactment and recapitulation of these patterns in other relationships. This is the basis of what Freud referred to as the repetition compulsion.

As I have described, parental behavior seems to be an extremely potent environmental trigger for previously learned social behavior. This most likely stems from the survival value of coherent group structure in evolution. As psychoanalysts have hypothesized, children internalize the values and role behaviors of their social system, and conformity to the group has in the past continued to have survival value throughout the life cycle.  

Parental behavior has such a powerful effect in triggering old schemata that it does not have to occur with any great degree of frequency in order for its effects to continue. In adults, the reinforcement of schemata occurs in a manner analogous to the learning theory paradigm of a variable intermittent reinforcement schedule. That is, the powerful parental behavior may be witnessed infrequently but unpredictably, leading the patient to continue to react rigidly in ways consistent with old role-relationship expectations.