Don't Trust What You Read (Except This, Of Course)

In 'Wrong Number,' author Aaron Brown aims to 'smash any illusions you might have that the most respected sources in the world would notice or care about obvious false claims.'

Scroll down to watch the video interview I did this week with Aaron Brown, the author of “Wrong Number: How to Extract Truth from a Blizzard of Quantitative Disinformation.”

I knew before I read “Wrong Number” that a lot of published research was shoddy, but I didn’t realize how pervasive the shoddiness is. In 30 chapters, Brown shows that even respected academic journals can’t be counted on to screen out bad work. And once bad work slips past the gatekeepers at the journals, it often gets into the prestige news media. Which means that over the years, you have probably read dozens of articles about research that simply isn’t true.

It pains me to write this because Brown’s book could further erode the already low trust that people have in academia and the mainstream news media. I worry that they’ll turn to even less reliable sources, such as sensationalist news outlets, rumor-mongering social media and self-serving politicians. (Personally, I try to make sure that studies I write about are correct, but I admit I’ve probably been insufficiently skeptical at times.)

I’ll give just one disheartening example from the book, which concerns an article that appeared in The Lancet, a prestigious British medical journal that brags, “We select only the best research papers.” The article, which came out last year, said that funding from the U.S. Agency for International Development prevented 91,839,663 deaths worldwide from 2001 to 2021.

Brown shows that the world death rate did indeed decline from 2001 to 2021. But the decline reduced the number of deaths by only 79 million over the period, not nearly 92 million. So the paper was essentially claiming that this one American aid program saved more lives than the total number of lives that were saved by all causes.

That’s just silly. As Brown writes, it implies that “all other foreign aid and efforts to reduce global mortality backfired to the tune of killing 13 million people.”

The shame is that the U.S. Agency for International Development really did save lives before the Trump Administration shut it down last year, transferring some of its responsibilities to the State Department. Papers such as the one in Lancet feed the perception that U.S.A.I.D. was worthless, and its supporters were unfairly attacking Trump.

There’s a cadre of people who do what Brown does, namely poke holes in published research. Deirdre McCloskey and Steven Ziliak wrote “The Cult of Statistical Significance.” Gary Smith of Pomona College wrote “Standard Deviation: Flawed Assumptions, Tortured Data, and Other Ways to Lie with Statistics.” Andrew Gelman of Columbia and Xiao-Lin Meng of Harvard are two other notable critics of bad research. There’s a great website called Retraction Watch.

The profession knows it has a problem. One promising solution is for scholars to pre-certify their research, spelling out in advance their hypotheses and methods. That prevents them from mining the data in search of a publishable effect when their initial plan yields nothing. Releasing all data and all statistical tests to the public also engenders confidence. But most scholars aren’t doing these things, and the bad research keeps coming out.

I asked Brown whether “Wrong Number” could turn people nihilistic. He said he’s a libertarian himself, not a nihilist, and he counsels critical thinking, not outright despair.

“Well, I think the answer is, don’t, don’t disbelieve things, be skeptical,” he said. “If somebody tells you something, say okay, that person believes it, maybe they have some good arguments for it, whatever. But don’t be too sure.”

That’s fair.

TRANSCRIPT

Peter Coy 0:11

Hello, I’m Peter Coy of the Substack Economics for Everyone. Please follow me at petercoy.substack.com. Today I’m speaking with Aaron Brown. Aaron Brown is a past professional poker player, trader, finance professor, portfolio manager, and head of mortgage securities. Among his jobs has been the chief risk manager for AQR Capital Management. He’s been a columnist for Wilmott, Quantum, and I think Reason magazine, and he is out with a wonderful new book, and its title, I should get this right, is Wrong Number, and I’ve been reading it. How to, subtitle is, How to Extract Truth from a Blizzard of Quantitative Disinformation, and when he says blizzard, he means blizzard. Aaron, welcome to the program.

Aaron Brown 1:40

Thanks for having me, Peter.

Peter Coy 1:42

I wrote a review of Poker Face of Wall Street. I forgot to mention Aaron’s also a multiple time author, and he had a wonderful book called The Poker, The Poker Face of Wall Street, which I reviewed for BusinessWeek magazine a long time ago. Love that one, and I said, well, have to read this new one from Aaron, and I’m glad I did. I recommend it to all you viewers. Aaron, first of all, thanks for being on the show. Second, as I thought about how to do this interview, I thought it’s not easy, because some things that are easy to convey in print are harder to get across in, you know, video interview like this. As soon as you start wading into p values, it becomes rather abstract and hard to get across. So, but since you presumably been talking about this with a lot of people, you’ve probably gotten a feeling for how to translate what’s in a book on the page into something that makes sense to people just kind of hearing it and seeing it, so give you the first shot at trying to explain the book.

Aaron Brown 2:55

Okay, well, I think the first message is you don’t need to think about p values, that my book is not about subtle statistical errors people make. This is not a book for statisticians to sharpen their pencils. My point is, these wrong numbers are not fraud, they’re not subtle errors, they’re egregious errors that any moderately skeptical person who thinks about what they’re reading would reject offhand, for example, one of my early chapters is on USAID aid cuts supposedly saved 92 million lives between 2001 and 2023. Well, the entire decline in global mortality over that period was 79 million lives, so the authors are claiming USAID saved, everything is responsible for the entire decline in global mortality and saved 13 extra million lives from nowhere, to boot. So, you think, okay, I don’t have to be a statistical genius, I don’t have to know everything there is to know about mortality or how foreign aid works or anything like that. I mean, I know USAID, USAID is only about 40% of US foreign aid, and US foreign aid is only about a third of all government foreign aid, and there’s plenty of private sector foreign aid, so one program can’t possibly be responsible for 114% of the entire decline. Now I go further in the book, and I show you how you know all the other many things they did wrong in this study, but it’s published in the most prestigious and oldest medical journal in the world, The Lancet. It was reported on by all the major media outlets, and nobody seemed to have asked a question, you know. Gee, is this remotely possible? And I have 31 chapters like that of just studies that are just on the surface obviously egregiously wrong and. I yet seem to get no skepticism from either journalism or for journalists or from the general public. They get quoted in the Congressional Record, they’re the basis of legislation, they get cited in court cases, and this is something I’m waging a campaign to fight.

Peter Coy 5:19

Yeah, now I think you’re selling yourself a little short, though, because you’re saying that these studies are egregiously on the face wrong, and yet clearly they weren’t egregiously on the face wrong to the reviewers who they passed, they passed peer review, and they also were quoted in the media, and so on, so that must mean that the errors were not quite as simple as two plus two equals five, and I will say another thing, which is that you have a bachelor’s degree in what is it

Aaron Brown 6:00

Applied mathematics

Peter Coy 6:01

Applied mathematics, which is heavy duty stuff, and you also have an MBA, and you worked on Wall Street, and you’ve been a risk professional, so you are much, and you’ve been doing this, has become a hobby of yours to accumulate these examples and to dig into them in some depth, so things that might seem egregiously on the face wrong to you might not strike other people that way. What do you say about that?

Aaron Brown 6:30

Well, I confess, I consider myself an expert at this stuff, and I think I can find a lot more problems with the work than a person can, but I maintain for all the studies in the book, I think a person of average skepticism – pretend this study was trying to sell you a mutual fund, and you read the abstract, the questions you would ask from that, you might not say, ‘Gee, this is the worst study in the world, I know it’s wrong, but you certainly wouldn’t buy the fund, and, and you should be able to, you should be able to read the abstract now. Now, there are fields where that isn’t true, right? You might read some abstract for a cancer paper, and there’s not one English word in it, it’s just all jargon and whatever this and that, and you have no idea what it means. But the papers I pick are ones that are meant for a general audience. One of my points is where science, you know, where science happens is in the woven web of conjecture. That’s a quote from ancient Greek Xenophanes, and you know real scientific work. Somebody publishes it, it’s not true because that one publication came. It’s because other people build on it or try to refute it. They come at the same question from different directions. They weave it together, and eventually you have something where there is a consensus that has considerable intellectual force behind it, even if some of the threads are wrong. The studies I show in my book are things that are released directly to the media. Nobody builds on them, nobody tries to replicate them, they never get put in part of any web, and a guy named a statistician at Stanford, John Ioannidis, wrote back in 2005 most published research findings are false, so when somebody says a study, I have a study that proves this, your answer should be okay. It’s more likely wrong than right. I am more likely to believe the opposite than than to believe your study, yet that hasn’t filtered through to people. I think it’s a little bit like religion, you know, people hundreds of years ago, most people gave up the idea that there’s like an old man of the sky who does miracles, where you know the Bible is literally true, things like that. But despite the fact that the core belief in religion has mostly gone away for most people these days, people still go to church, they still pray, they still join religious groups, they still, you know, talk about being religious. I think the scientific establishment has gone that I think most people realize that most public, most published research is false, but they go ahead and cite it. They go ahead and believe in experts. I think a lot of people have realized Philip Tetlock’s work. Philip Tetlock’s a friend of mine who wrote Expert Political Judgment and said experts’ predictions are worse than random, and the more prominent the app expert, the worse the predictions. Yet people still listen to experts. Yeah, I write an op-ed for Bloomberg. Some people believe it, you know, but they shouldn’t, you know. The average op-ed from Bloomberg probably is wrong.

Peter Coy 9:38

Wow, you know, I see your point, I guess. I’m congenitally less skeptical than you are, even though I take all your factual points. I think you’re right on the facts, but I guess my problem is that where does that really leave you? We already have a very skeptical society, distrusting society, and that one of our problems is we don’t trust each other as people, and that is a corrosive thing. I don’t think we should trust each other just when it’s not merited, of course, wouldn’t say you should just just take everything at face value, but it, it’s like I just don’t know what message to absorb from the book on a meta level. Should we just not believe anything we see anymore, or what?

Aaron Brown 10:41

Well, I think the answer is, don’t, don’t disbelieve things, be skeptical. If somebody tells you something, say okay, that person believes it, maybe they have some good arguments for it, whatever. But don’t be too sure. Yeah, I think I think people are overconfident, so you say, be more humble, recognize there’s a variety of opinions, and you’re probably not smarter than everybody. You’re probably not smarter than the wisdom of crowds. So, so have your opinions, you know. Read, read, make up your mind, you know, see what seems right to you, but don’t start betting that it’s going to be true. So, it’s skepticism. The fact you read it in the Wall Street Journal, you read it in the New York Times. The fact that Peter Coy told you that on a Substack, you know that that’s okay, that’s a piece of evidence, that’s something in its favor, but it’s not the whole thing. And, and I find it across the political spectrum. I’m a libertarian myself, and you know libertarians have their own plenty of wrong numbers. I took at University of Chicago, I took courses from Sam Peltzman, a really smart guy, and wrote some great work, and he wrote papers that are remembered by many libertarians I know as the FDA kills 100 people for every person that saves, and seat belt regulations kill more people than they save, and if you go back and read the papers, there are a lot more nuance to that, and there’s a long literature on these things since people have come in, and they have. We now have a woven web of research on these topics. You know what the SBA does, and some, you know, does some good things, does some bad things. Seat belt regulations, there are some unanticipated consequences, but there are also some lives saved, and yet you know people remember the exaggerated version of the early work and don’t correct it, and it’s not just libertarians. I mentioned that only because I want to point out it’s not just things I disagree with. I didn’t write a book about wrong numbers of that. I don’t agree with ideologically. I tried to pick from across the spectrum. I do have to say the book does come across as populist. Populists distrust everybody, and I think that’s the problem you’re getting at.

Peter Coy 12:56

Yeah,

Aaron Brown 12:57

so the book does kind of come in, and if you only read my book and you read nothing else, it might turn you into a bit of a populist, you know. The government lies, the media is irresponsible, experts are all idiots, and I do not be mean to give that message. Just be more skeptical of them.

Peter Coy 13:18

Yeah. Oh, that’s fair. I mean, as a journalist, it’s.. I’m paid to be skeptical, so I thought that the book should be read by all journalists, and probably some of them are watching this right now, because we’re sort of the front line. We’re the ones who take the stuff in, we get the press releases, we see the journals, and we’re not even at the front line, probably the front lines are peer reviewers, but the second line would be assuming it gets past them, and we’re, we should be there to make sure it doesn’t get past us, and you have a lot of great material in there about what sort of things to watch for, and you know, maybe one thing I can do now is, again, I don’t want to go through

Aaron Brown 14:08

again for just a minute, I do want to say something about peer review. Okay, peer review is is a method for enforcing conformity. It is not a useful gatekeeper. Peer reviewers are not paid, it’s to really peer review a paper is a lot of work. Yeah, most of them don’t bother. Typically, what I find most peer reviewers do is they scan the paper for two or three comments they can make, so they can prove they’ve done their job, and they send it in. And to the extent they make a decision, it’s based on their feelings in the field, you know, part of the fields they like, the ones that don’t. Now, what you want is, you want review by statisticians, you want review by experts, say the methodology is right. You want a named journal editor who actually makes a decision, this is an important thing or this is not an important thing, and a statistician to say the work is good, the work is not. Good, the peer review step is something we should, we should just get rid of. Peer review should happen after publication. People should contest the work if they disagree with it, or they should validate the work if they agree with it. But that is one of the problems that people take peer review as a sign of credibility, and it’s almost the opposite.

Peter Coy 15:22

Wait a minute. Why do you think the peer review should happen after publication? It seems like then you’re just saying the journal should just open the door and let the stuff in.

Aaron Brown 15:33

Okay, well, you’ve got a journal editor who’s a real person, named person, and that person reads the paper and says, yes, this is an important paper, this is right. I’m an expert in the field, and, and I picked this paper. I’ve got 100 people sent me in papers. We can only publish four of them. These are the four I picked. I sent them to statisticians who validated, yes, all the work was done properly, all the data has been uploaded, all the code has been uploaded, and these are credible researchers. These are, you know, and so we publish the paper. If I disagree with it, if I’m one of the peer reviewers and I disagree with the paper, then I publish my own paper or comment to the paper and attack it and explain why it’s wrong. These discussions should be out there. The journal is a gatekeeper, but this anonymous, you know, letting anonymous colleagues, you know, be part of the gatekeeping process is just wrong. If a call, if the methodology is sound, if a journal editor thinks it’s worth publishing, and you disagree, you should make your objections in public.

Peter Coy 16:38

Well, I will say, my wife does peer review. She’s a doctorate in demography, does a lot of stuff on religious demography, and so I see up close the problem you’re talking about. One thing is very hard to recruit peer reviewers because people don’t want to do it, they’re not getting paid, they don’t get any credit for it, and so the the few people are willing to do it are barraged with more and more papers, here would you review this, would review that, and it’s my wife’s, my wife is meticulous about it, but it’s easy not to be. On the other hand, I want to say something that you have it in your, in your book, which is there are there have been efforts to improve matters. One of them is pre-registration of studies, which seems to me like if it were widely done, would be a huge benefit. The idea of that is you say upfront what your hypothesis is and what your methods are and all that, and that prevents you from going back and sifting through your data again and again, trying to find something publishable where you’re bound to eventually, you know, throttle the data into churning something up that seems like a big improvement. Another is, and you mentioned this in your book as well, being disclosing your data to anyone who wants to check it out, the raw data, so it’s relatively easy to see if it’s replicable. Do you think that those two things have made a meaningful improvement?

Aaron Brown 18:22

Well, none of the studies in my book do that. I shouldn’t say none. I think I think I want to correct that a little bit. I think I think a few of them did. But basically, if somebody does that, the best practice today, you pre-register your hypothesis, you upload all of your data and all of your code to GitHub or some, some other repository like that. In some cases, if you have, you may have contractual or privacy reasons. I mean, you can’t give the entire data, but you give a, you give an anonymized version of it, or something. When the people do that, I would say, yeah, I say most published research findings are false. I would say 80% of the ones that do that the finding is defensible or true, and 80% of the ones that don’t, you’re probably wrong, because, and also I write to people, I say, okay, I read your paper, I’d like to replicate the result, not because I’m questioning it, although I do question it, but because I want to understand exactly what you did. So, somebody will write something and say, “Well, the rate was 20% lower in counties that did this. Well, you know, there’s a lot of different ways you could define that, you know, and I want to know exactly what you did, or they said, you know, we did some, you know, Mann-Whitney correction for this estimator. Well, there’s, you know, five different versions of that in our code that you can do, and I want to know exactly which version did you run, and I sometimes I would, you know, my classes, I sometimes assigned to. People, they say, okay, take this paper, replicate it, because it’s all public data downloaded from FRED or downloaded from some public source. They tell you what they do, you’ll almost never replicate the results of the paper. Now, sometimes it’s because the data changed over time, these data sources get, but sometimes it’s because you used one version of a regression package, and they used a different regression package. Sometimes it’s a misunderstanding, you did something wrong. Often it’s a step the authors left out. Oh yeah, that’s right, we excluded those data points. But so this is why you have to upload things, so people know exactly what you did. And with AI, by the way, AI is dramatically changing publishing. I know the people have figured out right now. If I want to do a wrong number, if I see a paper and I’m skeptical, I like Claude. I use Anthropic Claude. I’d say, Claude, go to GitHub, replicate it, tell me if you got the same number, and usually the answer is no. And we say, okay, what happened? You know, and it’s trivial stuff most of the time, you know, it’s, you know, they did this, they did that, often the GitHub is often not exactly the version they did in the paper, like they might, they published the paper, but they continued to update their data or something like that, so there are all kinds of benign reasons why that could be true, but I then say, okay, Claude, I don’t, I don’t buy their complicated model that they put in. Let’s just do the simple, straightforward regression of this. Do we get the same answer? And you know, sometimes it’s yeah, yeah, you do. It’s all their fancy stuff didn’t really make much difference. Sometimes it’s no, it’s completely different. And that’s when you’re suspicious, like maybe they did the straightforward thing, it didn’t work, so that’s why they did all this complicated, you know, techniques they did for it, but the difficulty of doing that has, you know, used to be a week’s work to really try to download the data myself and replicate a paper if the officers were being cooperative. Now it’s, you know, well, it’s an hour for Claude, and I can be drinking coffee, or you know, watching videos.

Peter Coy 22:05

Well, that’s a good thing, too, then, right? It’s Claude police the system,

Aaron Brown 22:10

I think. I think it will.. I think Claude will do what I can’t do, because Claude, Claude can scale.

Peter Coy 22:17

Yeah, and

Aaron Brown 22:18

Claude has credibility. A lot of people are going to read the book and say, okay, he’s just a professional critic trying to sell books, and I mean it’s not totally false, you know, I am a professional critic, and I would like to sell books, but you know, if you’re used to working with an AI, and you start trusting it, and by the way, be very careful, there’s a certain kind of trust you can give it, and a certain kind you can’t, you know, they do make incredible errors too, right, but, but still, you say, “Okay, hey, Claude, is this a reasonable paper? And Claude will come back and say, “Well, you know, it seems to be well written, and he did post to GitHub, and it seems to all check out, but here, you know, eight people disagree with him and did it differently, and there’s, you know, there were some comments submitted that said it was methodology, methodologically fault, flawed, and you can find this stuff out,

Peter Coy 23:06

you know. I just want to put a shout out to other people who do what you do. There’s like a community of you, and I guess you probably know each other who find studies that don’t stand up. There’s the website Spurious Correlations. Gary Smith, Pomona College, has done a lot of this work, and I think you cite him in your book, Deirdre McCloskey, who’s at Cato now, and her co-author Stephen Ziliak, who wrote the book The Cult of Statistical Significance. These are all people whose work I’ve seen over the years. Xiao-Lin Meng, by the way, Harvard Data Science Review, really Andrew Gelman at Columbia, is

Aaron Brown 23:52

another Retraction Watch. The website watch,

Peter Coy 23:56

yes, must be mentioned.

Aaron Brown 23:58

I would toss in David Zweig has been very he, David Zweig, did An Abundance of Caution. He only focuses on school closures during Covid, but it is in many ways a much more devastating book than mine, because it, you know, he goes into a lot more depth, and he shows exactly how, you know, how bad research was, you know, got in formed policy, and how deaf people were. It’s the, you know, in sociology. One of my mentors in college was Harrison White, the sociologist, and he very big discipline on, you know, don’t ask individual motivation, you know, look at the social structure, look at the network, and whatever, and there is this, you know, systematic irresponsibility that nobody is responsible for, you know, this USAID paper, the Chinatown bus study, you know, the different things in my paper, no one person is responsible, right, the peer reviewers are anonymous, and they. As well, you know, I looked at the paper, looked okay to me. The journal editors don’t take well, the peer reviewers made the decision. The guy doing the research, well, you know, he, there’s usually 12 co-authors on these papers, and sometimes the co-authors can’t remember, you know, who did what. One of the things there was a, you know, Francisco Gino, who was an honesty researcher, was convicted of or

Accused of, and pretty clearly nobody could remember, you know, the name of the anonymous wandering graduate student who actually did, you know, the Excel data, and down, you know, did the data. I mean, this is really dispiriting when you start drinking into these things, because there’s no person who stands up and said it was my responsibility to get this right. The journalists publishing it, we trusted the Lancet,

Peter Coy 25:49

sure,

Aaron Brown 25:49

and this is, in a way, this is a problem that there is, you know, no single source of responsibility, and everything is sort of vague and overlapping responsibility, that’s another problem with peer reviewer. When you put in a layer of gatekeeping that isn’t really done, I mean, the peer reviewers say, well, I mean, I’m sure your wife doesn’t, but a lot of peers say, well, you know, the guy who wrote this is a credentialed expert, and the journal is a big journal, and all the other peer reviewers liked it, so you know what? Am I gonna.. I don’t really have to check everything. One very big problem I document in this book, in a lot of papers, there’s a key assumption you need for the paper that’s absolutely fundamental for the paper. The marijuana and heart studies, for example, it relies on the fact that you can call people up in random surveys and they give you honest answers about their drug use and their health, and he cites two papers. He says, “Here are papers that says this is reliable data, and you go to the papers and you just read the abstract of the papers, and they both say, “This data is so unreliable as to be useless.

Peter Coy 26:53

Yeah,

Aaron Brown 26:54

so none of the peer reviewers checked that. You know, the Lancet editors didn’t check it. The methodology reviewers didn’t check it. I wouldn’t necessarily expect a reporter to check it, but you know, an aggressive, I mean, a reporter who was really looking at this in depth could easily do it. You don’t need a degree in statistics to read the abstract of the paper that says this kind of data is is unreliable, and by the way, it’s pretty common sense, you wouldn’t expect to get, you know, really accurate health information or drug use information in a random telephone survey.

Peter Coy 27:28

So, this reminds me, it’s going to sound like a non sequitur, but I was on a hike on Saturday, and as a place I had destinations, the place I’d been before, and I remember take a right when you reach the power line, and I didn’t, kept not seeing the power line, so I said, “Well, I guess I might have taken my right yet. I got like a mile off course before I realized, no, no, no, no, I should have turned right way back there, because you can’t see the power line from the place I need to turn, and I went back there, I saw all these signs that made it very clear that I had just missed the right turn. It was because I was blind. I had it so set in my mind that I needed to see the power line that I ignored all of that evidence, and I felt like an idiot. But it occurs to me that this is relevant to your book, because I think in a lot of the cases you, you give the USAID example, some of these school closings, some of the vaccine data, people thought one thing very strongly, and they had sort of motivated reasoning that led them to a certain conclusion and sort of blinded them to contrary evidence. It’s not that they lied on purpose, although you do have cases where people lied on purpose. It’s more that they just didn’t see things that went against their preconceptions.

Aaron Brown 28:59

Yeah, I think, and the most of these papers, they do simplify the world quite a bit, so you know, you know the old thing, the map is not the territory, so well, you’re writing a journal article, you’re really writing about a map, so I have a chapter, no data, no problem, about people who did studies where they had absolutely no data, and and you know, and by the way, you can do it, and this is not a crime in academics, they, you know, people teach you all sorts of clever ways you can get answers without having any data, but gathering data is important in its own right, I mean, if you, you know, if you do a study, the first thing you do is gather the data, and the first thing you do is you learn there’s a lot of stuff about the data you didn’t suspect. There are some of the I also have some examples in economics where people do studies of economic numbers and they don’t really understand the numbers. There was some of the greedflation data. Greed inflation studies, we saw a few years back, when people were trying to prove that inflation was a result of corporate greed, that treated the producer price index as if it were a cost index, you know, of course it’s not. A producer price index is the cost the producers pay, it’s a cost index, it’s, I mean, in a broad sense, it’s telling you the same thing as the consumer price index, except it’s measured in a different, you know, at a different point in the process, but it’s, you know, a deviation between the two doesn’t prove whether there’s corporate greed going on or not. There’s an entire field of economics, the kinked demand curve theory, that was based on misunderstanding how the government surveyed prices, and so this is, yeah, you’re looking for a certain thing, you see a statistic, it seems like it should tell you what you want to know, and you find out, well, it really doesn’t, it really means something else.

Peter Coy 30:58

Yeah, one of the things I thought about when I read the book is that it would help if the members of the general public had a little more grounding in statistics that would help them evaluate some of these claims. Do you think that statistics should be taught more in school?

Aaron Brown 31:17

You know, I really don’t, and I feel like, you know, there’s Ernest Rutherford. He probably didn’t say it, but he’s credited with saying, if your experiment needs statistics, you should have done a better experiment. If you’re, you know, if you’re a journalist and you’re writing a story and you need statistics, you probably should write a better story. The argument should make sense to people. The statistics is kind of the foundation. The only thing statistics tells you it doesn’t tell you about the logical strength of the argument, it only says one thing: if you buy the logic, could it all be coincidence, could it all just be random chance? Now that’s important to ask, but you don’t lead with that, you know the fact that, okay, this can’t be random chance if you’re testing two drugs, the normal thing you do is, you say, okay, my null hypothesis is that this drug makes absolutely no difference of any kind, which is never true. You’re not going to be testing anything that has zero effect of anything. Yeah, you reject that, and you say, my p value is 2% So, okay, so the drug had some effect, that doesn’t prove the drug works, that doesn’t prove it’s a good idea. And so you read the logical argument, if you like the logic, you just check the p value on, you know, on your way out the door. Focusing on statistics is a way to avoid looking at the logic of the argument. We need some statisticians in the world, but we don’t need everybody to be a statistician.

Peter Coy 32:39

Yeah, Michael Mandel used to be my boss at BusinessWeek, and he used to say, don’t do a regression analysis. If you need a regression analysis to make your point then the effect is probably too small to bother with. The things that we want to be writing about are just really big and important effects, he said. Sure, run a graph, and if you see a number shooting up to the right, assuming it’s done correctly, that tells you something. But you know, and you have, you have an example in the book. Actually, you said a rule of thumb is that the odds ratio should be, should be at least three before you pay any attention to it, so subtlety is not your friend.

Aaron Brown 33:24

Yeah, so odds ratio, so a lot, an awful lot of medical studies in all fields, but particularly in medicine and nutrition, whether it’s an odds ratio. So it says, okay, you’re, you’re, you know, 1.3 times as likely to die of a stroke if you, you know, if you’re a canoe, or then if you’d never been in a canoe or something like that. Well, the thing is, what we learn is just from experience that it’s very, very easy with absolutely random data to come up with these odds ratios that are statistically significant but are completely meaningless, and if you think of all the headlines you’ve read over the years that just quietly went away, it’s almost all these odds ratios of 1.3 1.8 You really should get an odds ratio of three to one. Now, that said, you might do a study, and you might find an odds ratio of 1.3 That okay, people who eat carrots have better eyesight. You refine it, you work on it, you say, okay, is it everybody who eats carrots, or is it only people who eat, you know, raw carrots, or is it only, you know, you know, people of Asian ethnicity, or something like that. And once you isolate it to a group where there’s a three to one odds ratio, now you’re thinking of something that might plausibly have a causal effect, but an awful lot of economic research, and in other fields as well is done on really marginal effects that are statistically significant, but have little practical significance. The Chinatown bus study, Chinatown busses are seven times as dangerous as it turns out not be true, but even if it were true. A bus passengers almost never get killed. A bus, as a bus passenger, you are more likely to die of a heart attack or a stroke, or you know, on the bus than to die from a traffic accident for riding in the bus. So, even if they were seven times as more dangerous, that’s less risky than crossing four streets. So, if you have to cross four seats to get to the bus terminal, you’re still better off taking that seven times more risky curbside bus service. So, again, the, you know, just the statistics mask the underlying truth, and by focusing on the statistics, people get distracted from asking the important question.

Peter Coy 35:42

Well, I do think, though, I take your point about how it’s not all statistical analysis. People, there are people who take statistics but never get the intuition behind it, so they, you know, might be able to calculate a p value, but they don’t know why to calculate a p value, and when you shouldn’t calculate a p value, but it does still seem to me that if you can, if there were a course that used your book as a textbook, I think that’s really more what I’m talking about, that would convey the kind of things that people ought to know.

Aaron Brown 36:18

I actually, yes, I’m working on using it, not as a sub, you know, not as a course textbook, but as a supplement, where you know each chapter has some exercises. I’m going to put up on online for instructors, so if any instructors are out there, they want to use it, I should have that up by the end of the summer. But no, the real way you want to teach this is have people bet, you know, have them play poker, have them, you know, bet on prediction markets, you know, I would run these in my course. I would say, okay, you know, we’re going to have, you know, part of your grade is going to be making predictions on things, and there’s a hedge fund chain, and not a hedge fund, their proprietary trading shop, Jane Street, and they come up with these games that help you not, not, not, not make prediction to know how much you know, so Estamos estimathon. For I’m not quite sure how to accent that one, but it, it’s a game where you, you’re asked things you, you think at first they’re called Fermi problems. They’re things you first think you think I know nothing about. How many piano tuners are there in Chicago? But if you think about it, you know something, they’re not a million, there aren’t two, but are there 100, are there 300, whatever. And your job is to come up with an interval such that there’s a 50% chance it’s inside and 50% chance it’s outside, and you get scored, you know, the perfect score is half your intervals are inside and half your intervals are outside. It’s actually a little more complicated than that, because you could game that, but you know, you can make half your interval zero to infinity, but, but, so, so they scored a little more carefully than that, but the point is, do you know what you know, and the fact is, most people don’t. Most people give either outrageously large intervals or outrageously small ones, and that skill, and you can learn that skill by practicing it. Yeah, knowing what you know is important. Most people are unreasonably certain of things they know very little about, and fail to use a lot of the knowledge they do have.

Peter Coy 38:26

Well, that gets

Aaron Brown 38:27

trading, by the way.

Peter Coy 38:28

You mentioned Philip Tetlock earlier. Sure, Superforecasters, the people who are best at forecasting are the ones who are not set in their ways, who take in new information, take it seriously, evaluate it, and they end up doing better than the subject matter experts who have too deep into their own preconceptions.

Aaron Brown 38:51

Yeah, and one thing, one very important thing you left out there is you start with a base rate.

Peter Coy 38:57

Yeah,

Aaron Brown 38:57

so the problem is, so most people, so you’re thinking of something like, who will, who will be the Democratic nominee for president in 2028 You know, and most people will start, they’ll just start thinking everything they might, you know, think that is relevant to this. Well, you want to start out, you say, okay, you know, how often did the, you know, sitting vice president, the last president get nominated, how often did the, you know, early front runner get it? Whatever, and you get these base rates based on history, based on prediction markets, based on something else. Then you only look for facts that are relevant for that, and if you, so you’re thinking of Gavin Newsom, for example, you say, okay, you know, he was the early front runner, I guess now Harris is now the front runner, but you know, so, and he had a, I think, a 35% chance, so you say, okay, he’s got a 35% chance, I’m not looking for, you know, October surprises that might derail him, that’s what I would do if you were a 90% chance, I’d be. Looking for very low probability events that might derail him. I’m not looking for some magic thing that might happen to really pull him up, that his, you know, he suddenly gets a Nobel Peace Prize or something. I know that would help, but you know, you’re looking for things for a 35% candidate, and then if you’re looking at a 2% candidate, you look for different kinds of things, and this discipline of starting with a base rate is very important.

Peter Coy 40:26

Yeah, well, doctors talk about how, if you hear hoof beats, think horses, not zebras.

Aaron Brown 40:33

By the way, doctors are the worst. A friend of mine, Dylan Evans, wrote a book called Risk Intelligence, and he got - he did a lot of research on different professions. The worst people are doctors. They actually are incredibly bad, mainly because of their professional responsibilities. There were two that were really good: professional bridge players. Professional bridge players are really great at this, and US weather forecasters, and the differences in the UK weather forecasters give quality, you know, rain is likely. US weather forecasters are required to give probability estimates, and they get rated by them. And this, this is what makes you good at it. And this, by the way, is why experts are terrible, because experts coach their couch their predictions such that they can always explain away whatever happens, and and that’s what they do, and so instead of changing their minds about things, they explain why they were really right all along.

Peter Coy 41:34

Right. Well, the old expression on Wall Street is give a number or give a date, but never give both in the same sentence.

Aaron Brown 41:41

I’ve noticed, by the way, if you, if you go and you read the, you know, popular books, the people predicting disaster always tell you when, but they never tell you how big the disaster is going to be, you know. So it’s the crash of 79 it’s the, you know, disaster next year, and the people who are predicting good things, you know, the Dow 35,000 they always tell you how big it’s going to be, but they never tell you when it’s going to happen, asymmetrical,

Peter Coy 42:10

right?

Aaron Brown 42:10

I don’t know why, but I

Peter Coy 42:12

think I do know why a lot of

Aaron Brown 42:14

people

Peter Coy 42:15

caught, yeah.

Aaron Brown 42:16

Well, I understand why you don’t give both, if you give both, you can be wrong,

Peter Coy 42:19

right? Yeah,

Aaron Brown 42:20

but which one people prefer? I think I think people who are optimists want to see that big number, and people who are pessimists want to know when it’s going to happen. They know the pessimists know there’s going to be a disaster already, that’s no not news to them. If they’re hoping you can tell them when.

Peter Coy 42:34

Sure. Well, it’s been a fascinating conversation. Think we’ll wrap it here. We’ve been speaking with Aaron Brown, the author of Wrong Number: How to Extract Truth from a Blizzard of Quantitative Disinformation. Aaron, I want to thank you very much. I’d also like to

Aaron Brown 42:56

plug - I do some videos on the same subject for Reason magazine, so if you don’t read books, you can go watch a 10 minute video of the same material.

Peter Coy 43:04

Was that reason.org

Aaron Brown 43:06

Reason, yes, or you can get them on YouTube. Wrong number, Aaron Brown, you’ll find it.

Peter Coy 43:12

You’ll find it. Yeah. Well, Aaron, thank

Aaron Brown 43:15

you very much, Peter.

Peter Coy 43:16

Thank you very much. Great to talk to you, and I can’t wait for your next book,

Aaron Brown 43:21

right. You’ll have a fair wait, I think. Okay,

Peter Coy 43:25

that’s all right.

Aaron Brown 43:26

Want to take some time off?

Peter Coy 43:27

All right. Good. You deserve it. All right. Thanks again. Bye, bye,

Aaron Brown 43:31

bye.

Transcribed by https://otter.ai