666. Decoding Disinformation in Numbers and Narratives with Aaron Brown

Speaker A: Professors fm.

Speaker B: You're listening to the Unsiloed podcast with Greg LeBlanc. Produced by University FM, Unsiloed is a series of interdisciplinary conversations that inspire new ways of thinking about our world. So wherever you are, enjoy today's episode. And here's your host, Greg LeBlanc.

Speaker C: Welcome to Unsiloed. This is Greg LeBlanc and I'm here today with Aaron Brown, who is an author in, well, a bunch of different domains. I'm going to say you are a poker player, a sports gambler, a risk management professional, having been chief risk officer at aqr, the well known hedge fund. Also a teacher of finance and quantitative methods at a bunch of different schools including Columbia and New York and now New Mexico State, and also the author of a couple books. The book I have right here is called Wrong how to Extract Truth from a Blizzard of Quantitative Disinformation. And this was preceded by a book called the Poker Face of Wall Street, Red Blooded A World of Chance. And what Risk Management for Dummies is that one? That's one I have to get. I think I gotta get that one. Did I get em all, Did I name all the books?

Speaker A: You know, Four Dummies. Four Dummies is a really good series. Get em for everything.

Speaker C: Yeah, no, I actually have a colleague who did something like Business Strategy for Dummies and he still, actually still gets pretty good royalty checks from this thing on a regular basis.

Speaker A: Yeah, they sell pretty well. I will say the experience writing books for Wiley is like being a singer songwriter. You know, you write what you want, you get up on stage, you sing it for Dummies is like being the front piece in some arena rock thing where you are just the annoying talent in front of this gigantic machine and you get ordered around, told to rewrite things and they have all these crazy rules about things. But it's really good for figuring the best way to learn something is to teach it. Well, the best way to teach it is to write a four Dummies book.

Speaker C: Yeah, well, okay, so in this book, Wrong Number. At the end of the book you said something about how in the enlightenment we went from a world where everything was an argument from authority to a world where evidence and data were going to be behind all the arguments. And this meant that pretty much anybody had the capacity to potentially evaluate arguments and claims if they only understood a bit of math. But it seems like that never really happened. I mean we still live in a world where people, they kind of choose what to believe based on who says it and whether they like It. Right. And I find this to be true not just for people who don't have a lot of education, but even people who do have a lot of education. So I never get terribly disappointed when I hear about uneducated people being misled or choosing to be misled. But when I find out that highly educated people, people in academia in particular, people who are editing journals and writing for these journals, when they blatantly disregard basic quantitative hygiene, it disappoints me. And I guess it clearly disappoints you, perhaps even makes you a little angry. And so I guess you have a bunch of different examples and you have a couple of arguments. But I mean, why do you suppose it is that people continue to get misled either willingly or unwillingly by quantitative information, which is relatively easy to disentangle?

Speaker A: Yeah, I'll take a slightly different. I have really no opinion. If someone wants to get misled, fine. I'm a libertarian. Mislead yourself all you like. And I understand there are a lot of professional reasons to do this, right? I mean, there are political reasons, academics, you have to publish papers. You can't publish enough high quality papers to satisfy people, so you got to publish some low quality papers. The thing that I really wanted to get at, at the book book is why does nobody care? So, for example, the opening story about the Chinatown bus study from the National Transportation Safety Board. So they take a whole bunch of Greyhound accidents, they call them curbside accidents, and then they say, hey, curbside accidents cause seven times as many deaths as traditional carriers. And then they shut down all the Chinatown carriers. So fine, I understand why somebody reading that. It's a slick. The NTSB put out this slick color brochure with lots and lots of stuff about how great the NTSB is, how hard the study was. There are hundreds of names in it. I mean, this is a study you could do in five minutes, right? You have a list of fatal accidents and you just say, okay, what's the rate per passenger mile of these two things? But I understand. So you read it. You trust people. It's the ntsb. They're in charge of highway safety. Why would they lie? But when it was exposed, none of the papers that ran headlines on this stuff ran retractions or did investigations or called for the heads of the people who had done this. It was just so, okay, doesn't care. Move on to the next story. And I know these journalists, they're not idiots and they're not people who like to get pushed around and lied to. And there's a lot of serious coverage about this issue with good hard hitting journalism. But yet this one aspect that. Oh, the NTSP lied, eh? Ah, who cares? Similarly, with all the other examples in the book, what really gets to me is we know, and it's been documented over and over in lots of different ways that most published research findings are false and yet nobody seems to care when somebody cites, oh, I have a study that proves it. They say, well, most studies are wrong. So that's actually evidence against your point. And this is what I'm trying to break through.

Speaker C: Well look, I mean it's clear there's a marketplace for information, a marketplace for studies, right, and reports. And so is this just a matter of, I don't know, producers catering to demand? And I forget who it was who said that. Whenever you think that something is due to malfeasance or to, I don't know, evil motives, most of the time it's just due to incompetence or stupidity. I mean, do these things have to be motivated by greed? I mean, clearly the bus companies, they're bus companies that benefited from this study. So they're going to be supplying part of the demand. But is there another demand that's being satisfied which just the reading public just wants to have somebody they can point a finger at?

Speaker A: Well, here's what I had to say about it. Okay, so yeah, you can say this is a conspiracy, but it was too incompetent to be a conspiracy. See, if you were going to have a conspiracy, you'd stuff the ballot box with the Greyhound. Fatal accidents. But that's the only thing you do and you carefully hide it. And by the way, the NTSB fought Reason magazine for six months to release the name. So the National Transportation Safety Board is saying their bus company is seven times as dangerous as the rest. But we won't tell you who they are. And they only released the information by accident in the end. But they made dozens of other statistical errors that were pretty obvious. So if you're a competent conspiracy, this is not what you would do. And you don't even need a wrong number. If you're a conspiracy, just go shut down the Chinatown. You just say they're dangerous. Nobody's asking for a study to justify it. But it's also too conspiratorial to be incompetence because all of the errors went in the same direction. They all made the thing look more dangerous. And this is actually the key thing I'm kind of struggling with. It's not conspiracy, it's not Incompetence. It's something else. David Zweig, I don't know if you know him. He wrote an, uh, Abundance of Caution. Great book.

Speaker C: I interviewed him.

Speaker A: Yeah, it's sort of along the same lines as mine, except he focuses very closely on one issue instead of 31 issues. And he and I talked about this. It's some kind of tribalism. It's something that you want something to be true. You have to say it's true. To be a member of the tribe, the tribe needs it to be true. Nobody is overtly dishonest, or at least you don't need to assume some people are dishonest and conspiratorial and driven by greed. You don't need to assume everybody's stupid, but everybody somehow finds their own accommodation somewhere between malice and stupidity, and they find their own comfortable niche where they can not care about this.

Speaker C: Well, okay, but there are these people who actually invested a considerable amount of time and energy and effort in producing these studies. So wouldn't it be easier just to make stuff up than to actually.

Speaker A: Exactly, yeah. If it's a conspiracy, you don't need to do all this stuff.

Speaker C: Like, Donald Trump just makes numbers up out of whole cloth, right? Whenever he wants to make a point, he just says, I don't need to study. Like, I just say, pick a random number out of the thin air, and I'm giving people what they want. So it must not simply be that people want to hear a number. There needs to be some, I don't know, some trapping of at least some effort that went into putting together something that looks remotely like a study.

Speaker A: Yeah, so this is a macro phenomenon. You can't justify it on individual, rational grounds. You have to say, there's some macro phenomenon. I call it tribalism. Not just because that's a convenient thing, that no one individual is completely malicious, no one individual is completely dishonest. But people need varying combinations of these two things. So you need to put together something that the malicious people can use. The incompetent people can just assert, uh, that they trust, and everybody can get together and agree on this thing. If I was so inclined, if I were a historian or something, I might start saying, well, a lot of religions kind of work this way. You got the true believers, you got career opportunists, you got different people accept certain parts of it and would deny others, but they all get together and form a religion that can fight wars, that can take over governments, that can make mass movements. I think what's going on, you might call Them sort of a. There's a mini religion about these things. And, um, you need some things to satisfy some believers. You need some things to satisfy others.

Speaker C: Well, let's go through some of these examples.

Speaker A: Sure.

Speaker C: I mean, one of the examples is this USAID study where I guess people who were supporters of USAID came out with the claim that they saved more people than was actually physically possible. Right. And I guess the question is, why do that? Is it because you think, well, this number is ultimately going to get watered down, so we got to kind of make an aggressive case. I mean, do you think that anybody who actually participated in the study believed the number, or did they just not look at it long enough to ask the question of whether they believed it?

Speaker A: My personal feeling, and it's just a guess, I have not talked to the authors of these studies, but I've talked to a lot of people who have put their names on bad studies. I spend a lot of time trying to replicate studies, and I often have to get in touch with the authors to find out stuff they left out or to get their data or something. And often they just refuse. But often I'll get a co author who's pretty far down the line, not one of the main authors, and they'll be helpful and they'll tell me, they say, I had no idea. They asked me to do it. I needed a publication. I did not. Five minutes. I did a day or two of work on one little issue, and I never read the paper. I think this is very common. So I think it is entirely possible that a public health guy who is not an economist, who's not an expert in data or anything like that, got the idea, I want to write it. I want to figure this out. It's an important number. Right. And got some colleagues together, and they sort of figured out, here's how we would do it. But they were trained data people who knew how to do this. Right. And then they roped in a data person and they kind of said, here's the conclusion we want. Here's the data. We got. Work on it a bit. And he came up and said, well, you know, data don't really support it. No, no, no, it's got to support it. And somehow no one person maliciously, no one person out of incompetence did this. But the group did something that no individual would have done. It's entirely speculative, this is my guess. But it also got past the Lancet reviewers. And if you read what Lancet claims they do for rigorous review, hiring outside data to Thoroughly audit everything. And none of this was done, clearly. And you get science journalists who are putting this, the headline. And these are not uneducated people. These are not people who don't care about the truth. But if they had looked at the study, they would have said, wait a minute, this just doesn't make sense.

Speaker C: Is this just a free rider problem? It's kind of like you're on the assembly line, you notice that there's a flaw in the vehicle. But do you really want to pull the andon cord and stop production? Or is it easier just to kind of let things go so you can make it to your lunch break? I mean, is that really what it is?

Speaker A: That's a little unfair? I think these are people who would pull the cord when they were sure, but nobody's quite sure. And everybody trusts a little more than they should. So you see something a little funny like, wait a minute, that gear shouldn't be there, but the guy behind me did there. And there's an inspector.

Speaker C: Somebody else will. If there's a problem, somebody else will catch it.

Speaker A: Yeah, there must be. You know, there's a great story here about the Channel underneath the English Channel from Iglich de France. When they built it, one of the big concerns in the engineering community was there fire in the tunnel. And there was a study put out and it said, oh no, the chance we'll get one fire every 800 years and it won't cause much. And as you may know from history, it's every two or three years. And some of them have been very serious. And the reason when you read the study, you find out what they did is they had a list of like 20 things that are going to happen. Right. There's an inspector who watches every train go into the tunnel. Well, when the first fire happened, it turned out nobody was there because everybody knew that there was all these other safety precautions. As soon as, as the fire started, all the smoke detectors got knocked out and a lot of them weren't in place in any way because people knew, well, we don't really need the smoke detectors because we got all this other stuff. So this, I think, is a lot of it. This idea that you put 20 layers on something and then nobody takes any one layer seriously. You don't have a clear person. Nobody stood up and said, I'm the person responsible for this bad study. This ridiculous study that everybody can see on its own face is wrong. I'm the person responsible for it getting published. You got a diffuse thing with 12 CO authors, 20 people working in the journal, science journalists reporting on it, and no one person is standing up. If there were one person who was required to do it, one person who was going to get fired if this wasn't true, then you'd get better system.

Speaker C: Yeah, I mean, look, you've been in the world of finance, where anytime you make a decision to believe something, you're putting money at stake. And same is true, of course, in the world of, uh, poker and gambling and so forth. And so, I mean, I feel like if you were to go up to any one of these authors or any one of these journalists or any one of these reviewers on the review board and say, okay, would you be willing to put money on this? They'd be like, no way. I'm not going to put money on this. And I mean, I feel like that's the kind of question that people ought to ask themselves before they allow themselves to publish something. I mean, how confident are they that what they're releasing and publishing and endorsing is at least likely to be true or possible to be true?

Speaker A: Yeah, betting is the tax on bullshit. Uh, I've forgotten who said that. I was defending Nate Silver, but yeah, that's true. And there's a conference I go to, the International Conference on Gambling and Risk Taking, which is a really great conference, um, because it gets all kinds of different people. We get casino executives, advantage gamblers. We get economists, economists, mathematicians. We get people who treat problem gamblers, and we get them all together in a room. And this comes up every time somebody says, well, how much would you bet on that? There's a group of academics who are offended at the question, why would I bet on my. I just put together this system that says, you can make a lot of money betting on this or that. No, I haven't bet any money on it. And that's a rude question. Why would I sully my hands with something like that and then sell books on it instead? Right. And the advantage of gambler says, that's the entire question. If you see a fortune teller, you say, well, why couldn't you tell that you were going to get in a car accident today? If you're a faith healer, why are you sick? But no, it's just a complete disconnect between the people who are willing to bet on, uh, what they say. And if you hang around. There's a lot of downsides to hanging around gamblers. There's a lot of unpleasantness about it. But one thing I always like about it is in a company of gamblers want to bet is not a rhetorical question. And you get very careful about what you say, because anybody can say, want to bet? And if you won't, or if you hedge it in such a way that you can't lose, you've lost your status. You really have. Every opinion you venture, you got to be willing to put money on.

Speaker C: So, I mean, is there just too many degrees of freedom when it comes to research methods? Right. Because you recount a bunch of studies where people just decided to delete tons of data. Right. Just say, oh, well, that data is irrelevant. Or, those are outliers, they're inconvenient. Let's get rid of them. I mean, or deciding what variables you want to include and what variables you want to exclude. I mean, it seems like researchers have almost like a, I don't know, blank slate when it comes to deciding what methods to use.

Speaker A: Yeah, that's kind of the Fisherian. You know, Ronald Fisher in the early 20th century. That was their movement. Their movement was make everything rigorous, make everybody do things exactly the same way, make every calculation done exactly the same way. And it just didn't work. And it didn't work for two reasons. One is it stifles innovative research. You got to be able to come in and say, okay, everybody else looked at this. Everybody else did a regression. But regression is just the wrong tool. I'm going to do this other thing, or I'm going to look at the data differently, I'm going to count things differently, or I'm not going to use the unemployment rate. I'm going to use some other series or something like that. But the other problem that didn't work is you just can't do it. People can always find degrees of freedom that you left out. It's like you can't write a law code that decides every case in advance. Somebody is going to find a loophole.

Speaker C: Yeah, well, a lot of the studies that you point to that are problematic are ones that are, uh, associational or observational, and then they make some kind of causal inference.

Speaker A: Right.

Speaker C: And it seems like no matter how many times you tell people correlation isn't causation, they need to be reminded of it over and over and over again. Right. I mean, shouldn't professionals understand this? Isn't this. I mean, this is the kind of thing we teach over and over and over again. But it seems like at some point the lessons get lost, Right. When you're trying to. I mean, obviously a causal story is one that you can act on. It's something that we want the demand is for a causal story. No one really cares. In medicine you hear this term risk factor all the time, and it's very carefully chosen to make it clear that it's associational. But then the minute people start trying to put it into practice, they kind of forget that it's merely associational and they act as if it's causal. So I mean, why haven't we adequately got this to sink in?

Speaker A: I think the answer is pretty simple, right? You spent 0.01% of your life in statistics class being told that correlation is not causation, and the entire 99.99% of your life, you're learning in the real world that association is often causal, or at least assuming it is safer than not assuming it, right? If you pull, uh, a switch or something and all the power goes off in your house, well, you don't say, well, association, it's just an association. I'm not going to worry about it. You say, okay, it probably might be causal and I better investigate and see. Or I ate this and then I was sick the next day, oh well, I'll eat it again and see what happens. So you got millions of years of evolution, you got life experience that tells you take association seriously. But what's misleading about that is it's a very filtered. Your brain is filtered. Your brain knows that what might or might not be causal or what might or might not be important, when you start doing it numerically, you can generate millions and millions of non causal associations very easily. And frankly, causation is hard to prove. I mean, everything causes everything in some sense. A butterfly flapping its wings in Brazil causes a tornado in Texas three weeks later. So untangling that is very, very hard. Association is incredibly easy to prove. Especially today. Uh, you can go on your Claude or your AI, whatever AI you like. You say, go out and find me some data on this, run a statistical test on it and show me the correlation and do thousands of them until you come up with one. I can publish. And I'm not saying that's exactly what people do, but what they do isn't a lot better.

Speaker C: Mhm. Well, I mean, in the world of medicine it really does matter quite a bit because doctors are in the business of giving out advice. And you looked at one study in particular which had to do with marijuana consumption, and you pointed out that this was entirely associational and was actually poorly designed and there's lots of data exclusions and so forth, and yet it still managed to convince people who were pretty good otherwise at Seeing these Marty Makari,

Speaker A: who I really respect, who's written his own book about this kind of stuff and yeah, he liked the study. But here, what I really like about this study as an example is it's a case where we know it's not causal because they studied lifetime heart attack and, and uh, cardiovascular events and marijuana use the last 30 days. So clearly it could not be causal just by design, it couldn't be. And yet they again they're careful when they write the paper association and this and that. But then they have the thing on the top advice. Physicians should tell people not to smoke marijuana, not to use marijuana because it'll cause heart attacks. And there's just zero evidence for that in the paper.

Speaker C: Well, I mean that at least suggests that people are aware of multiple levels of scrutiny, right? So if you use a certain language in your publication and then a, ah, different language in your interviews with the media, let's say then at least you're aware that there are some more rigorous standards that you have to pass through at the publication stage.

Speaker A: See, I'm not sure that's actually true. I think the lead author of the paper probably doesn't really understand this or care about the distinction that the statistical people in the paper, the methodology people wrote that stuff. Or maybe some editor came in and said no, no, you can't say. Cause you have to say association. But the top level stuff doesn't get reviewed. I review papers a lot. I never get to see the paper as published with its clinical recommendations, with its non technical summary, anything like that. I don't get to see any of that. I just get to see the kind of guts of the paper. So I might go through and I might say okay, this is good, they've qual it, everything is right. And then when the paper appears in print it's got stuff on the top that I never would have signed off on. This is part of the problem. The anonymous reviewers, they don't get paid. It's uh, a lot of work, it's thankless, everybody hates you for it. And you don't really get to keep out the bad stuff. You can say okay, none of the stuff I looked at was bad. But then they could add bad stuff at the end.

Speaker C: Well, I mean, look, academic papers are kind of like movies in that they get a lot of reviews and people decide whether to see the movie based on the reviews. And I think people decide whether to pay attention to the paper based on the reviews. And like in Science and Nature they publish the papers but then at the beginning, they have sort of somebody who summarizes them. And then ultimately that paper might get recognized at New York Times science pages and so on, and it'll filter down. Maybe People magazine will have a piece on it and so forth. So it seems like there's this, I don't know, this distribution system for how knowledge gets disseminated. And one would think that at some layer there's going to be some journalists that are going to have a very skeptical eye. And it seems like that is the area where we would expect these things to kind of get caught.

Speaker A: That would make sense. And, uh, there's actually a really great institutional example of this. There's the ipcc, the International Something for Climate Change panel, I guess, on climate change. And they put out these reports. And the IPCC reports are really great. If you care about climate change, you read one of these reports. They're big, they're like 3,000 pages, but you can get through them. They only come out every three or four years or so, and they really summarize what we know about it. But they have these summaries, so they have the whole thing. Then they have the, I don't know, summary for X summary. And the final one is summary for policymakers. And you can see as you progress up these things that the nuance gets lost, things get more confident, things even get misstated sometimes, and then it expands out again. And so then you got the serious journalists covering it, and then you got the less serious journalists, and then you got the word on the street. And what happens at the end has no relation to what happens if you read the papers, if you tell people, here's what I read in the actual IPC report, they'll call you a climate denier. Because none of that stuff got to the policymaker summary. And then it got totally reversed by the time it gets out to the common knowledge. What you're seeing there is just a documented example of what I think is happening everywhere, that if you actually go down and read the papers and talk to the scientists and researchers, you actually learn a lot. Science is not in terrible shape, at least most fields. Some fields are, but most of them aren't. And an intelligent skeptical person, without a lot of technical education, you just sort of basic. If you had, uh, a freshman college course in sciences, you can follow this stuff and you can learn a lot, and then these people know a lot. But there is this compression to the abstract of the article and then this expansion in the journalistic press to the point where what actually gets out there to form policy, to decide legal cases for people to vote on has no relation to what the researchers know.

Speaker C: Yeah, I mean, you say that in reality there's very few solutions, but lots of trade offs, it seems like. I mean, I try to teach my students to read scientific papers, give them the tools to evaluate evidence and so forth. And I know they have the capacity to do this. Whether they have the motivation is a different story. And it seems like understanding nuance and complexity requires a little bit of effort, a little bit of mental energy that people, put simply, don't have, and particularly they're not going to bother to use it in areas that don't really affect their lives. I've done some other podcasts about misinformation, and if you go around saying that there's Pizzagate or something, fine, it doesn't affect your daily life. But there are a lot of these things that actually do affect your daily life in a major way. How you decide to live and how you decide to eat and how you decide to pursue medical procedures and so forth. And it seems like even in those domains where people are going to be investing an enormous amount, they don't invest nearly as much in evaluating the claims as they do in acting on them.

Speaker A: Yeah, well, the best example of that, I don't know if you know the book Outlive by Peter Attia. Here's a great book. I came across it a couple of years ago and he has done this in medicine. He's cut through all those things. He said, okay, here's what you need to do to live 10 years longer. And it's simple, by the way. It's not unconventional. It's nothing that would really shock anybody about what is advisable. But this works. This doesn't. These are things, things you should consider taking. These are things you should stay away from. And it's just sensible. It's very much against the medical idea of, um, oh, let's cure the people who come into the hospital. It's how to stay out of the hospital in the first place. And so I love this book. So I gave it to everybody I know. You got to get this book. It's take you a couple hours to read it and it'll add 10 years to your life. And more than that, you won't have to just trust your doctor. You won't have to agonize over choices. He's actually got the, uh, information and data for this stuff. I don't think anybody had gave it to Reddit. Everybody said, oh, thank you, that's Great. And I talked to him later. Oh, yeah, I'm going to read it someday. It's not hard to read. It's easy to read and it's a great book. It's really sensible. So, yeah, if they won't do it to add 10 years to their lives, they're not going to do it to find out if USAID really saves 92 million lives.

Speaker C: Right. But again, I mean, if there's no action taken as a result of these studies, these faulty studies, then I guess we shouldn't care. But it seems like there are actions that are taken that are affected by these studies.

Speaker A: And Covid is the great exam. We shut down schools for two years. We ended eviction moratoriums, bankrupted lots of landlords. We made people wear masks and stay away. And, um, some of it was good. We told pregnant women to take the COVID vaccine, which had never been tested on pregnant women. And we only now are learning that it doesn't cause horrendous developmental difficulties in their babies. It's not another thalidomide. These are all incredibly consequential social decisions that were taken on the basis not just of unknown. Uh, it's not that we didn't know. It's not that we had weak studies. We had studies that were clearly wrong. And they were driving it. And they were driving it because they told people to do what they wanted to do.

Speaker C: Yeah, maybe we can talk about that eviction moratorium study, which seemed fantastical.

Speaker A: Yes, actually, there are several of them, but the one that I. Well, one of the ones I put a lot of emphasis in the book is from Duke University. And Duke University, as you may know, has had enormous problems with faked data, uh, with academic fraud. Whenever they paid a huge fine, they set up all these procedures to fight it for the future. That always seemed to have worked. So, paper comes out.

Speaker C: It doesn't seem like whistleblowers ever really benefit. There's no whistleblower reward system.

Speaker A: It's not a wise choice. It's a. The whistleblowers get investigated, they go to prison, and the scientists get zero consequence when they're exposed. But anyway, so they posted a completely ridiculous study that said 40% of COVID deaths were caused by evictions. Which you start doing the numbers, and that says, okay, every single evicted person must be dying of COVID And people study evictions. And they would notice, hey, there's nobody to study because they're all dead. People study COVID deaths. They'd say, hey, 40% of these guys had just been evicted, so they would notice these things, and they don't, because it isn't true. But anyway, so they didn't give their data. They didn't say where they got their data, except one place they said they got their data, but it didn't have the data they said they got from it. So I write them. Um, they refuse to answer. I go to the Duke organization that's supposed to. They won't do anything about it. And BER National Bureau of Economic Research published it. They said, oh, no, we don't make authors reveal their data until it's peer reviewed. And of course, the article was never peer reviewed. And again, this is the kind of thing. So here's a study, and this study was cited several times in court cases and legislation. And this is the reason why a lot of landlords went bankrupt. And it was just an absurd thing. Now, there were other eviction studies that were good, and they showed far lower costs. But somebody might say, okay, that's a big enough cost. The problem with the whole thing, by the way, is it's just economically irrational. So the claim is evictions increase crowding because the evicted people go live with others. But of course, it also frees up a place. Crowding is the number of renters divided by the number of apartments. And eviction doesn't destroy apartments. It just moves different people in. So it was crazy in the beginning. Now, the stress of eviction might cause health problems. I mean, it might be said that, okay, it's a pandemic. We don't want to stress people, okay, fine. And you could study that, and you could probably find a small effect, and you might say, well, reducing eviction stress will save 100 lives. And maybe that's worth it, maybe it isn't. You have to, uh, look at other things. But those studies never got published. Nobody cared about that. They wanted. It'll save a million lives.

Speaker C: Well, I mean, does this mean there are inadequate checks and balances? I mean, look, we have, you know, the FDA that evaluates drugs before release. Uh, it seems like the refereed publications, they're supposed to play a role. But also, I think peer pressure is supposed to play a role. Your colleagues, when you publish a bogus study, your colleagues are supposed to kind of look askance at you. And of course, we have examples of this, right? So Dietrich Stoppel, who you talk about in the book, I mean, he just made stuff.

Speaker A: So most of them don't confess.

Speaker C: He just made stuff out of whole cloth. We've seen a couple People like Francesca Gino get burnt at the stake equivalent of in academia. But it seems like a couple sacrificial lambs are thrown out there just to kind of say, oh yeah, look, we're doing our job. But the vast majority of these bogus studies just sort of go through without any difficulty.

Speaker A: And there's a secret there that you'll see that peer pressure is very important, and it is what keeps science honest. But the way it works is so you're a scientist, you're a good scientist, you're doing good research, and when you come up with a good paper, you present it at conferences, you tell all your peers, you emphasize it a lot, then you do a bad study that you just got to get because you just got to notch up another publication. So you send it into some journal, you don't talk about it, three years later it sees print and nobody reads it, and you don't see anything about it, and it just sort of goes there and dies. So this is the standard way a lot of academics work. And if you look at their publication record, you'll see a few big good publications that they talk about a lot that get cited, and you'll see a lot of junk. The disk gets put out there and they don't talk about it. And so they don't get any pushback when, ah, first of all, nobody proves it wrong because nobody cares. Nobody thinks it's right in the first place. And in the second place, nobody would really care because. Because everybody does it. The Franciscanos of the world, their crime is their hyping their bad stuff, and that is what can get you caught. Or confessing writing a book about how you did it, that's the other way to get caught.

Speaker C: It's not the crappy studies that disappear that are the problem, according to your book. I mean, it's the ones that actually get notoriety, that get a lot of attention, that people actually design policies around these things, that people radically change their behavior because of these things. I mean, those seem to be the ones that cause the most damage, not the just crappy, poorly designed studies that fade off into the sunset.

Speaker A: Yeah, I agree. And I think most researchers are careful about that and they're careful not to let the university press office get a hold of their bad study. They're careful not to go out and give interviews on it. The ones who forget that the ones who their study happened to get the public attention, the ones that push that, they're the ones who cause the problems and they're also the ones who get Caught. I mean, not many people do get caught, and the consequences of getting caught are pretty low. But it can happen. If you want respect in your community. When I say you get caught, when I say the consequences are low, you're probably not going to get fired, you're not going to go to jail, nobody's going to find you. I mean, that's very, very rare. You lose professional credibility. And that's extremely important. I mean, that's really the be all and end all for most researchers I know is what their peer researchers think of them. And that's where you get hurt. In fact, you get hurt even for getting publicity for your good work. There still is a real feeling in a lot of sciences that the guy in the headline is not a real scientist. That if you're getting too much attention, and it's a, uh, it used to be a, uh, serious mortal sin in academics to talk about a paper in public before it's been published, whereas now you send it off to the journalists before you even send it to the journals, and it may never get published. And this is where sort of things, a lot of the checks and balances that were set up were under that assumption. The old thing, my dad's day. My dad was a scientist of the old days. The idea was if you were forced to talk about your work in public, like you never talked about your specific work, you never led with it anyway. You would say something like, well, here's what people think in the field, and I've done some work that suggests it might be this. But now you never see that. Now it's, I just proved X and my one paper. Forget the context of the field, forget what other people think. My one paper has proven X. This would have been a, uh, heresy in my dad's day. He just never would have gone anywhere in academics doing that. Now, in some fields, it's the norm.

Speaker C: Well, I want to turn to the world of investments because I'm always surprised at how. I don't know why I'm still surprised, but I'm often surprised by how scientifically minded many investors are. I mean, I've been to a number of hedge funds. I know people who work there, and many of them, they're reading scientific articles, they're carefully poring over research studies. I mean, it seems to be an area where motivated reasoning isn't going to get you very far in the world of investments.

Speaker A: Yeah, well, it'll kill you. And the experience of trying and losing is humbling. And you learn, okay, here's how much I know crowds are pretty smart 1% of the time. I can get a little bit of an edge on them if I'm careful. If I use a Kelly sized bet, I don't bet more than my confidence can justify and so on. I get very careful. And the other thing you learn is one mistake can wipe out 20 good decisions. So you get very careful. And I mean, I like to read a lot of science, I just find it interesting for one thing. But yeah, it's a very good way to kind of hone your thing. A good science paper can teach you a lot about how to find truth. Somebody will come up with a new idea for how to test something in physics or chemistry or whatever, and they'll say, hey, I can use that. I can use that to find out some interesting stuff. I also read a lot of sports analytics. People are doing a lot of really interesting work in baseball and you can learn from it.

Speaker C: Well, I teach a course on behavioral finance and I do a lot of work also with investment professionals, particularly in the area of behavioral finance. And it seems like professional investors, they love to learn about biases, they love to learn about decision problems, they love to learn about mistakes, they love to go through some decision hygiene lessons and so forth. But this is actually a relatively new field. I mean, I remember back in the 90s people didn't talk about behavioral stuff. And you've been affiliated with quantitative investment programs for a long time and most of them didn't have components on behavioral investing or psychology of investing. And I'm, uh, wondering why. And do you see it as integral now? Do you see it as mainstream? Do you see it as something that every investment professional is now thinking about?

Speaker A: Yeah, well, I was at the University of Chicago, PhD program in the early 80s, and we had Robin Hogarth, we had some of the early, uh, behavioral economics people teaching in business schools. And it was controversial and a lot of the students thought it was one of the fluff courses and whatever. But I think the prejudice at the time was. And this prejudice has a good reason that there's a long tradition in social science of, uh, explaining things by people are stupid. And the trouble with it is you can explain everything that way. And if you explain everything, you explain nothing. And so people said, no, I demand a rational explanation. Unless you can give me a rational mathematical reason why this happens, you're just, it's worthless. It took Daniel Kahneman, Amos, uh, Tversky, and a few other pioneers, Robin Hogarth among them, to convince people they weren't saying everyone is stupid. They weren't saying, I can explain everything. They are saying, I can explain certain things that are persistent, that aren't just after the fact justifications for what people do. And it's a hard lesson because it's been like a century of fighting against that to get it out of the field. It's a little bit like physics had to fight against magic, against astrology, against alchemy, and say, okay, no, we're only interested in reproducible stuff you can do in experiments. And it took a long time and it took a lot of money because it's easy to do these lab experiments, but they're not very convincing. And then when you get out in the real world, well, there's so many confounding things that it's hard to demonstrate. So it really did take a generation of researchers to do it. And it's still, for a lot of people, it's still in kind of a. They're still suspending judgment on it. What I find is I give a lot of talks and I find it only works if you first demonstrate it on the audience. So I usually have, I give them a game beforehand. I mean, there's some famous examples, but I give them a game, I say, okay, here's 10 things, I want you to bet on them. M. And I'm going to prove to you afterwards when the say, okay, everybody in this room bet more when they were wrong than when they were right.

Speaker C: I've made a lot of money in these demonstrations.

Speaker A: Yeah, unfortunately, it is more effective that way. You do the one where you auction off a hundred dollar bill, but the top bidder gets it for what they bid and the second bidder has to pay.

Speaker C: Yeah, who needs a speaking fee when you can do that with a bunch of hedge fund managers? I mean, no speaking.

Speaker A: It absolutely works. Absolutely works. Or you have bid for things. So there are a lot of these standard things when you do it, and people see, okay, I just did what he said I was going to do, then they kind of open up and listen. I'm not sure the lesson really sticks, though. It's very hard to get outside yourself and say, okay, I have to do. And even when you do it, it, what do you do about it? So, you know, you have this behavioral bias, but so you do the opposite of what you intended to do. Right. So as a risk manager, I coach traders, right? So I go to them, I say, okay, I've gone through your trades and I can show you. You bet more when you're wrong than when you're Right, so let's work on it. You can't just. Okay, I'll reverse all my bets. Right, because you'll start fooling yourself. Why are you doing it? What is it about? What are the things you can rely on? What are the things you can't? And we can make them better traders and we can show it. Your P and L went up, your Sharpe ratio went up.

Speaker C: How do we explain that finding? I mean, people do bet more when they're wrong than when they're right.

Speaker A: Well, this is just my pop psychology. I don't claim that I verify this in any way. My feeling is you have a lot of voices in your head. And by the way, this work's not just in your head. This is also for institutions. And you have these loud ones like fear and greed that are just not reliable. And you have these quiet ones that are reliable. When fear and greed are shouting, that's when you bet big. That's when you know you got it. When fear and greed are kind of quiet and the quiet voices are 60, 40 for something, that's when you're right but you're not very confident about it. It's not that you. And in fact what it is is when you're not very confident you're right, you shouldn't be all that confident. You should bet a moderate, sensible amount. When you're highly confident, you're demonstrably wrong. And that's just something in human programming. Like I say, it's just a pop psychology way to think about it. I think about it in terms of you got a committee, 20 people are together and you got a few of them are shouting and claiming they know what's right. And if you follow any of them, you're probably wrong. If you say, okay, everybody shut up. I'm only going to listen to the quiet voices and I want the median quiet voice. That's my general rubric for life. Listen to the median quiet voice. Don't worry if some people are 70% sure and some people are only 49%, 49% is one side, 70% is the other. They count equally. And that's where you are most likely to find the person who's right.

Speaker C: Well, I mean, I wonder if things are different in the world of Venture because in Venture you're going after a different tail. It seems like they always say you want to go for the thing that is non consensus. Probably you don't want to listen to the medium quiet voice if you're in that business. Is there a different approach to risk management? There.

Speaker A: Yeah, I think that's a population thing. Okay, so there's the people who are. You're trying to make money from. Right. You don't want to go with their consensus. Right. You want to go against their. You want to see, I guess, the difference in media. So I got my team and we're trying to make an investment and I say, okay, what's our media and quiet ways? What do we think? Then we go out and say, okay, what does the crowd think? And for the crowd, I don't think we need to go this medium quiet for the crowd. We just want the simple average. What's a crowd going to do? But yeah, you have to assume that people know what they're trying to do. So let's say you're trying to make a venture investment and you know it's only got a 10% chance of success, but you want the one that's a 10% chance of a million shot, not the one that's a 10% shot of a 20 to one shot. So you've got to make sure you're asking the right question. We're not asking is this likely to work. Were asking how big would this be? If it does work, I want to

Speaker C: go back in time because it sounds like you knew what you were looking for very early in life. And you decided to go to Harvard because you wanted to study with John Tukey and Frederick Mosteller.

Speaker A: Yeah, Tukey was at Princeton. I met him through Mosteller. John Quine. Actually, the biggest one was Harrison White, the mathematical sociologist. He did a paper that changed my life.

Speaker C: Now, did you want to study with them because you wanted to be a degenerate gambler or why did you decide you wanted to study with these guys?

Speaker A: I actually had that debate with my father. I thought of gambling. I mean, the fact that I could make money betting on horse races, the fact that I could make money at poker, These were validations to me. These were a life change. You got to figure out, can I beat the crowd? If I could beat the crowd? There's one life path for me. Go against a crowd, come up with your own ideas. And if I can't? Then I should listen to the crowd and I should follow established paths. So I didn't think of it. So I thought of it as I am, um, learning if I can be a pathbreaker or whether I should stick to the conventional. He was kind of a semi communist, Bob. Um, and he just had no respect for money that wasn't earned. So he was very uncomfortable with his. But simultaneously he was very proud that I could do it. So this was always a tension between us, sort of. But it was okay with him that I was going to study with. Mostella was very big on this stuff. Harrison White was very big on the stuff. Klein wasn't. Fischer Black, the other guy, who was my mentor, was totally opposed to gambling, hated gambling, but they knew how to do it. But now Harrison White's paper was. I was a big science fiction fan, so I read foundation and Empire, the Asimov series. And he has psycho historian who can and say, well, individual humans are predictable, but the mass of them is statistically predictable. And he can write the next thousand years of history and not only write it, but figure out how to change it for the better. And of course, that's pretty ridiculous, but it was a goal. It was something, boy, that would be really great if we could do that. And Harrison White actually started doing it. He started building these mathematical models that could explain incest taboos and inheritance cycles and things like this. And this was just. To me, this was, okay. This is the future. Unfortunately, sociology took a different turn. And very little I read from sociologists today has any validity or interest for me. But his School of Mathematical Sociology and some of his students, like Duncan Watts, I worked with all these people. He had us all in a seminar. And once a week for four years, we'd meet and we'd talk about these things. And it was very highly, uh, formative event for me.

Speaker C: I found it interesting. You described how the bookies, or the odds setters, the organized criminal conspiracies, these guys would harvest insight from the good gamblers, the knowledgeable gamblers, to help set the odds for the uninformed gamblers. And you were one of those people who they were looking to basically to help set the odds for a bit.

Speaker A: Yeah. So if you're a blackjack card counter, so you're going in and you're winning in casinos, they kick you out. Or worse. In those days, it could be worse. So it's a real shock that when I'm winning at sports betting, they get really friendly and they start offering me extra terms and good conditions and things like that. And they like me. And of course, it's obvious, right, because they were using my bets to, uh, make money. And this is back in the organized crime days when there was a monopoly on sports betting and the mob only cared about balancing the bull. They didn't want their people guessing. They were very suspicious of that. If the book was winning too much, they were suspicious, because if it Was winning too much. Somebody could be skimming. But if it was having exactly equal bets on both sides and they were just taking their 5%, great. That's a business they understood. And by the way, that's one reason why they weren't offering to hire me. Not that I would have gone to work for them, but they wanted me outside the organization. I was a civilian. I was a civilian who they were making money from. And by the way, when I figured this out, I quit. I mean, I realized I wasn't beating the bookie. I was working for the mob and ah, had better things to do. But it did take a while to figure out. It seems obvious in retrospect, but at the time, it just seemed like, hey, this is great. They're not, uh, threatening me like the blackjack people. They're nice to me. They're saying, oh, you don't have to put up any cash. We'll pay you in cash immediately afterwards. And you never have to put up any cash. We'll just carry you for as long as you want. It just seemed like, wow. Hey, that's nice.

Speaker C: Well, it makes me wonder, do you think that we should incorporate gambling into our educational system? It seems like a lot of the statistical insights that folks like Pascal came up with. Right. Were solutions to gambling problems. Gamblers basically bankrolled the entire statistics industry for a while. But it seems like.

Speaker A: By the way, let me just quickly interject. Not only that, professional gamblers underwrote the telegraph and telephone wire. That's where all the capital came from from them. Um, and professional sports. Every professional sport was founded by people so they could bet money on it or so they could make money off other people betting on it.

Speaker C: It.

Speaker A: So, yeah, so it's a big social

Speaker C: deal, But I mean, should we just give our kids an allowance and say, okay, go start gambling? Because. Not because we want them to be degenerate gamblers, but because we want them to learn about how to be more objective when it comes to assessing probabilities and outcomes.

Speaker A: Well, I have two answers for that. There's the hard truth. I actually have a Bloomberg column on this today. The hard truth is that Only something like 3 or 5% of people can do this successfully. So you're training people in something that 95%, 97% are going to fail.

Speaker C: Well, look, I mean, not with the expectation that they're going to make money from it, but this would be like an expense. It would be like an educational expense.

Speaker A: Okay? And it is true. Everybody should learn how good or bad they are and how to get better at it. But it's still a hard thing. And I think we have subjects like that where we really need a few people who are going to be really good musicians. So we make every kid take music and most of them hate it, and most of them drop it the second they get out of school or something like that. Now, do some of them get some appreciation for music? Do some of them have richer lives because they were forced to practice piano for an hour a day for two years of their life? I don't know. Maybe not, But I think it's a little bit like that. What we certainly want to do is identify the people who are good and train them.

Speaker C: But it seems like learning that you're bad at something actually has educational value in this domain, which it doesn't have in other domains. I mean, learning that I'm bad at music isn't going to really improve my life, but learning that I'm bad at evaluating probabilities, I mean, that could have a profoundly positive impact on one's life. Right?

Speaker A: I totally agree. And this is my defense. There's all this thing about how most poly market, most prediction market players lose. Okay? Yeah, well, they're buying something valuable from this. But like I say, it is a hard truth. We don't like to say 97% of people are failures athletes. We don't like to tell you go out and play football for high school because you could make $100 million playing for the NFL. Well, yeah, you might, but most people don't. So again, I think it is an activity like that. We defend sports that way. Right. Sports build characters. Teachers, teamwork teaches people what they're good and bad at their limitations, things like that. But we have to get comfortable with the fact that only one team wins the championship, only one person gets the medal, only one in a thousand college athletes makes it to the pros and things like that. And people are not comfortable with that truth.

Speaker C: Sure. But doesn't it help you in other domains of your life if you're trying to learn to be less overconfident and learn to. To make rash decisions and learning you want to learn how to not fall prey to spurious probabilistic arguments? You can always hearken back to those experiences you had, like, yeah, I really wasn't very good at playing those odds.

Speaker A: Well, you're preaching to the choir. But I do think most people, I think, are more comfortable not knowing. Most people are more comfortable going with the crowd, not asking the question or going against the crowd. Because they like going against a crowd, even though they have no evidence that their views are any better. And I don't think you'll convince either of those people that they would be better off knowing whether they were actually good at something or not. For the same reason those academics at the International Conference of Gambling and Risk Taking didn't want to bet. They were happier not knowing whether this system really worked.

Speaker C: Well, I guess. Last question. I mean, you are coming across as a Bayesian in your work. And I mean, Bayesian reasoning is something which, it's hard to deny its value. And yet it seems like the statistical methods that we all learn to know and love in university, they're not always Bayesian. Right. Which can lead us astray. So what's both the, um, promise and the risk of taking a Bayesian approach to things?

Speaker A: Yeah, well, okay, I studied under a lot of religious fanatic Bayesians, I'll call them. I don't mean that in a pejorative way. People who are very rigorous, I respect rigorous thinkers. So people who say, okay, I am going to pick something and I am going to believe it, I'm going to follow it wherever it goes. But I want to put that aside. The alternatives to Bayesianism, in my opinion, are, uh, all philosophically incoherent, but some of them work pretty well. So I call myself an empirical Bayesian. Um, and the key is you don't have to go all the way to Bayesian. You don't have to drink the Kool Aid to use its insights. So the empirical Bayesian, Strict Bayesianism is you search your brain, you decide what your prior belief is, and then you interpret evidence to change that prior belief. And it all happens inside your head. It's entirely subjective. Well, okay, fine, but that doesn't answer a lot of questions we need, like which drugs to approve, where we need consensus, and where you have to talk to other people. And there are a lot of statistical techniques that are philosophically incoherent but work pretty well. So an empirical Bayesian says, okay, well, let me just take a base. Let me just sort of look out at. Without thinking about things too hard. What's the base rate here and the empirical base rate? That's what I believe. So if I want to know who's going to win the presidency in 2028, well, okay, I just say, okay, how often do Republicans versus Democrats win? Or how often does the switch after a president use two terms or something like that? Then I go out into evidence and I use my evidence against the base rate. So for a Bayesian, that's heresy. That's. No, no, no. You can't use the base rate and it's philosophically incoherent, but it works really well. You don't have to go all the way to Bayesian to know that what they're doing in the journals is wrong. With the journal, the Frequentist, the Fisher Classical hypothesis testing, the, uh, gold standard double blind control trials, those things wrong. And you don't have to go all the way to Bayesianism. You can just say, okay, we can just show mathematically that those don't work.

Speaker C: Well, Aaron, thanks so much for joining me. The book is called Wrong Number, but I'm, um, now intrigued and I want to go check out some of the other books that you've written.

Speaker A: I want to learn more about A quick plug in here. There's a video series on Reason by the same name that more fun to watch.

Speaker C: Yeah. And I think you have a much larger set of examples that didn't make it into the book about wrong numbers. Yes. Well, thanks so much for joining me and hopefully we'll chat again soon.

Speaker A: Thank you for having me. Bye.

Speaker B: Thank you for tuning in to the unsiloed podcast produced by University fm. Um, if you enjoyed today's episode, please give us a five star rating and review in your favorite listening app. To listen to our other episodes, please visit our website at www.unsilopodcast.com.