Expertise: The Knowledge of Things Unseen

I've written before about the value and importance of expertise. We used to believe, especially in the realm of science, that experts really did know more than the rest of us. Now, in a world of echo-chamber social media and fake news and "alternative facts", a lot of us (meaning here Americans) have chucked this notion out the window. Many of us now believe that we and our friends know the real truth, and that everybody else is either a dupe or a liar.

One reason why it's easy to fall into this trap is that we feel good about our echo chambers - they make us feel powerful and affirmed, a sort of antidote to the fear we've been taught is the proper response to the modern world. That part of the psychology that leads people to reject expertise and accept otherwise wacky ideas is pretty clear.

But there's another aspect to expertise that actually contributes to its widespread rejection. The nature of expertise is that people who are experts see things that non-experts can't see. They perceive things in the universe that are, quite literally, invisible to the rest of us.

This phenomenon has been well-documented in all sorts of arenas. Elite athletes, for example, have been studied extensively. It turns out that, while they tend to be in excellent health and have certain physical gifts, they're not especially more physically gifted in general than the rest of us. It's that the tens of thousands of hours of practice they put in have rewired their brains so they can perceive things other's can't. That's why the best hitters in professional baseball actually stand a good chance of hitting a baseball thrown by a professional pitcher, traveling at more than 95 miles per hour. He can see things about that ball that are invisible to the rest of us.

The same is true in medicine. An experienced doctor will see in a list of symptoms, or the way a patient answers a question, possible diagnoses that we know nothing about. Nor can we understand the connections between those little bits of information and the much larger issue. Doctors carry around a whole world of knowledge in their heads that is inaccessible to non-experts.

So it goes for nearly every field of human endeavor. Architects see things in buildings that the rest of us miss. Musicians hear things in music we can't hear. Engineers, lawyers, designers, auto mechanics - in almost any human endeavor involving expertise, experts are privy to a world out of reach of the rest of us.

Unfortunately, this makes it easy to dismiss expertise. It's easy to assume that everything you see is everything there is to see. We're pretty good at accounting for the data coming into our senses, but generally terrible about accounting for what's not there. Arthur Conan Doyle immortalized this in his story "Silver Blaze", in which Sherlock Holmes solves the otherwise unsolvable case by observing that a dog didn't bark.

I encounter this all the time in my own area of expertise - politics - because, as John Stewart Mill put it over 100 years ago, politics "is a subject which no one, however ignorant, thinks himself incompetent to discuss". In the political realm, we all think that we can see everything there is to see. And when "experts" come along and try to point out what we can't see, we often dismiss them because, well, we can't see what they're pointing at. We think they're just making it up.

There are two conclusions here. First, humility is not only a moral virtue, it's an intellectual necessity. We all need to know what we don't know (the height of Aristotle's wisdom). Second, we need to make an effort to determine where real expertise lies - not in who shouts the loudest or in who says things we want to believe, but in who has really put in the time and effort to establish a track record. Anybody can claim they're an expert - evaluate those claims carefully, especially when the supposed expert is simply confirming your own biases.

Expertise Matters

Like most of us, I get a fair bit of nonsense that floats by my feed on Facebook. This is actually the byproduct of a deliberate strategy of keeping a diverse group of FB friends and connections. I don't want to fall into the "echo chamber" trap, and I find it interesting to see what people who think very differently from me are thinking and sharing among each other.

Sometimes, though, I run across something so ridiculously silly I just can't ignore it. Today is one of those days:

Now, I get that there are people out there convinced that everything that pharmaceutical companies do is evil. Pharmas like Eli Lilly and Pfizer have unfortunately done enough bad things that, if you cherry-pick your evidence, it's not hard to justify this viewpoint (I don't say it's right, but I can see where you could get the evidence for it).

But the response to this belief is for folks to go out and "do their own research". Do they first bother to go out and obtain degrees in biomedical sciences so they can do that research? Of course not! Because anybody can read peer-reviewed scientific research in obscure journals and understand it, right?

I'm not surprised, but it does still amaze me that people can manage to tie their minds up into this particular pretzel. Even allowing for the different meanings of the term "research", this is an absurd proposition. If by "research" we mean here going directly "to the source" and reading the results that non-biased, non-industry scientists produce in their labs, the notion that anybody with less than a graduate education in the relevant field can do this is laughable. I have as much chance of getting into this summer's olympics as the average suburbanite with a college degree in communications has of understanding the real meaning of an article published in the International Journal of Toxicology, much less in understanding the broader scope of literature in which that article is situated.

This, of course, is one of the factors that allows for communities of people to band together and believe absurd things. The echo chamber that is Facebook facilitates this, but long before Mark Zuckerberg some folks figured out that you can ignore the rest of the world and believe whatever you want about it, so long as you discard the idea of expertise. In order to uphold this notion, you have to believe - even if you don't articulate - that education, practice, and experience don't really matter, and that anybody can master anything with just a little bit of effort. I mean, how hard can biochemistry be really?

The problem, of course, is that the world doesn't care what we think. As Richard Feynman famously wrote in the last sentence of his report on the Challenger disaster, "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled."

So folks are welcome to think that drug companies are evil (and somehow, amazingly disciplined - you'd think that more of the thousands of employees would leak the plans of their evil mastermind overlords). They're welcome to think that research that reaches conclusions they don't like isn't true, and that junk science and psychobabble that confirms their beliefs is real. What none of this will change is the reality underlying the science. Bacteria, genetic mutation, climate change - all of these things will happen whether we understand them or not, and whether we like them or not.

So if you find yourself patting your friends on the back for doing their "own research" that fits some broader conspiracy theory about science, take a step back. Adopt a little humility. Imagine for a moment that the thousands of people who have collectively spent millions of hours in labs struggling to master complex subjects might actually know something that you don't. Then imagine a world in which we struggle to find the truth together through dialogue and real research rather than snarky, self-congratulatory memes.

Where's the Truth? The Challenge of Scientific 'Controversies'

I wrote a little over a month ago about the use of the term "research" in public arguments over science. I argued at the time, and still maintain, that people often claim to have done "research" when what they really mean is that they have (very selectively) read other people's research. This is related, I think, to the erosion of respect for expertise, but that's a subject for another day.

At the time, a fairly robust Facebook discussion broke out among my friends about whether there isn't another reasonable form of the word "research" - what we might call "library research". Isn't it fair, some argued, for a layperson who is not expert in a given field to say that they are "researching" some issue by digging into the literature and reading what the experts have written? I will concede this point - though I would prefer the term "library research" to distinguish this from the sort of research that generates new knowledge, I think it's fair for non-experts to talk about "researching" a topic in the sense of informing themselves about the current state of what is known.

Of course, many of those who try to stir up "controversy" on scientific subjects (vaccines, climate change, etc.) aren't really doing this kind of "research" either. There's a difference between reading The Literature on a subject to try to find out what the experts think about something, and selectively reading that same research to bolster a predetermined conclusion that you've already arrived at (vaccines cause autism, climate change isn't real, the earth is 6000 years old - take your pick). Those people still aren't engaged in research, even of the "library research" variety, any more than a small child putting on his father's tie makes him an employed professional with a job. It's just window dressing, and usually pretty ill-fitting at that.

But in the midst of that Facebook conversation, one of my friends raised a very good question: if you are a non-expert faced with a controversial subject, how do you go about trying to research that topic to figure out, as best you can, what the truth is? This is actually harder than it might seem, in part because experts often don't do a very good job of communicating with the public and in part because those who are trying to create "controversy" as a way of arguing for extremely unorthodox (often demonstrably false) ideas like to muddy the water as much as they can. In the midst of that kind of free-for-all, what's a reasonable non-expert person to do?

There are a few fairly easy rules of thumb that will take you a little ways down the road. Just as the corollary to Godwin's Law states that whoever mentions Nazis first in an argument automatically loses, the adoption of any argument predicated on a vast conspiracy of silence on the part of thousands of otherwise-autonomous (if not competitive) individuals is a guaranteed loser. Efforts, therefore, to dismiss "climate science" as a cooked-up conspiracy fail on their face since such a theory would require the complicit cooperation of thousands of individual scientists around the world who all know better but have been convinced to lie to the rest of us. Ditto for arguments that the CDC  and NIH are somehow engaged in a conspiracy of silence - anybody who watches government agencies for any length of time knows that most of them leak like sieves and can't be trusted to keep much of anything secret. So if one side of a "debate" is relying on this kind of argument, it's safe to say they're probably wrong.

Beyond that, however, the waters get pretty muddy. Many folks involved in these controversies make claims to certain kinds of authority, while denying the authority claims of the other side. Take, for example, the work of Dr. Stephanie Seneff, a senior research scientist at MIT's Computer Science and Artificial Intelligence Laboratory. She has given a number of public talks recently claiming that modern chemical agriculture, and Monsanto's RoundUp product in particular, are going to make 1/2 of all American children develop Autism by the year 2025. This has been widely reported in a number of fringe science websites, often with headlines line this:

MIT States That Half of All Children May Be Autistic By 2025 Due to Monsanto

On the face of it, this is absurd. "MIT" states no such thing - in fact, the university itself does not endorse the work of any of its researchers. Moreover, Dr. Seneff is a highly controversial figure at best - an electrical engineer and expert in computational algorithms for understanding human language who has wandered into matters of public health, biochemistry, and epidemiology that are pretty far afield from her established expertise. Beyond presentations and talks, much of the work she has published in this area has been in the journal Entropy, itself a highly controversial publication whose parent organization, MDPI, has been accused of shady scientific practices. All of this gets into the realm of claim and counter-claim which can be VERY difficult for non-experts to sort through.

Folks that like to cite impressive institutions (MIT, Harvard, Johns Hopkins) on behalf of their claims are often quick to dismiss other equally-impressive institutions when scientists disagree with the argument they're trying to advance. This is another good indication that you may be dealing with a questionable argument - a persistent habit of attacking people rather than ideas (similar to the reliance on conspiracy theory, above). It's legitimate to question someone's credentials, because that's germane to whether they are a reliable source of information. Questioning motives, on the other hand, is generally out of bounds. Inconsistency doesn't look good either; note this article's introduction of its purported expert as a "Johns Hopkins University graduate" even though nearly every other graduate of that same institution engaged in research in the field of vaccines disagrees with him.

In the end, if you REALLY want to know what the state of knowledge is on a given subject, you first have to become conversant in the basics of the scientific method and you have to be willing to wade through an awful lot of material, some of which may be deliberately obfuscatory or misleading. That's a tall order for most folks, but if it matters to you to be right then that's what you have to do. This is one reason why scientific literacy is so important - more people need the tools to do this right. But equally important, people need the mental discipline to not "reason backwards", picking the answer they want based on their social tribe and then cherry-picking evidence to support it. That's probably the hardest barrier of all, and it's what separates actual scientists from wannabes with agendas.

When You're Losing, Change the Subject: Vaccines and the "Freedom" Agenda

I have blogged recently about the misuse of the term "research" by various movements seeking to argue with mainstream scientific consensus around a range of topics. People opposed to vaccines (led in particular by those who believe that vaccination causes autism and other maladies) have been in the news much lately, although the same arguments have been active around issues of climate change, evolutionary biology, and other issues. National Geographic recently published a cover story on this, provocatively titled (in the print version) "The War on Science".

The anti-vaccine movement has taken a lot of flak and pushback recently, both from the medical & public health community concerned that dropping rates of vaccination will undo progress from the last 100 years and from concerned parents afraid that the drop in herd immunity will expose their children to diseases long since removed from the population at large. It has been, from a social and political point of view, a fascinating and rare example of a loud minority being countered by a much larger majority. On a number of other issues vocal minorities hold sway simply because the larger population doesn't care enough. Apparently, public health is not one of those issues.

Given this pushback, it's unsurprising that the anti-vaccine movement is trying to change the terms of debate. Whenever somebody is losing an argument in the US, they try to turn it into an argument about Freedom, because Freedom is the one trump card that people think always wins the argument. This meme, now circulating on the internet, is emblematic of this approach:

I'm surprised that they didn't throw mom, the flag, and apple pie in there, but you get the picture. I mean, how can anybody possibly be against any of these wonderful things?

The organization sponsoring this particular argument, an outfit in California called Your Family, Your Choice, is trying to fight legislation that will take away the philosophical exemption to childhood vaccination currently allowed under California law. Many of the folks who oppose vaccination are indeed also opposed to science and research, at least as these things are understood by the medical research community. Such folks often want their own science, want to reach (or have already reached) their own conclusions, and are not kind (and often not very civil) to those who disagree with them.

But all of that is neither here nor there - what really interests me is the "I am pro-freedom" part of the argument. This is indeed the "go-to" for folks on the losing end of a public debate. Recently we've seen certain segments of our society opposed to gay marriage making the same claim on behalf of small businesses that don't want to serve gays - just as two generations ago, similar folks claimed "freedom" as a justification to turn away interracial couples. "Freedom" was the cry of George Wallace on the steps of the Alabama schoolhouse when he railed against "the oppression of the rights, privilege, and sovereignty" of his state in the face of integrationist pressure.

The fact is that we all give up a measure of freedom as the price of living in a civilized, advanced society. We agree not to drive through a red light. We agree to wear our seat belts - 49 out of 50 states in the US have some form of mandatory seat belt laws on the books. We agree to file certain kinds of information with the government at various levels. We agree to pay our taxes. We agree not to discriminate against fellow citizens when engaging in public commerce or service. Failure to do these things comes with the penalty of government sanction.

We suffer these infringements on our freedom because there are some collective goods that cannot be had otherwise. Because of our traffic laws - actually quite draconian by the standards of much of the world - we enjoy some of the safest highways and streets in the world, vastly safer than they were 80 or 100 years ago. Because of our attention to civil rights, populations once voiceless and enslaved are now freer and much better off and we are closer to realizing our ideals as a republic of equals.

Those who hold up the "pro-freedom" banner are trying to escape this reality. They want to free-ride on the rest of society, to deny that there are some things we can only have if we all contribute. Herd immunity from disease is one of those things, however much some folks may want to deny it.

So I find this latest adaptation by a political movement interesting, even entirely predictable. But I also suspect that it may be a last gasp of an effort that may soon collapse. The argument against vaccination is too weak, and the consequences too severe, for these folks to win. I have no doubt that we will see a great deal more shouting and gnashing of teeth in certain corners, and likely many more interesting internet memes. But as Richard Feynman reminded us a generation ago, for every successful technology "reality must take precedence over public relations, for Nature cannot be fooled."

Vaccines, Climate Change, and the Misuse of "Research"

Because of the increase in measles cases across the United States, a lot has been written recently about vaccines and the anti-vaccine movement. I've avoided blogging on the subject, partly because the discussion has been a lot of heat and not very much light and partly because it's not so much a debate or a conversation as a minority of folks screaming at the top of their lungs while holding fingers in their ears lest they listen to anyone else. Frankly a lot of it is pretty ugly, so while I have taken some pains to make sure I read some "authentic" voices on the anti-vaccine side, on the whole it all just makes me sad.

I do think that the continual parade of claims and counter-claims, of facts and factoids and made-up nonsense, is counterproductive because it isn't an argument about science or what's true. The primary drivers, as they so often are, amount to tribalism - people protecting their self-identities. I read this piece the other day and was glad to find a medical professional who gets this. Click the link - it's well worth a read.

So what do I have to add to this conversation? I want to focus on the use - and more particularly the misuse - of one word, "research". I have seen this word misused in recent weeks by anti-vaccine parents, I have seen it misused in exactly the same way by climate change "skeptics" in recent years, and in similar fashion by partisans on any number of issues over time.

As an illustration, let me borrow words from an anti-vaccine parent posted to the internet (underlining added):

You want to vaccinate? Go for it. I choose not to. But shouldnt we all have the freedom to choose what's right for our family? My unvaccinated child is not putting your vaccinated (and therefore protected-right?) child at risk. In fact it's the opposite. Your vaccinated child sheds his/her vaccine for days putting my child, and others who cannot vaccinate due to medical reasons, at risk. But I'll assume that risk and hope his immune system is strong enough to protect him. You do your thing. I'll do mine. Just stop assuming I don't care about others, because I do. I care enough to watch Vaccine Nation to understand herd immunity and research vaccines for myself. I've read the package inserts from the manufacturer websites. Have you?

Now, to be clear - I do not doubt this person's sincerity or motives. I believe this author genuinely believe these words. The problem here is a fundamental misunderstanding of the term "research".

I see this claim a lot, not just from anti-vaccine folks but from climate change skeptics, gun-rights advocates, and many other partisans for various issues. They will claim to have "researched" something to come to their own conclusions - a subtle cultural argument that appeals to Americans' individualism. After all, who wouldn't want to come to their own conclusions instead of being one of the "sheeple" who follow the herd, right?

The problem is that what these people call "research" consists only of reading stuff that other people have written. The most generous thing that could be said is that they are engaged in literature review - reading the research of others in order to get a sense of what's already out there in the field. Lit reviews, as anybody trained in any academic or scientific field knows, are a necessary precursor to doing research. But they are not research itself.

Actual research involves:

• Formulating a falsifiable hypothesis
• Designing a legitimate and non-biased test for that hypothesis including valid and reliable operational measures for the key concepts
• Gathering original data to perform that test
• Analyzing the results
• Carefully drawing conclusions by comparing the data with the hypothesis
and then (the gold standard)
• Allowing the entire process to be author-blind reviewed by other established experts in the field who are qualified to judge the work.

The parent cited above, and all of this ilk, haven't done any of this. All they have done is read some other stuff that other people did, usually based on non-falsifiable hypotheses to begin with (for example, "the CDC is the center of a global conspiracy to cover up the truth.")

Moreover partisans on a given issue don't even do lit review right, because they only read or watch the stuff that already agrees with their point of view. They will occasionally cite a peer-reviewed article, often misreading its conclusions in the process (since they are rarely, themselves, actual experts in the field). They dismiss studies that draw different conclusions and evaluate studies with the grossest and most obvious of double standards. Watching a documentary film made by this man hardly qualifies as a thorough review of the relevant literature on any subject.

This, of course, is why actual expertise is worth something. People who are qualified to conduct actual research in a given field have to spend years learning how to do it right. Because of my education, I am qualified to do research on political phenomena. I am not qualified to do research on vaccines. And neither are nearly all of the anti-vaccine voices out there.

So the next time you run into someone spouting what appears to be a questionable point of view on a given subject and claiming to have "done their own research", call them on it. Ask what that "research" consists of. Ask what data they have collected, what testing and analysis methods they used, and what hypothesis they were testing. I expect the answers you get will be much more heat than light, because this isn't really about science. But at least people should admit that.

How to Tell Your Industry Is In Trouble: Scare Tactics Disguised as "Research"

I've blogged before (here being the most recent) about the troubles of the for-profit, online education sector. The stories of closures, cutbacks, and retrenchments - not to mention investigations and legal woes - over the past year have been legion, and at some point I just stopped chronicling them. The evidence of an industry sector in crisis is pretty strong at this point.

So what does an industry in crisis do? Why, commission its own "research" and try to frighten states into backing off with their pesky investigations, that's what. We are now treated to just such a spectacle designed to "prove" how much states would have to spend if it weren't for those wonderfully civic-minded for-profits picking up the slack:

If For-Profits Vanished

One can only wonder what sort of hair-brained economic assumptions are baked into this "research" model. Given the radical decline of state support for higher education in recent years, I doubt very much that state legislatures would feel at all obligated to pony up $8.4 billion even if Phoenix and all of its ilk disappeared tomorrow. And given the marginal success rates of some institutions in the for-profit sector, it's likely that many of those 1.4 million students wouldn't go to college at all - which might not be a bad thing, especially if there are good alternatives in vocational schools and community colleges.

The fact that this "research" is funded by Phoenix's parent company and its founder's foundation, of course, rather gives the game away. No, funding does not always buy the results that you want. But given that the authors of this report seem to have gone out of their way to avoid bringing folks who might have a more objective (or even dissenting) view on board, it seems a pretty good bet that this was largely "made to order". I just hope that state legislators (including those in my own state) ignore this bit of nonsense.

More 'Advice' for Higher Education from the Peanut Gallery

There have always been plenty of outsiders willing to tell universities and colleges how they should run themselves. In recent years, some of this unsolicited advice has been based on legitimate grounds - rising debt levels among students, lower-than-expected graduation rates, questionable practices (especially in the for-profit sector), and scandals of various sorts have all contributed to the notion that something is wrong in higher education. And there is no shortage of suggested diagnoses for what that is.

Unfortunately, into that field have wandered various folks who don't really know what they're talking about, and whose advice isn't worth much. Today's Chronicle brings a story of just such an example: a report put out by two "financial consulting" firms about how a great many universities are on a "financially unsustainable" path.

As a side note - to make things interesting, one of these "consulting" firms is Mitt Romney's former haunt, Bain & Company. That doesn't really have any bearing on this particular report, but it's likely to heighten awareness of the story - and skew how people tend to see it. My dislike of the report has nothing to do with Romney, and everything to do with their approach itself.

The fundamental problem with the "analysis" done by these firms is their timeframe. They look at some 1700 public and private institutions specifically in the arena from 2005 to 2010 - in which fall some of the wildest fluctuations in endowment value, enrollment, state support, and revenue in living memory. Only a great idiot - or somebody with an axe to grind - would take this rather narrow time-frame of years (five years is the blink of an eye to a university) and attempt to make a trend-line argument about an "unsustainable path".

Then there is the broader global reality about outcomes and who has credibility to give advice to whom. University bankruptcies and closures are stunningly rare, even in recent years, and most universities have been around for decades - many for a century or more. What's the average lifespan of companies that Bain "advises"? Historically speaking, universities have been the model of "financial sustainability", as measured by how long they actually stick around as solvent institutions. Find me another industry with this many players (1700+), most of which last for decades or centuries.

Finally, they toss into their argument the old bugaboo about rising administrative costs. Yes, admin costs at universities have gone up. Bain blames boards and presidents, who need to "put their foot down" on rising administrative costs. As I've written before (here, here, and here), the forces that drive increasing administrative costs are complex and varied - and many of them are outside the control of universities. To argue that all a president or board need do is "put their foot down" is to display stunning ignorance about the industry - the kind of ignorance that should disqualify your "advice" from being seriously considered.

Higher education does indeed need some serious reform. There are significant problems, and past success (as the financial sector likes to say) may not indicate future results. But "advice" from consulting firms with questionable track records, and little discernible knowledge about how universities really work, is not going to help. Here's hoping that this "report" goes on the shelf quickly, there to be ignored in favor of serious conversation.

Where the Ivory Tower Meets Reality

As a political scientist, I've always had an interest in things that governments are also interested in. My interests and theirs don't overlap completely - there is much that political scientists do that people in government couldn't care less about - but in general, people in my field study things that are the province of governments. That's what we do.

So the culmination of this story should be of some interest - and serve as a warning - to me and my colleagues:

Boston College Must Release Records in IRA Oral-History Case, Appeals Court Says

In this case, researchers at BC have been studying something I too have studied - the Northern Ireland conflict. And they have done so using methods that friends of mine have also used - talking to actual participants to that conflict about what they did and why they did it. What the US courts have said is clear: if a government wants your research notes and data, your promises of confidentiality to those people don't mean squat.

I've been aware of this case, and I can't say that I'm surprised with the outcome. Rare indeed is the circumstance where some other value is permitted to trump a government claim that this or that is necessary for "state security". Even in this case, where the investigations are of past activity, the logic still holds.

This case reminds us of an important lesson for researchers who care about stuff governments care about (including political scientists but also sociologists, psychologists, and all sorts of other social scientists). The words from the ruling cited in the Chronicle article are telling:

"The choice to investigate criminal activity belongs to the government and is not subject to veto by academic researchers" 

The line drawn here seems clear enough - "criminal activity". But as historians and political scientists both know, governments can and have defined a broad range of behavior as "criminal activity". If someone decides to study homosexuality in Uganda (where homosexuality has been outlawed) can that government demand that the researchers turn over their records, and get the US courts to enforce that order, so it can hunt down gays and prosecute them?

I suspect (of course) that there's a political element here - that US courts are much more likely to rule in favor of a government we like (Britain) over an issue we agree with them on (terrorism) than side with a government we have a much less cozy relationship with (Uganda) over an issue we disagree about (homosexuality). Which suggests (again, no surprise) that researchers in the United States need to be careful about researching things that might run afoul of American interests as perceived by the US government.

Again, none of this is particularly surprising. The freedom granted by governments (and the Declaration of Independence notwithstanding, you only get freedom two ways - it is granted to you by the government, or you wrest it from the government through the use of power) has always been constrained by "state interests". As academics, we like to believe that there are higher values - and if you don't mind going to jail on occasion, you can press that view. Just don't expect "academic freedom" to be high on the priority list of those in power.