How Do We Measure Quality Teaching?

One of the hardest nuts to crack in academia is measuring the effectiveness and quality of faculty. This week's decision in Northern Iowa will probably only add fuel to the fire of that debate. Though the decision to disallow the use of student evaluations as the basis for merit raises was based on the legal issues of the union contract, not the merits of the issue, I think it's probably a better outcome anyway. And it opens up broader questions of how we measure the quality and productivity of faculty work.

There is one aspect of what faculty do that can be measured well: research and scholarship. It's very easy to develop metrics for both the quantity and quality of scholarly productivity - number of articles, books, or papers, citation counts, etc. These can be easily adapted discipline by discipline, while remaining rigorous and fair. Even the softer notion of "reputation" within the field is fairly easy to measure, which is why come promotion & tenure time we call on outside experts in the candidate's field to tell us this particular person's niche in their chosen pantheon.

Quantity of service is easy to measure - how many committees do you serve on? Given that most university committees produce little, and what they do produce is often at the expense of a great deal of time, quantity doesn't tell us much here. Quality of service is a much more challenging thing to measure. Over time, the true stars in the service field - those who can get things done, run efficient and effective meetings, and are critical to moving the institution forward - do emerge. Some of these get pulled into administration, where P&T measures don't matter as much. As for the rest, most faculty reward systems seem content to rely on a pretty minimal standard of "service" - so long as you're above that bar, there's no penalty, but there's not much reward for being far above it, either.

The real Gordian knot, of course, is teaching. This is what professors, in the public imagination, are primarily paid to do. At many universities, it is the primary mission - scholarly research being a somewhat distant afterthought (I have taught at such institutions). In today's economic climate, with the public and politicians demanding to see a "return on investment" in universities, what they mean is - are we doing a good job educating students for jobs, careers, and as productive members of society?

At the level of the individual faculty member, student teaching evaluations - especially quantitative measures - are a terrible means of getting at this. I don't mean that they lack value entirely; as a device to weed out the truly awful, they're great. But most professors, even those whose students may not be learning all that much, can manage to get at least passable marks on student evaluations. Some of them manage to get really good ratings, because they are charming and charismatic and popular (yes, there is a "popularity contest" component to student evals). 

This is not to say that student evaluations should be done away with - they can pick out the really bad teachers, and as a formative tool the comments that students write are very useful. I've used many student comments over the years to hone my own teaching, I think for the better. But when they are the only measure of teaching - as they usually are - they tell you next to nothing about what you want to know - who deserves the merit money this year and who doesn't?

Some years ago I was pulled into a debate, at both the department and university level, about how to measure faculty quality and productivity. Across the university there was very little consensus (in part because of deep philosophical and political divisions). Within the department, we all agreed that we had to have some additional measures beyond student evaluations. The only ones we could come up with were:

• Quantity measures - how many students did you teach? (funny how that doesn't come up much - but as a measure of productivity, surely it matters) 

• Peer evaluation - which is time-consuming, and dependent on the peers doing the evaluating themselves knowing what good teaching looks like, but which adds a component of validity when done well.

To this we might have added interviews of graduating seniors, to ask them about specific faculty and their impact on that student's formation. That, too, takes time and effort to organize and perform - but I suspect that it would tell us a lot.

The fact that the productivity of teaching is so hard to measure is actually much to the liking of many faculty, because it is a means of escaping accountability. So long as I am teaching my classes and my students are passing, if you can't tell how well I'm doing it I am free to put as much or as little energy into it as I like. This is not universal, but it is more common in higher education than we faculty would like to admit publicly.

Where we have dedicated, energetic teachers - and there are plenty of those - it is because of their own internal motivation, not because their universities reward them for it. And that, it has always seemed to me, is a terrible shame. Systems tend to produce more of what they reward, and less of what they ignore. If we really want higher-quality and more productive teaching, we need to find a way to seriously reward it.

Real Security is Hard; 100% Security is Impossible

Sometimes, the targets are too easy. We've heard stories for years of problems with the security screening network in the US; now comes this:

LA Airport TSA Screeners 'ran drug ring'

To calm a frightened public in 2001 and 2002, Americans were told that the "new" TSA and DHS would guarantee our safety. But there are no guarantees. TSA jobs are, by all accounts, frequently unpleasant and not all that well-paid. Oversight can be lax (no pun intended here). When that happens, you're bound to get things like this. And if you can bribe someone to smuggle drugs through, what else would they smuggle for money?

Too, there's the problem of "fighting the last war". In this case, we have poured billions upon billions into securing the air traffic network. Why? Because 9/11 involved airplanes, and because it looks like it might be doable. But the two largest terrorist attacks/attempts on US soil prior to 9/11 both involved rental trucks (one directed at the World Trade Center, no less) - did we go after the rental truck industry? Then there are boats, problematic border crossings, and all the rest of it.

The fact is that terrorism is all about asymmetry, which makes guaranteed security impossible. Moreover, in the grand scheme of things terrorism ranks far behind murder, traffic accidents, and any number of health issues as a threat to American lives. Yes, it's more economically disruptive - to a point. But those disruptions are incredibly rare - indeed, vanishingly so in recent years.

For the most part, terrorism isn't much on the public radar these days. The presidential candidates have other things to talk about that are more attuned to Americans' concerns. And it isn't that we don't need some kind of response to terror threats, just as we need systems for dealing with natural disasters, auto accidents, murders, and a host of other problems. Here's hoping that this year's president campaign doesn't wander into the silly territory of pandering, wherein candidates promise us complete immunity from terrorists. Because there ain't no such thing.

The "Mission" of College

For those of us in higher education, there is an age-old debate about what the primary mission of college is - to prepare students for productive careers, or to mold students into better citizens and community members. The tension is most keenly felt between the liberal arts and the professions. It's a divide that every institution experiences, either within itself (if they have both liberal arts and professional education on the same campus) or with the wider world ("pure" liberal arts schools have to constantly justify their existence to the outside).

Today the debate is as acute as ever. With a stagnant labor market, sluggish economy and high government deficits, universities and colleges are called on to justify the federal dollars spent on them in terms of their economic impact and contribution to the growth of business. As this article from the Chronicle of Higher Education points out, this is an ongoing conversation that affects all levels of the academy - all the way up to boards of trustees.

I should say by way of self-disclosure up front that I went to a "pure" liberal arts college, and loved it. I am a loyal alum, would not have traded that experience for anything, and hope that my kids get to experience the same. I got a first-class education there, and in the process came out a more developed person than I was when I went in - though the latter was more a function of learning to interact with my peers than through any formal college programming.

I also understand and agree with most of the arguments made by liberal arts programs about the potential value of those disciplines, both for career preparation and for making people better citizens and community members. Every year or so the New York Times writes another article about how businesses can't find employees who can write coherently, for example, underscoring the extent to which studying English really does give you an edge in the job market.

With all of that said, however, this latest conversation among trustees and university leaders about resisting the "pull" to "stray from their mission" of making better people instead of "just" doing job-training strikes me as a little disingenuous. It's not that these folks don't mean what they say - they are, I believe, largely sincere. The problem is that, if the "primary mission" of colleges and universities is to make "better people" instead of being job-training programs, there is precious little evidence that we're any good at it, and plenty of internal evidence that we have no idea what we're doing.

Take one of the more visible aspects of "good citizenship": voting. It is true, within any given election in the US, that the more educated you are the more likely you are to vote. See, the universities might say - a college education makes you a more engaged citizen! But this only makes sense if you ignore two really big trends over the last few decades: the substantial expansion of higher education (far more people go to college today than in the 1960s) and the long-term decline in voting (far fewer people vote today than did in the 1960s). If education is going up over time, why is voting going down?

It is also easy to point to examples of college-educated people being really horrible citizens and community members (Enron, anyone?) And there are certainly areas of the academy (MBA programs, perhaps?) that have some questions to answer about how people with their degrees so frequently seem to end up crossing ethical lines (the Wal-Mart bribery scandal being just the latest in a long string). Usually, the answer is, "well, that's not our fault - just a few bad apples!"

But in truth, we don't have any evidence that a university education systematically contributes to people becoming better citizens and community members. Maybe it does, maybe it doesn't - but if we want to claim that this is the "primary mission," you would think we would want to gather the best data available. Yet studies of this sort of thing are few and far between, and data hard to come by.

Moreover, faculty have no real idea how to do this kind of education. Scholarly organizations may have small corners in which discussions about "citizenship education" take place, but by and large they are ignored by the rest of us - and that's in political science, where some of the most vocal public arguments are made. Promotion & tenure processes don't reward "citizenship education" even remotely. Teaching & learning centers on campuses don't have workshops about it. There are neither incentives nor resources for faculty to engage in this kind of education, and so by and large we don't.

There's a legitimate argument about whether this is a bad thing, or whether things are OK the way they are. But universities can't credibly claim that educating people to be better citizens and community members is the "main mission" without evidence that we're actually doing it, or without spending any time and energy seriously trying. If we really want to be "educating the whole person," shouldn't we be able to demonstrate that we actually are?

How to Keep a Good Faculty-Administration Fight Going

If you are looking for a blueprint for a way to continue a running faculty-administration conflict, Idaho State appears to be writing the book.

Idaho State U. to Hold Faculty Senate Elections on Administration’s Terms

First the administration disbands the faculty senate, then installs an interim body with an expiration date. Then it imposes elections for a new body - which apparently doesn't have a constitution yet - and bans anybody who served on the previous two bodies from running. That's quite a string of decisions.

I have no idea about the specifics of this case - I don't know anybody at Idaho State. It could be that the faculty have been (to borrow a phrase often applied to Harvard faculty) "ungovernable". It could be that the administration is power-crazed and unreasonable. But in this case, I don't have to know who's right and who's wrong to know what will happen next.

Any body elected to represent the views of the faculty at large will either represent those views or it won't. At this point, it's a good bet that there's a healthy amount of anti-administration sentiment among the faculty there. It's likely that the reps who served on the disbanded senate, and possibly the now-expired one, represented those views faithfully. Maybe the administration hopes to eliminate those views from a new senate.

But that's where this is guaranteed to continue the fight. If anti-administration sentiment is significant, one of two things will happen. Either the faculty will find new people to elect that represent that view, or the administration will get its wish of a pro-administrative senate. In the latter case, anti-administration faculty will feel disenfranchised, and deny the new senate (and any new constitution) legitimacy.

This is the thing about university conflicts. Unlike the world of civil wars, there is no "winning". You can't beat the other side into submission, especially if you're the administration side. Every once in a while, a faculty manages to run off a president - though serious problems in administration usually go deeper than one person. But in generally, there is no equivalent of a "military victory". The only choice is finding a way to get along with the other side, and get done what needs to be done.

Sometimes, of course, that's not possible, and so the conflict continues. Idaho State may very well be stuck right now in that kind of impasse. If so, they will stay there until a solution is worked out that is acceptable to all sides - whether that takes a year, or five years, or twenty. There are no alternatives - only longer and shorter conflicts, and more or less cost to the institution.

Keeping Government Waste in Perspective

A brief response to an article in today's Chronicle on Congressional earmarks for university research:

I'm as much against government waste as the next person - heck, I probably regard a broader range of government spending as "wasteful" than many. But $44 million is, in all senses, chump change. Obviously, from the perspective of the US government budget (north of $2 trillion and rising), this is a rounding error. But even for universities, it's almost nothing. My own university, a mid-sized state institution that does a reasonable amount of research, pulls in nearly $40 million per year in competitive research grants all by itself. There are dozens of institutions like mine across the country. Large, major research universities - and there are a goodly number of those - pull in ten times as much or more.

I've always been uncomfortable with legislative earmarks - they are basically spending public money at the behest of a particular legislator on some pet project. A previous college I worked for got a chunk of money this way from a state legislator who was retiring. I can't say that the state in question got any public good for its expenditure of a half a million or so taxpayer dollars - especially given that the institution in question was a private school, and had no clear plan for what to do with the money. That always struck me as shady - a means by which an outgoing legislator could thank his friends.

So I'm no fan of earmarks, especially for colleges and universities. But if we've gotten down to only $44 million in research earmarks, we should be celebrating. After all, how often is it that government measurably improves its performance?

The Growth of Higher Ed Administration, Part II

A little while back, I wrote a blog about the cost of administration in higher education - and how a lot of forces drive it. Undoubtedly heartening to all of my Republican friends, one of those forces is clearly government regulation. When governments create new regulations for the higher ed community, administrative costs go up to comply with those regs.

I came face-to-face with a real-life example of this just a couple of days ago. My institution, and likely every other university larger than a small college, has been forced to add at least one full-time staff person to its academic affairs division - not because we like hiring more administrators, but because the world just got more complicated.

The background to this new regulation has been widely covered. The number of for-profit universities has skyrocketed in recent years, most of them relying heavily on online education. A few, like the University of Phoenix, actually put a physical presence in places where they teach; many do not. And in that sector, there have been some serious abuses - universities enrolling very poor students who could get large, federally-subsidized loans and then delivering substandard education in return. A few of the most egregious cases likely crossed the line into outright fraud.

To combat this problem, the federal Department of Education created a new set of rules regarding on-line education. Of course, when you create a rule about something you affect everybody who engages in that something. On-line education has been growing in the non-profit sector, too, especially in certain fields like nursing and business. Much of this activity is quite legitimate, high-quality education. But the law is the law, and we all have to follow it.

The crux of the new rule is that universities have to have permission to do business (that is, to teach students) in every state in which their students reside. So if I, at an Ohio institution, want to enroll a student from Arizona in my on-line program, I have to get permission from the state of Arizona. So far, this seems reasonable.

The problem comes in actually complying with the rule. No matter where you are, there are 49 other states that you could potentially have to seek permission from. Every one of those states defines "education" a little differently. Every one requires a different paperwork process to file for permission, some of them quite lengthy. Many of them charge fees - some of them quite hefty fees, $15,000 or more. Some of them require permission by program, with a separate fee (up to $5000) for each program you want to offer to the residents of that state.

Now imagine the task of looking up all of those laws and regulations; contacting the relevant state officials in all 49 states that aren't yours; re-contacting them when they don't get back to you in a timely fashion; filing out the necessary paperwork; tracking down which programs within your own institution have students in which states; arranging payments to each state as needed; and then tracking all of this going forward in time, because many states require re-registration after a given period. That, in and of itself, is a full-time job for a professional staff person, even if your university only has a dozen or so online programs. If you're a really big university with a lot of online programs, you might employ two or three people doing this.

The momentum of regulation, of course, is always forward and upward. Governments almost never take rules away; they just add more. So the addition of each new rule will add another few staff to every university. It's a sort of back-door jobs program for university administration. And over time, it does tend to tilt the proportion of resources away from the delivery of academic programs and towards the administration of them. Just don't blame it on the greedy, evil administrators. Blame the folks in Washington, and the 50 state capitols, who are telling your universities how to spend their money.

Why I Am a Self-Defense Pacifist

I spend a lot of time thinking, writing, and teaching about conflict. Much of the writing and teaching has been about the use of force - our polite academic term for violence or, more bluntly, killing other people. I think that at some level, you can't spend that much time thinking about this topic without developing a personal philosophy - because to study violence is to confront the question, when would I be willing to use violence?

Some years ago, I was struck by this quote from Elbert Hubbard, an American writer in the late 19th & early 20th centuries who died when the Lusitania went down:

“So long as governments set the example of killing their enemies, private individuals will occasionally kill theirs.” 

Political scientists sometimes define government as the thing that holds "a monopoly on the legitimate use of force", but that's never true for two reasons. First, very few governments prohibit force in direct self-defense. And second, "legitimate" is a circular claim: if a government does it, it's legitimate, because "legitimate" and "legal" are generally synonyms. And who stops the government from illegitimate actions? When has a government ever admitted to illegitimacy? 

Every government defines its wars as legitimate, regardless of how they are entered into (when is the last time the US Congress declared war per the Constitution?), how they are conducted (Abu Ghraib? Afghanistan? My Lai is apparently not so far behind as we would like), or how much support they do or don't have from the public. Governments have always used force in patently illegitimate ways - think of the internment camps for Japanese-Americans in World War II. This isn't going to change.

And because governments set this example, others will follow it. The latest headline-grabbing case is Anders Breivik in Norway. His claim is that he was acting as a patriot, to protect the "legitimate" claims of the "Norwegian nation" against a government that had sold his nation out. He insists that he was justified, and that he would do it again if given the chance.

Very appropriately, we want to put both his argument and his actions beyond the pale. The moral high ground, it seems, is easily found here. But it's harder to claim that ground when we condemn the killing of civilians in some contexts and condone it in others. Our willingness to erode a host of protections - one recent example being a legal opinion within the White House that it would be OK to kill a US citizen abroad without trial - in the name of "security" just blurs the line further. As Gandhi reminded us,

What difference does it make to the dead, the orphans and the homeless, whether the mad destruction is wrought under the name of totalitarianism or the holy name of liberty or democracy?

The more different justifications we come up with for killing, the more individuals and governments will find ways to twist those justifications to perverted ends. I do not believe we are likely to get a societal consensus on who we think should be killed and why - but at the very least, I wish more people would seriously confront the question.

The confusion, the blurring of lines, the stretched justifications for me cry out for a simpler answer. I do see legitimacy in using violence to fend off an attack, provided you are actually being attacked. Pre-emptive violence against those who might attack later, or retributive violence against those who did something bad before, or violence against metaphorical attacks against "honor" or "purity" - all of these open a door to very dark places time and again. Instead of contributing to the problem, let's draw a line where we can all see it and agree upon it - and let's hold ourselves and our representatives to that line. If we want to lay claim to being a civilized and peaceful people, the way we want to understand those words, it's the least we can do.