So the culmination of this story should be of some interest - and serve as a warning - to me and my colleagues:
I've been aware of this case, and I can't say that I'm surprised with the outcome. Rare indeed is the circumstance where some other value is permitted to trump a government claim that this or that is necessary for "state security". Even in this case, where the investigations are of past activity, the logic still holds.
This case reminds us of an important lesson for researchers who care about stuff governments care about (including political scientists but also sociologists, psychologists, and all sorts of other social scientists). The words from the ruling cited in the Chronicle article are telling:
"The choice to investigate criminal activity belongs to the government and is not subject to veto by academic researchers"The line drawn here seems clear enough - "criminal activity". But as historians and political scientists both know, governments can and have defined a broad range of behavior as "criminal activity". If someone decides to study homosexuality in Uganda (where homosexuality has been outlawed) can that government demand that the researchers turn over their records, and get the US courts to enforce that order, so it can hunt down gays and prosecute them?
I suspect (of course) that there's a political element here - that US courts are much more likely to rule in favor of a government we like (Britain) over an issue we agree with them on (terrorism) than side with a government we have a much less cozy relationship with (Uganda) over an issue we disagree about (homosexuality). Which suggests (again, no surprise) that researchers in the United States need to be careful about researching things that might run afoul of American interests as perceived by the US government.
Again, none of this is particularly surprising. The freedom granted by governments (and the Declaration of Independence notwithstanding, you only get freedom two ways - it is granted to you by the government, or you wrest it from the government through the use of power) has always been constrained by "state interests". As academics, we like to believe that there are higher values - and if you don't mind going to jail on occasion, you can press that view. Just don't expect "academic freedom" to be high on the priority list of those in power.